Your Cheap, Scalable and Safe DropBox with Git-Annex, LVM, Raspberry PI and USB Sticks

There is no cheaper storage than USB sticks. When writing this, they're 30 bucks for 64GB. Unlike hard drives, they don't eat a lot of juice when you keep them plugged in, they're quiet, they are not sensitive to having a dusty home (thank god!), and they last. It's also very easy to wipe out the dumb cruftware that comes pre-installed on them.

If you're a command line guy like me, it's obvious that you must have a Raspberry Pi server at home. It's also obvious that you need to cram it full as many USB sticks as you can, in order to thumb your paranoid nose at Dropbox. Before pointing said nose at a full screen terminal and punching in powerful, arcane commands that connect you to the fathers of Unix. And feel like them. Even though you have a mouse, copy-and-paste, and command completion.

What gets really, really annoying is having your assorted drives from different eras and sizes on different mount points. Enter LVM.

When you get your next bigger thumb drive and cram it into your pi, don't do mkfs /dev/sdx1 right away. (/dev/sdx1 is the placeholder device name used in this post for your new usb drive, which assumes you have already connected 23 thumb drives to your pi, which is the only reasonable thing to do. If you're new to this, your actual device name could also be /dev/sda1 or /dev/sdb1).

So instead of doing mkfs -t ext4 /dev/sdx1 right away, you do: # Prepare a device to become part of a volume group, # which is LVM's equivalent of a fraternity hazing # a prospective new member pvcreate /dev/sdx1 # Create the new volume group and add /dev/sdx1. # Since the member no langer has a notion of individuality # he may be admitted to the group. vgcreate vg0 /dev/sdx1 # Create a new logical volume. Delightfully, the member # now expresses himself exlusively through this new group identity. lvcreate -n store --size 64G vg0 mkfs.ext4 /dev/vg0/store mount -t ext4 /dev/vg0/store /mnt/store So far this is just typing extra commands, but now you can transfer the data of one of your existing 23 usb sticks into /mnt/store. mount /dev/sdw1 /mnt/old mv /mnt/old/* /mnt/store umount /mnt/old And admit it to the group # Haze pvcreate /dev/sdw1 # Admit vgextend vg0 /dev/sdw1 # Extend block device lvextend -L+32G /dev/vg0/store # Extend file system umount /mnt/store resize2fs /dev/vg0/store mount -t ext4 /dev/vg0/store /mnt/store

Et voila! Instead of two annoyingly invidual mount points with 30G and 60G capacity, you have one with 90G capacity!

This means, whenever you run out of space, instead of extending your dropbox plan, you just add another thumb drive, haze him, and admit him to your group.

Of course you need some data security, so theoretically you should use half of your usb drive army to create a software raid1, and then pile your LVM on top of that.

That's where git annex does a world of good.

Like with any decent dropbox replacement, you will probably have all those files on your laptop, your server, your home laptop, and the server in the basement you forgot about. That is more than two copies, and that is a good thing. You also get checksums.

So if you put your git annex on your LVM, you will have a pool of checksummed extendible storage that you can sync to any command line you come across.

And if you care about someone enough to bother, and that person has a recent mac, you can set up the same thing with git annex assistant.