CrashPlan for Backup on Nexenta

For some time, I've been using cron jobs with rdiff-backup on Cygwin for backups. The cron job runs on a Windows server I have on my home network and iteratively mounts remote Windows shares (if they're present) and runs rdiff-backup over them, with the destination also being local. Then the job runs rsync to mirror this backup to my Nexenta NAS running ZFS raidz.

This solution gives me a fair amount of local redundancy: two separate machines, with two copies of the backup data, plus the RAID-5-like redundancy and checksumming integrity that ZFS provides. Rdiff-backup is quite reassuring with respect to restores too: it stores files in the filesystem directly, along with differences (the rdiff bit) so you can go back in time. That means that restoring is as simple as copying the files straight out of the backup and deleting the rdiffs.

Of course, a backup strategy isn't solid without a remote copy. Today, I finished configuring CrashPlan, a really neat backup solution built using Java. You can read about the features etc. of CrashPlan on the website - I heard about it from the Java Posse podcast, episode 298. Initially, I opened an Amazon S3 account, and was considering mirroring my backups to S3 with s3sync, but after I evaluated CrashPlan, it looked like it made more sense than further pursuing my homegrown approach. Not only is it cheaper than S3 for my data (I guess they depend on overselling like ISPs), but CrashPlan has features that make it work well in my case.

The fact that it's primarily Java meant I could install it on my Nexenta box. CrashPlan don't have an installer for Nexenta, but they do have ones for Linux (LSB) and Solaris, and Nexenta is like a hybrid. I'm documenting the steps here in case I need to set it all up again, though hopefully not for a restore.

I'm running NexentaCore NCP 2, which doesn't come with Java. Installing a JDK (overkill, but I wanted to test the Java environment with a simple hello-world):

$ sudo apt-get install sun-java6-jdk

I was running a release candidate of Nexenta and I had to do an apt-get update and upgrade to resolve all necessary dependencies, but it was fairly stress free because Nexenta's apt-clone checkpointed the root file system with a ZFS snapshot. I did lose my almost half-year of uptime though - my Nexenta machine has been the most dependable of all my machines, in both hardware and software.

Installing CrashPlan itself needs bits from both the Linux and Solaris installers. I downloaded both, and unpacked both. In the Linux install, I ran the provided install.sh and followed the steps, putting it in /usr/local, with daemon init script in /etc/init.d and runlevel symlink in /etc/rc3.d. But the Linux install isn't enough. In particular, the init script assumes GNU ps, but Nexenta uses Solaris ps. So I swapped in CrashPlanEngine (the target of the init script symlink) from the Solaris installer in the place of the installed CrashPlanEngine that was here:

/usr/local/crashplan/bin/CrashPlanEngine

But that wasn't enough. CrashPlan loads a library called libjtux via JNI, a kind of POSIX API for Java programmers who want direct access to the OS. The libjtux.so from the Linux install was linking against libc.so.6, assuming GNU C library versioning. Replacing libjtux.so with the version from Solaris, linking against plain libc.so, solved this problem - here:

/usr/local/crashplan/libjtux.so

Finally, I had to configure CrashPlan. My Nexenta install is headless - all my interaction with it is either over Samba shares, HTTP for wiki servers etc., and of course ssh for everything else. Here's one of the really neat features of CrashPlan: the user interface for configuration is a client that depends on a single TCP connection with the local backup server. All I had to do is get the client to connect to a different local port, and tunnel that port to Nexenta using SSH, after enabling TCP forwarding in Nexenta in /etc/ssh/sshd_config. This bit is described on CrashPlan's site describing how to configure a headless client.

I'm running CrashPlan+, a for-pay version of the engine, on two machines - my main desktop and my Nexenta box - to get all the encryption, compression, deduplication etc. goodness. And considering that CrashPlan supports peer to peer backup, I may simply replace my existing ad-hoc rdiff-backup approach with local CrashPlan backups, as CrashPlan supports multiple destinations for backups, as well as receiving backups from other machines.

A key limitation of CrashPlan, and one that I found particularly annoying, is that the Windows client doesn't support network shares at all, in any shape or form - whether they're mapped to drive letters or not. The backup engine runs under the SYSTEM account, so it doesn't have network credentials, and also means it may not be able to access all the files you're trying to back up - especially EFS encrypted files. (CrashPlan doesn't seem to use Windows' EFS backup capability, e.g. ReadEncryptedFileRaw.) I changed the CrashPlan engine's service account to my own account to try and close this hole, but still no go on accessing network shares. This meant that I had to get CrashPlan running on my Nexenta box in order to store local backups on ZFS, should I so choose.

But apart from that, I've been impressed with CrashPlan's feature set and usability.