Saturday, March 13, 2010

CrashPlan for Backup on Nexenta

For some time, I've been using cron jobs with rdiff-backup on Cygwin for backups. The cron job runs on a Windows server I have on my home network and iteratively mounts remote Windows shares (if they're present) and runs rdiff-backup over them, with the destination also being local. Then the job runs rsync to mirror this backup to my Nexenta NAS running ZFS raidz.

This solution gives me a fair amount of local redundancy: two separate machines, with two copies of the backup data, plus the RAID-5-like redundancy and checksumming integrity that ZFS provides. Rdiff-backup is quite reassuring with respect to restores too: it stores files in the filesystem directly, along with differences (the rdiff bit) so you can go back in time. That means that restoring is as simple as copying the files straight out of the backup and deleting the rdiffs.

Of course, a backup strategy isn't solid without a remote copy. Today, I finished configuring CrashPlan, a really neat backup solution built using Java. You can read about the features etc. of CrashPlan on the website - I heard about it from the Java Posse podcast, episode 298. Initially, I opened an Amazon S3 account, and was considering mirroring my backups to S3 with s3sync, but after I evaluated CrashPlan, it looked like it made more sense than further pursuing my homegrown approach. Not only is it cheaper than S3 for my data (I guess they depend on overselling like ISPs), but CrashPlan has features that make it work well in my case.

The fact that it's primarily Java meant I could install it on my Nexenta box. CrashPlan don't have an installer for Nexenta, but they do have ones for Linux (LSB) and Solaris, and Nexenta is like a hybrid. I'm documenting the steps here in case I need to set it all up again, though hopefully not for a restore.

I'm running NexentaCore NCP 2, which doesn't come with Java. Installing a JDK (overkill, but I wanted to test the Java environment with a simple hello-world):

$ sudo apt-get install sun-java6-jdk

I was running a release candidate of Nexenta and I had to do an apt-get update and upgrade to resolve all necessary dependencies, but it was fairly stress free because Nexenta's apt-clone checkpointed the root file system with a ZFS snapshot. I did lose my almost half-year of uptime though - my Nexenta machine has been the most dependable of all my machines, in both hardware and software.

Installing CrashPlan itself needs bits from both the Linux and Solaris installers. I downloaded both, and unpacked both. In the Linux install, I ran the provided and followed the steps, putting it in /usr/local, with daemon init script in /etc/init.d and runlevel symlink in /etc/rc3.d. But the Linux install isn't enough. In particular, the init script assumes GNU ps, but Nexenta uses Solaris ps. So I swapped in CrashPlanEngine (the target of the init script symlink) from the Solaris installer in the place of the installed CrashPlanEngine that was here:


But that wasn't enough. CrashPlan loads a library called libjtux via JNI, a kind of POSIX API for Java programmers who want direct access to the OS. The from the Linux install was linking against, assuming GNU C library versioning. Replacing with the version from Solaris, linking against plain, solved this problem - here:


Finally, I had to configure CrashPlan. My Nexenta install is headless - all my interaction with it is either over Samba shares, HTTP for wiki servers etc., and of course ssh for everything else. Here's one of the really neat features of CrashPlan: the user interface for configuration is a client that depends on a single TCP connection with the local backup server. All I had to do is get the client to connect to a different local port, and tunnel that port to Nexenta using SSH, after enabling TCP forwarding in Nexenta in /etc/ssh/sshd_config. This bit is described on CrashPlan's site describing how to configure a headless client.

I'm running CrashPlan+, a for-pay version of the engine, on two machines - my main desktop and my Nexenta box - to get all the encryption, compression, deduplication etc. goodness. And considering that CrashPlan supports peer to peer backup, I may simply replace my existing ad-hoc rdiff-backup approach with local CrashPlan backups, as CrashPlan supports multiple destinations for backups, as well as receiving backups from other machines.

A key limitation of CrashPlan, and one that I found particularly annoying, is that the Windows client doesn't support network shares at all, in any shape or form - whether they're mapped to drive letters or not. The backup engine runs under the SYSTEM account, so it doesn't have network credentials, and also means it may not be able to access all the files you're trying to back up - especially EFS encrypted files. (CrashPlan doesn't seem to use Windows' EFS backup capability, e.g. ReadEncryptedFileRaw.) I changed the CrashPlan engine's service account to my own account to try and close this hole, but still no go on accessing network shares. This meant that I had to get CrashPlan running on my Nexenta box in order to store local backups on ZFS, should I so choose.

But apart from that, I've been impressed with CrashPlan's feature set and usability.


Jon Carlson said...

CrashPlan does have a workaround for accessing mapped drives. This workaround should also work for Windows EFS:

Barry Kelly said...

Jon, I'm already running CrashPlan under my own account rather than the SYSTEM account because the SYSTEM account doesn't have permission to read all the files I want to back up; also, some files are encrypted, and running the service under my account will let them be read.

But even under my account, which does have mapped network drives, the drives don't show up in the UI. Things aren't made easier by the relative lack of logging provided by the UI - e.g. access denied errors the service runs into when traversing the file system are silently swallowed, rather than raised to the user sooner or later.

Jeff said...

Great post, and was just what I was looking for when trying to figure out how to install CrashPlan on Nexenta 3.01. I followed your instructions, but ran into some issues. Being a newbie, maybe you might have some advice. First, when running the Linux CrashPlan installer, I get the following error:

"The current version of Java, , is incompatible with CrashPlan.
Please install one of the following version of Sun Java Runtime: 1.5 1.6 1.7

The current installed version of Java is not the Sun Java Runtime Environment.
CrashPlan requires the Sun JRE.

ERROR: Failed to find an acceptable Sun Java Runtime Environment
These paths were searched: /bin /usr/bin /usr/local/bin
Would you like to download the JRE and dedicate it to CrashPlan? (y/n) [y] n

We're sorry, CrashPlan requires a valid Sun Java VM. Please install one and then
rerun this installer. Exiting."

I had already installed Sun Java 6 JDK via sudo apt-get install sun-java6-jdk. I then let CrashPlan install the JRE and swapped out the mentioned files with the Solaris versions. When trying to start the CrashPlan engine, I got the error "Java: Cannot find /lib/" Obviously, this file isn't available on Solaris and Crashplan's engine won't start and run without it. I tried symlinking to /lib/, but it didn't help. Thanks for any help you could provide.

Barry Kelly said...

Jeff - if CrashPlan is installing a local JRE, it'll be the one corresponding to the installer's target platform; if it's the Linux installer, then it'll be a Linux JRE, which of course won't run on a Solaris kernel (as I understand it).

It's very possible they changed their installer's mode of operation since I did my installation.

You definitely don't want to be using a Linux JRE. The JRE from apt-get etc. is, as you note, of the 1.6 flavour, and it does work.

I'll write up a post on roughly how to put it together manually with the current installer.

Glenn said...

This is sort of a response for Jeff. You can't install the local copy of the JRE that the installer wants because it's the linux version and you need to use the Nexenta's java. I was doing a fresh install Nexenta 3.01 and Crashplan 3.02. The Java that gets loaded

apt-get install sun-java6-jdk

would cause the crashplan linux installer to say incompatible java version. To fix this I had to install

apt-get install sunwlibc

this resolved the missing library.

Then follow the steps above and it's working like a charm. If you installed the local one.. make sure you change crashplan's java location.