Monday, November 15, 2010

CrashPlan manual installation approach on Nexenta

I wrote an earlier post about installing CrashPlan on Nexenta. However, it seems that CrashPlan have changed their Linux installer's modus operandi, and it tries to install a (Linux) JRE of its own. Of course, that won't work; it needs the Nexenta JRE (but I get the whole JDK, as you never know when you'll need to brew up some Java):

$ sudo apt-get install sun-java6-jdk

Anyhow, I received some comments and emails about setting all this up, and figured I'd break it down a little more for people who still want to get it all working. It's a process of hacking things together, though, not a blind recipe; that's why I'm proceeding almost as if it's a debugging session.

Anyhow, once we have a JRE installed (from apt-get above), we can try and extract out the guts of the Linux installer, so rather than running its install.sh, we can set it up manually.

I went to the CrashPlan Linux download web page, and started downloading the installer in a scratch folder (this was the URL at the time of writing):

$ wget http://download.crashplan.com/installs/linux/install/CrashPlan/CrashPlan_2010-03-08_Linux.tgz

That's a gzipped tar archive, so I extracted it:

$ tar xzf CrashPlan_2010-03-08_Linux.tgz
$ cd CrashPlan-install # the dir it extracted
$ ls
CrashPlan_2010-03-08.cpi  EULA.txt  INSTALL  README  install.sh  scripts  uninstall.sh

I didn't know what the big CrashPlan*.cpi file was, so I checked:

$ file CrashPlan_2010-03-08.cpi
CrashPlan_2010-03-08.cpi: gzip compressed data, was "CrashPlan_2010-03-08.cpi", from Unix, max compression

So it's a gzipped file! I decompressed it, and tested the result:

$ gunzip < CrashPlan_2010-03-08.cpi > test
$ file test
test: ASCII cpio archive (pre-SVR4 or odc)

A cpio archive! I extracted that too, but in a separate directory:

$ mkdir t; cd t; cpio -i ../test
$ ls
bin  conf  doc  jniwrap.lic  lang  lib  libjniwrap.so  libjniwrap64.so  libjtux.so  libjtux64.so  log  skin  upgrade

Now this looks very close to the root listing of my actual current CrashPlan install on Nexenta, and by and large it is. Here's a rough difference between the two (listing.curr is my actual install, listing is extracted from the installer):

--- listing.curr        2010-11-15 06:32:12.548701734 +0000
+++ listing     2010-11-15 06:31:40.726884007 +0000
-CrashPlanEngine.pid
-bin/CrashPlanDesktop
-bin/CrashPlanEngine
-bin/CrashPlanEngine.lsb
-bin/run.conf
-bin/vars.sh
-conf/my.service.xml
-conf/service.login
-conf/service.model
-install.vars
-libjtux.so.lsb
-libjtux.so.sol

Let's go through those. This install tree is almost complete; it can be put in /usr/local/crashplan (that's where I have mine) or wherever you like, so long as the configuration bits are also hooked up appropriately.

CrashPlanEngine.pid is just the pidfile, a text file containing the process id of the currently running instance. I'm not sure why it's in the crashplan directory rather than somewhere like /var/run, but it is. I believe the crashplan service will create it.

CrashPlan* are all shell wrappers for Java applications; *.lsb (LSB for Linux Standard Base) are the original versions (I think I renamed them to this). I don't have an X server or libraries on my Nexenta install, so I'm uninterested in CrashPlanDesktop. CrashPlanEngine however is important; it's the main daemon file. Here's mine, which seems to work; I think I may have edited it to work with some GNU utils rather than Solaris utils, as I believe this came from the Solaris installer (or vice versa; actually that's probably more likely):

#!/bin/bash

TARGETDIR="`dirname ${0}`/.."

. ${TARGETDIR}/install.vars
. ${TARGETDIR}/bin/run.conf

cd ${TARGETDIR}

case $1 in
        start)
                PID=`/usr/bin/ps -Af -o pid,ppid,args | grep 'app=CrashPlanService' | grep -v grep | cut -f2 -d' '`
                if [ -n "$PID" ]; then
                  echo CrashPlan is already running with pid $PID
                  exit 1;
                fi
                echo "Starting CrashPlan Engine ... "
                nice -n 19 ${JAVACOMMON} ${SRV_JAVA_OPTS} -classpath "./lib/com.backup42.desktop.jar:./lang" com.backup42.service.CPService > ${TARGETDIR}/log/engine_output.log 2> ${TARGETDIR}/log/engine_error.log & 
                if [ $! -gt 0 ]; then
                        echo $! > ${TARGETDIR}/CrashPlanEngine.pid
                        echo "OK"
                else
                        echo "FAIL" 
                        exit 1
                fi
                ;;
        stop)
                echo "Stopping CrashPlan Engine ... "
                if [ -f ${TARGETDIR}/CrashPlanEngine.pid ] ; then
                  kill `cat ${TARGETDIR}/CrashPlanEngine.pid`
                  sleep 5
                fi
                PID=`/usr/bin/ps -Af -o pid,ppid,args | grep 'app=CrashPlanService' | grep -v grep | cut -f2 -d' '`
                if [ -n "$PID" ]; then
                  echo Still running, killing PID=$PID
                  kill -9 $PID
                fi
                rm -f ${TARGETDIR}/CrashPlanEngine.pid
                echo "OK"
                ;;
        *)      
                echo "$0 "
                exit 1
                ;;
esac

As you can see, that script sources (includes) a couple of other guys, bin/run.conf and install.vars. bin/run.conf looks like this:

SRV_JAVA_OPTS="-Dfile.encoding=UTF-8 -Dapp=CrashPlanService -DappBaseName=CrashPlan -Xms20m -Xmx512m -Dsun.net.inetaddr.ttl=300 -Dnetworkaddress.cache.ttl=300 -Dsun.net.inetaddr.negative.ttl=0 -Dnetworkaddress.cache.negative.ttl=0"
GUI_JAVA_OPTS="-Dfile.encoding=UTF-8 -Dapp=CrashPlanDesktop -DappBaseName=CrashPlan -Xms20m -Xmx512m -Dsun.net.inetaddr.ttl=300 -Dnetworkaddress.cache.ttl=300 -Dsun.net.inetaddr.negative.ttl=0 -Dnetworkaddress.cache.negative.ttl=0"

install.vars looks like this (for my current install):

TARGETDIR=/usr/local/crashplan
BINSDIR=/usr/local/bin
MANIFESTDIR=/tank/share/backup/crashplan
INITDIR=/etc/init.d
RUNLVLDIR=/etc/rc3.d
INSTALLDATE=20100310
APP_BASENAME=CrashPlan
DIR_BASENAME=crashplan
DOWNLOAD_HOST=download.crashplan.com
JAVACOMMON=/usr/bin/java

The MANIFESTDIR variable points to a directory I expect I provided when I ran the original install.sh, but there's nothing currently in it. The other paths are as you'd expect.

As far as I can make out, the conf/* files are created by CrashPlan itself in coordination with the configuration utility, which, as I linked in my previous post, can be run remotely from a Windows (or other OS) install of CrashPlan. There's a default.service.xml as a prototypical my.service.xml, while service.{login,model} appear to be login and encryption keys.

That leaves libjtux.so.*. These are renamed versions of the file libjtux.so which differs between Linux and Solaris. Primarily they differ in which C library they link against (note: running ldd on untrusted binaries is unsafe, but I trust these binaries):

$ ldd libjtux.so.*
warning: ldd: libjtux.so.lsb: is not executable
libjtux.so.lsb:
        librt.so.1 =>    /lib/librt.so.1
        libnsl.so.1 =>   /lib/libnsl.so.1
        libc.so.6 =>     (file not found)
        libmp.so.2 =>    /lib/libmp.so.2
        libmd.so.1 =>    /lib/libmd.so.1
        libscf.so.1 =>   /lib/libscf.so.1
        libc.so.1 =>     /lib/libc.so.1
        libuutil.so.1 =>         /lib/libuutil.so.1
        libgen.so.1 =>   /lib/libgen.so.1
        libm.so.2 =>     /lib/libm.so.2
warning: ldd: libjtux.so.sol: is not executable
libjtux.so.sol:
        librt.so.1 =>    /lib/librt.so.1
        libsocket.so.1 =>        /lib/libsocket.so.1
        libnsl.so.1 =>   /lib/libnsl.so.1
        libc.so.1 =>     /lib/libc.so.1
        libmp.so.2 =>    /lib/libmp.so.2
        libmd.so.1 =>    /lib/libmd.so.1
        libscf.so.1 =>   /lib/libscf.so.1
        libuutil.so.1 =>         /lib/libuutil.so.1
        libgen.so.1 =>   /lib/libgen.so.1
        libm.so.2 =>     /lib/libm.so.2

Anyhow, we need to get the right version of libjtux.so. It seems to be an open-source project and getting a version wouldn't be difficult, but the one in the CrashPlan Solaris installer works fine; let's get that one:

# current version when I ran this
$ wget http://download.crashplan.com/installs/solaris/install/CrashPlan/CrashPlan_2010-03-08_Solaris.tar.gz
$ tar xzf CrashPlan_2010-03-08_Solaris.tar.gz
$ cd CrashPlan/root/opt/sfw/crashplan
$ ls
bin  client-build.properties  conf  doc  installer  jniwrap.lic  lang  lib  libjtux.so  skin  upgrade

So here we have the entrails of a CrashPlan for Solaris install, and we can pick and choose which organs we need to transplant; and the thing that's particularly needed is libjtux.so; it needs to replace the Linux version.

All that should be left is getting CrashPlan to start at boot-up, which is easiest manually done in Nexenta with the init.d system. Here's the executable crashplan script I have in /etc/init.d:

#!/bin/bash
SCRIPTNAME=/usr/local/crashplan/bin/CrashPlanEngine

case "$1" in
start)
        $SCRIPTNAME start       
        ;;
stop)
        $SCRIPTNAME stop
        ;;
restart)
        $SCRIPTNAME restart
        ;;
force-reload)
        $SCRIPTNAME force-reload
        ;;
status)
        $SCRIPTNAME status
        ;;
*)      
        echo "Usage: $0 " >&2
        exit 3
        ;;
esac
exit 0

And the symlink in rc3.d:

$ ls -l /etc/rc3.d/S99crashplan 
lrwxrwxrwx 1 root root 21 Mar 10  2010 rc3.d/S99crashplan -> ../init.d/crashplan

That's it!

10 comments:

Ian said...

Hi, I tried doing this, and it tells me that it's working, but it's definitely not. What I mean is that the CrashPlanEngine wrapper says
Starting CrashPlan Engine ...
OK
but no process ever appears in top, and I can start it over and over and it doesn't know that it's already supposed to be running. Also if you stop it, it gives an "No such process" error. So it's not actually running something. I am not familiar with debugging this kind of thing at all, so any help you can give me would be great. Thanks!

Barry Kelly said...

Ian - I recommend you post your question on serverfault.com, as essentially you're trying to debug the startup of a service on Nexenta, a variant of Solaris.

Ian said...

With some help, I eventually figured it out. I was getting a libCrun.so.1 error, and it turns out that it has to do with a dependency bug in Nexenta's apt repository. sunwlibc is a required package for Sun JDK. So I ran:
# aptitude install sunwlibc
The Nexenta FAQ says it works for NCP 2, but I'm running 3.0.1, and it worked for me. Seems to be a bug which has persisted.

Thanks for the great article! Couldn't have done it without it.

Ian said...

I just noticed that CrashPlan had stopped running on the 10th (yesterday). Looking at the logs, it appeared to be that CrashPlan was self-applying the 3.0.2 update, and when it went to restart itself, it shutdown but didn't come back up. I thought I read somewhere that updating was funny, but I don't remember what exactly it said. Does this happen with your installation? Do you have any idea what might cause or remedy this?

Also, would you mind if I posted these instructions on nexenta.org's wiki? I will of course credit you and your blog. I think it would be helpful to get these instructions in a more centralized place for others who might be looking.

Barry Kelly said...

Go ahead with wiki etc., I don't mind.

My CrashPlan server stopped too. I'm not sure exactly how the upgrade is trying to do the restart, and I haven't looked into debugging it because I know CrashPlan will send me a notification if more than 3 days go by without a backup contact. Restarting it manually is just a matter of "sudo /etc/init.d/crashplan start". I've symlinked /etc/init.d/crashplan to /etc/init.d/CrashPlan in case it's simply looking for the wrong script to invoke - I seem to recall that the "proper" installation spells it that way - we'll see how it goes.

Kristan said...

Hi,

I've got the crashplan engine running and can connect from my windows machine, but whenever I try to select files to backup, I can't see my zfs volumes (mounted in /volume). Everyone has read access, I can't see any exclusions in the xml config files. Any ideas?
Thanks!

Anonymous said...

Thank you for a great writeup, I have encoutered one problem you hopefully can help me with.

When I try to extract the cpio archive, nothing happens...

jk@nexentacore:~/download/CrashPlan-install/t$ cpio -v -i ../test

(I tried with the verbose commande to get some more info, but nothing is happening)

I've let it sit there for 20 minutes, but it does'nt seem to work. The cpio command, just sits there. It does not consume any cpu power.

Any advice?

(I also tried to extract it on my linux box, but excactly the same thing happens there, nothing.

Jon said...

I googled the problem, and the solution was to issue the cpio command like this:
cpio -idmv

Jon said...

Also, I kept getting this error when trying to connect with CrashPlanDesktop to the headless install

Apr 20 22:05:03 nexentacore sshd[1201]: [ID 800047 auth.error] error: setsockopt TCP_NODELAY: Invalid argument

You can fix that by disabling ipv6 in the sshd:

Modifiy: /lib/svc/method/sshd
Find the line "/usr/lib/ssh/sshd" and change it to "/usr/lib/ssh/sshd -4"

Then edit "/etc/ssh/sshd_config" and comment
out "ListenAddress ::" and uncomment "ListenAddress 0.0.0.0"

Execute "svcadm restart ssh" and you are good to go.

Jon said...

I got everything running now, and it seems to work sort of. CrashPlan is backup up files, but it seems like it does not register new files.

I added some files this morning, 4 hours ago, but they have not been backed up. Last backup according to the CrashPlan web interface is 17 minutes ago. So it seems everything is working, except it is'nt.

It does work if I remove the backup source, and readd it. Then it finds the new files, but I cant keep doing that.

Any advice is much appreciated. Thank you!