CrashPlan for Backup on Nexenta

For some time, I've been using cron jobs with rdiff-backup on Cygwin for backups. The cron job runs on a Windows server I have on my home network and iteratively mounts remote Windows shares (if they're present) and runs rdiff-backup over them, with the destination also being local. Then the job runs rsync to mirror this backup to my Nexenta NAS running ZFS raidz.

This solution gives me a fair amount of local redundancy: two separate machines, with two copies of the backup data, plus the RAID-5-like redundancy and checksumming integrity that ZFS provides. Rdiff-backup is quite reassuring with respect to restores too: it stores files in the filesystem directly, along with differences (the rdiff bit) so you can go back in time. That means that restoring is as simple as copying the files straight out of the backup and deleting the rdiffs.

Of course, a backup strategy isn't solid without a remote copy. Today, I finished configuring CrashPlan, a really neat backup solution built using Java. You can read about the features etc. of CrashPlan on the website - I heard about it from the Java Posse podcast, episode 298. Initially, I opened an Amazon S3 account, and was considering mirroring my backups to S3 with s3sync, but after I evaluated CrashPlan, it looked like it made more sense than further pursuing my homegrown approach. Not only is it cheaper than S3 for my data (I guess they depend on overselling like ISPs), but CrashPlan has features that make it work well in my case.

The fact that it's primarily Java meant I could install it on my Nexenta box. CrashPlan don't have an installer for Nexenta, but they do have ones for Linux (LSB) and Solaris, and Nexenta is like a hybrid. I'm documenting the steps here in case I need to set it all up again, though hopefully not for a restore.

I'm running NexentaCore NCP 2, which doesn't come with Java. Installing a JDK (overkill, but I wanted to test the Java environment with a simple hello-world):

$ sudo apt-get install sun-java6-jdk

I was running a release candidate of Nexenta and I had to do an apt-get update and upgrade to resolve all necessary dependencies, but it was fairly stress free because Nexenta's apt-clone checkpointed the root file system with a ZFS snapshot. I did lose my almost half-year of uptime though - my Nexenta machine has been the most dependable of all my machines, in both hardware and software.

Installing CrashPlan itself needs bits from both the Linux and Solaris installers. I downloaded both, and unpacked both. In the Linux install, I ran the provided install.sh and followed the steps, putting it in /usr/local, with daemon init script in /etc/init.d and runlevel symlink in /etc/rc3.d. But the Linux install isn't enough. In particular, the init script assumes GNU ps, but Nexenta uses Solaris ps. So I swapped in CrashPlanEngine (the target of the init script symlink) from the Solaris installer in the place of the installed CrashPlanEngine that was here:

/usr/local/crashplan/bin/CrashPlanEngine

But that wasn't enough. CrashPlan loads a library called libjtux via JNI, a kind of POSIX API for Java programmers who want direct access to the OS. The libjtux.so from the Linux install was linking against libc.so.6, assuming GNU C library versioning. Replacing libjtux.so with the version from Solaris, linking against plain libc.so, solved this problem - here:

/usr/local/crashplan/libjtux.so

Finally, I had to configure CrashPlan. My Nexenta install is headless - all my interaction with it is either over Samba shares, HTTP for wiki servers etc., and of course ssh for everything else. Here's one of the really neat features of CrashPlan: the user interface for configuration is a client that depends on a single TCP connection with the local backup server. All I had to do is get the client to connect to a different local port, and tunnel that port to Nexenta using SSH, after enabling TCP forwarding in Nexenta in /etc/ssh/sshd_config. This bit is described on CrashPlan's site describing how to configure a headless client.

I'm running CrashPlan+, a for-pay version of the engine, on two machines - my main desktop and my Nexenta box - to get all the encryption, compression, deduplication etc. goodness. And considering that CrashPlan supports peer to peer backup, I may simply replace my existing ad-hoc rdiff-backup approach with local CrashPlan backups, as CrashPlan supports multiple destinations for backups, as well as receiving backups from other machines.

A key limitation of CrashPlan, and one that I found particularly annoying, is that the Windows client doesn't support network shares at all, in any shape or form - whether they're mapped to drive letters or not. The backup engine runs under the SYSTEM account, so it doesn't have network credentials, and also means it may not be able to access all the files you're trying to back up - especially EFS encrypted files. (CrashPlan doesn't seem to use Windows' EFS backup capability, e.g. ReadEncryptedFileRaw.) I changed the CrashPlan engine's service account to my own account to try and close this hole, but still no go on accessing network shares. This meant that I had to get CrashPlan running on my Nexenta box in order to store local backups on ZFS, should I so choose.

But apart from that, I've been impressed with CrashPlan's feature set and usability.

Multidimensional Separation of Concerns, or Matrix Inheritance

I just spotted a paper that I didn't know existed before. Abstract:

Done well, separation of concerns can provide many software engineering benefits, including reduced complexity, improved reusability, and simpler evolution. The choice of boundaries for separate concerns depends on both requirements on the system and on the kind(s) of decomposition and composition a given formalism supports. The predominant methodologies and formalisms available, however, support only orthogonal separations of concerns, along single dimensions of composition and decomposition. These characteristics lead to a number of well-known and difficult problems.

This paper describes a new paradigm for modeling and implementing software artifacts, one that permits separation of overlapping concerns along multiple dimensions of composition and decomposition. This approach addresses numerous problems throughout the software lifecycle in achieving well-engineered, evolvable, flexible software artifacts and traceability across artifacts.

I came up with a remarkably similar concept I called matrix inheritance in a previous job. Here's one description I've written up out there on the web:

[This is a description of a symptom of the problem.] Nested if-then-else can introduce non-essential complexity merely by having dead nested cases which are actually obviated by outer cases; looking at a complex version of this code, it can become quite difficult to see where exactly the code is supposed to flow under what circumstances, as there can seem to be conflicting assumptions in different areas.

Once upon a time I invented a scheme to solve this problem in the best way I thought possible, and called it "matrix inheritance". The problem with inheritance and subclassing is that it only handles a single dimension of customization. Suppose you have two dimensions, genre and type, such as [comedy, drama] and [movie, series]. If you were to try and classify any given thing under a classical type breakdown, you could subclass by one axes or the other, but you would need to duplicate subclassing for the remaining axes. So, you could end up with Programme, Comedy <: Programme, Drama <: Programme, but then you'd need ComedyMovie, ComedySeries, DramaMovie, DramaSeries, duplicating the kind axis in the two different branches.

The matrix inheritance concept basically takes the cartesian product of all option axes, essentially modelling it as an n-dimensional space, and then applies behaviour to sub-spaces of this space. So, you could apply conditions to [drama,*] and [*, series], with these two representing slices of the example 2-dimensional space described above. The advantage of modelling things this way is that it is declarative: you can analyse overlaps and identify un-covered space.

Here's the original post I made on comp.compilers when looking for related work. Here's a brief excerpt:

Object-oriented techniques have been rejected because they constrain me to using a fixed order of dimensions for reuse and overriding. A possible solution presents itself:

1. Creating a notional Cartesian-product of all dimensions of customization.

2. Overriding based on the application of rules whose scope is defined by sets described by a simple language: by example, '{a,b}' represents the set containing a and b, while '*' represents the set containing all the values for that dimension.

One-liner RAII in Delphi

Some C++ aficionados point at the ability, in C++, to create an object such that the creation allocates some kind of resource, and the destruction frees the resource, with much glee. The idiom is RAII: Resource Acquisition Is Initialization. The technique can be a useful, albeit somewhat opaque, way of acquiring a resource for the duration of a block of code and relinquishing the resource upon exiting the block. It's C++'s equivalent of Delphi / Java / C# / SEH's try / finally blocks (though of course C#'s "using" statement is an even closer analogue).

Delphi has a similar mechanism in the form of deterministic reference counting for interfaces. This can be (ab)used to functionally implement almost exactly the same technique as C++. The degree to which it is useful, however, is moderated by readability issues; RAII doesn't necessarily employ a visible nesting, and rather depends on the implied nesting of the scope in which it's used.

Anyway, here's a way of implementing RAII in Delphi, for people to whom the technique may not have occurred yet. It relies on defining a general-purpose class which implements an interface, and whose death will be triggered by scope exit:

type
  TScopeExitNotifier = class(TInterfacedObject)
  private
    FProc: TProc;
  public
    constructor Create(const AProc: TProc);
    destructor Destroy; override;
  end;

constructor TScopeExitNotifier.Create(const AProc: TProc);
begin
  FProc := AProc;
end;

destructor TScopeExitNotifier.Destroy;
begin
  if Assigned(FProc) then
    FProc;
  inherited;
end;

function MakeScopeExitNotifier(const AProc: TProc): IInterface;
begin
  Result := TScopeExitNotifier.Create(AProc);
end;

I would generally recommend hiding the actual class in the implementation-section of a unit - the functional interface of MakeScopeExitNotifier ought to be sufficient. Using this function, we can now implement a one-liner RAII equivalent of the following code:

  Writeln('Disable controls on ', {ASurface});
  try
    // (user code)
  finally
    Writeln('Enable controls on ', {ASurface});
  end;

Of course, I'm using strings etc. for simplicity, so that the demo translation is compilable. Here is the definition of the function which will be called to form the one-liner:

function DisableEnableControls(const ASurface: string): IInterface;
begin
  Writeln('Disable controls on ', ASurface);
  Result := MakeScopeExitNotifier(procedure
  begin
    Writeln('Enable controls on ', ASurface);
  end);
end;

And here it is in action:

procedure P;
begin
  DisableEnableControls('Foo');
  DisableEnableControls('Bar');
  Writeln('Controls on Foo and Bar are disabled now');
end;

Because of the way Delphi's deterministic reference counting and destruction works, Foo and Bar will be released in reverse order of construction, as expected for RAII. Functions that return an interface value in Delphi are implemented behind the scenes as procedures that take an out Result parameter; these are in effect hidden variables that keep the return values alive until the scope is exited.

The output of the program is as follows:

Disable controls on Foo
Disable controls on Bar
Controls on Foo and Bar are disabled now
Enable controls on Bar
Enable controls on Foo

Of course, it's not necessarily clear what DisableEnableControls(string) would do, and how it protects the scope. That's why I wouldn't necessarily recommend using this technique, except to counter C++-ite arguments. Using anonymous methods directly would seem to be a more readable approach in general, something along these lines:

procedure DisableEnableControls(const ASurface: string; const ACB: TProc);
begin
  Writeln('Disable controls on ', ASurface);
  try
    if Assigned(ACB) then
      ACB;
  finally
    Writeln('Enable controls on ', ASurface);
  end;
end;

procedure P;
begin
  DisableEnableControls('Foo', procedure
  begin
    DisableEnableControls('Bar', procedure
    begin
      Writeln('Controls are disabled now.');
    end);
  end);
end;

Output is the same as earlier. The extra complexity / verbosity of this abstraction over and above a literal try/finally would of course be more justified if the logic in the place of Enable and Disable sections were more involved, such as possibly setting up and rolling back transactions, etc.

However, the functionality of MakeScopeExitNotifier() routine is actually useful in the context of anonymous methods, because by capturing a variable containing the return value, you can get informed as to when the scope that created your anonymous method dies. This can be useful to e.g. free any objects that were allocated when the anonymous method was created:

function FreeOnExit(const Args: array of TObject): IInterface;
var
  toFree: TArray<TObject>;
begin
  // copy open array to toFree because open arrays can't be captured
  SetLength(toFree, Length(Args));
  Move(Args[0], toFree[0], Length(Args) * SizeOf(Args[0]));
  
  Result := MakeScopeExitNotifier(procedure
  var
    i: Integer;
  begin
    Writeln('FreeOnExit is freeing objects');
    for i := 0 to High(toFree) do
      toFree[i].Free;
  end);
end;

procedure Use(const x: IInterface);
begin
end;

function Q: TProc;
var
  onDie: IInterface;
  obj: TObject;
begin
  obj := nil;
  onDie := FreeOnExit([obj]);
  Writeln('Allocating an object');
  obj := TObject.Create;
  Result := procedure
  begin
    Use(onDie); // cause onDie to be captured
    Writeln(Format(
      'Inside callback, obj still alive: %p: %s',
      [Pointer(obj), obj.ToString]));
  end;
end;

procedure P;
var
  r: TProc;
begin
  Writeln('Before Q');
  r := Q();
  Writeln('Invoking result of Q');
  r;
  Writeln('Clearing result of Q');
  r := nil;
  Writeln('Result is cleared');
end;

The output is something like:

Before Q
Allocating an object
Invoking result of Q
Inside callback, obj still alive: 00970D20: TObject
Clearing result of Q
FreeOnExit is freeing objects
Result is cleared

This technique, capturing a variable of an interface or other deterministically lifetime-managed type, is essential for flexibility in cleaning up any resources that an anonymous method needs to for its lifetime. Note, however, that there can be a gotcha: the following alternative, using MakeScopeExitNotifier directly, will not work:

// <BROKEN CODE DO NOT USE>
function Q: TProc;
var
  onDie: IInterface;
begin
  onDie := MakeScopeExitNotifier(procedure
  begin
    Writeln('died!');
  end);
  Result := procedure
  begin
    Use(onDie);
  end;
end;
// </BROKEN CODE DO NOT USE>

The reason it's broken is that the anonymous method that's handed off to MakeScopeExitNotifier will itself keep the scope alive. So in order to make use of MakeScopeExitNotifier for this particular purpose, the anonymous method passed to it needs to be in a different scope.

Using anonymous methods in method pointers

Anonymous methods may have associated state. In particular, all variables that an anonymous method captures need to be kept alive so long as the anonymous method is callable. For this reason, anonymous methods are implemented with a lifetime management approach: anonymous methods are actually methods on an object which implements COM-style reference counted interfaces. Method references are interface references with a single method called Invoke.

If one badly wants to store an anonymous method in a method pointer, this information can be used to shoehorn it in. Here's how it can be done:

procedure MethRefToMethPtr(const MethRef; var MethPtr);
type
  TVtable = array[0..3] of Pointer;
  PVtable = ^TVtable;
  PPVtable = ^PVtable;
begin
  // 3 is offset of Invoke, after QI, AddRef, Release
  TMethod(MethPtr).Code := PPVtable(MethRef)^^[3];
  TMethod(MethPtr).Data := Pointer(MethRef);
end;

This procedure will take a method reference as the first argument, and then extract the two crucial pieces of data needed to put into a method pointer: the value to be passed as the first argument (the method, or interface, reference), and the code address to be called (the value of the Invoke entry in the interface's vtable).

The procedure can be used as follows. Note that the method reference still needs to be kept alive somewhere as long as the method pointer is callable to avoid premature disposal of the heap-allocated object:

type
  TMeth = procedure(x: Integer) of object;
  TMethRef = reference to procedure(x: Integer);
  
function MakeMethRef: TMethRef;
var
  y: Integer;
begin
  y := 30;
  Result := procedure(x: Integer)
  begin
    Writeln('x = ', x, ' and y = ', y);
  end;
end;

procedure P;
var
  x: TMethRef;
  m: TMeth;
begin
  x := MakeMethRef();
  MethRefToMethPtr(x, m);
  Writeln('Using x:');
  x(10);
  Writeln('Using m:');
  m(10);
end;

On the other hand, if the anonymous method's body never captures any variables, then it's not necessary to keep the method reference around. This loses much of the benefits of anonymous methods, but it does prove the point:

uses SysUtils, Forms, StdCtrls, Dialogs, classes;

procedure MethRefToMethPtr(const MethRef; var MethPtr);
type
  TVtable = array[0..3] of Pointer;
  PVtable = ^TVtable;
  PPVtable = ^PVtable;
begin
  // 3 is offset of Invoke, after QI, AddRef, Release
  TMethod(MethPtr).Code := PPVtable(MethRef)^^[3];
  TMethod(MethPtr).Data := Pointer(MethRef);
end;

type
  TNotifyRef = reference to procedure(Sender: TObject);

function MakeNotify(const ANotifyRef: TNotifyRef): TNotifyEvent;
begin
  MethRefToMethPtr(ANotifyRef, Result);
end;

procedure P;
var
  f: TForm;
  btn: TButton;
begin
  f := TForm.Create(nil);
  try
    btn := TButton.Create(f);
    btn.Parent := f;
    btn.Caption := 'Click Me!';
    btn.OnClick := MakeNotify(procedure(Sender: TObject)
      begin
        ShowMessage('Hello There!');
      end);
    f.ShowModal;
  finally
    f.Free;
  end;
end;

begin
  try
    P;
  except
    on e: Exception do
      ShowMessage(e.Message);
  end;
end.

Delphi 2010 RTTI Contexts: how they work, and a usage note

Delphi 2010 includes extended support for RTTI, also known as run-time type info or reflection. Many design approaches that have previously only been possible with managed languages like C# and Java because of the code annotation and introspection they required should now be possible in the Delphi world.

Something somewhat interesting about how the RTTI works is its approach to object pools. Delphi isn't a garbage collected language, so users need to be careful to free objects when they're no longer needed, either explicitly, or by designing or using some kind of ownership scheme, such as that used by TComponent, where the Owner takes care of destruction.

Type information usage scenarios don't mesh particularly well with a TComponent-style of ownership. Typically, when working with RTTI, you want to do some kind of search for interesting objects, do something with them, and then go on your way. That means that many objects may get allocated for inspection, but not actually be used. Managing those objects' lifetimes independently would be tedious, so a different approach is used: there is a single global RTTI object pool. While there is at least one RTTI context active in the application, this object pool keeps all its objects alive. When the last context goes out of scope, the objects get freed up.

The pool management works by using a Delphi record that contains an interface reference. The first time any given RTTI context is used, it fills in this interface reference. It can't fill it in any later than first use, because Delphi records don't support default constructors, which besides have their own problems. For example, how do you handle exceptions in default constructors, in all the places they can occur? Allocating arrays, thread-local variables, global variables, global variables in packages, temporary objects created in expressions, etc. It can get ugly, and in C++ it sometimes does.

So, this first use allocates an interface, called a pool token. It acts as a reference-counted handle to the global object pool. For so long as this interface is alive, the the global object pool should stay alive. Even if the RTTI context is copied around, Delphi's built-in interface handling logic, designed along COM principles, will ensure that the interface doesn't gets disposed of prematurely or get its reference count muddled up. And when an RTTI context goes out of scope, either by being a local variable in a function that is exited, or a field in an object that is freed, the reference count is reduced. When it hits zero, the pool is emptied.

The biggest upside of this approach is that RTTI usage should feel reasonably cheap, conceptually speaking. Code need only declare a variable of the appropriate type, and start using it:

procedure Foo;
var
  ctx: TRttiContext;
  t: TRttiType;
begin
  t := ctx.GetType(TypeInfo(Integer));
  Writeln(t.Name);
end;

A downside, however, is that lazy initialization can create a gotcha. Imagine this scenario:

1. Library A declares an RTTI context A.C
2. User code B declares an RTTI context B.C
3. B pulls some RTTI objects O out of B.C, in order to hand them to library A
4. B.C goes out of scope
5. Library A now tries to work with O, but discovers much to its surprise, that the objects have been prematurely disposed, even though A already has an RTTI context, A.C

The problem is that A never used A.C, so it never allocated a pool token. When B.C used its context, the pool came into being, and objects O were assigned to it; but after B.C went out of scope, the objects got freed.

The solution to this problem is for Library A, knowing that it has a long-lived RTTI context and it expects to communicate with third-party code which allocates objects from its own RTTI context and hands them back, it should ensure that the long-lived context's pool token is allocated. A trivial way to do this is like this:

type
  TFooManager = class
    FCtx: TRttiContext;
    // ...
    constructor Create;
    // ...
  end;

constructor TFooManager.Create;
begin
  FCtx.GetType(TypeInfo(Integer));
  // ...
end;

This will allocate only a bare minimum of RTTI objects, those needed to represent the type System.Integer, but more importantly, will ensure that FCtx has a pool token and will keep the global RTTI pool alive.

In future releases of Delphi, the static method TRttiContext.Create will make sure that its return value has a pool token allocated; currently, it does not. TRttiContext.Create was originally defined to make the TRttiContext record feel more like a class for people unfamiliar with the idiom of using interfaces for automated deterministic lifetime management. The corresponding TRttiContext.Free method disposes of the internal pool token, and should remain the same.