Monday, December 20, 2010

Scrolling: Chrome vs Firefox

I was using the Chrome browser on my laptop the other day (some browser compat problem). I don't normally use it; Firefox is my preferred browser. I couldn't help but notice how peculiarly laggy it felt to use. It came down to vertical scrolling: this highly common task - for any long article you'll be doing a lot of it - felt jerky and unpleasant. So I did a quick ad-hoc experiment: open up the same web page in both browsers with viewports of the same size, grab the vertical scroll bar, and wiggle it up and and down continuously so the browser is constantly redrawing text.

This is a crop from Process monitor on my desktop machine, which has 8 logical cores, i.e. 4 physical cores with hyperthreading. That means that 100% CPU usage on one core shows up as 12.5% CPU usage. The big wide bump on the left is the CPU usage while I was fiddling with Chrome; the one on the right is Firefox. (The bump in the middle is sqlservr.exe, which wakes up and does meaningless busywork every 60 seconds.) You can see that Chrome uses perhaps 50% more CPU usage on this task than Firefox, with more time spent in the user process and (proportionately) less in the kernel (presumably shifting bits around). Eyeballing an average of CPU usage, Firefox ranged from 6% to 7.5%, while Chrome pegged at 12.5%. Chrome is simply less CPU-efficient redrawing while scrolling, and it's very obvious to my eyes. (I did the same experiment with IE, and it was in the middle, with about 9% CPU.)

I might hazard a guess that Firefox spends more memory on caching bitmaps of the web page, or some similar trick trading off space for time. In any case, it's one of the reasons Firefox is still my primary browser, and also why I'm completely unconcerned about its memory usage. I have yet to encounter significant paging because of Firefox, not least because it's a 32-bit process, limiting its maximum usage, and I have alternately 12G and 4G of memory in my primary desktop and laptop respectively.

Mind you, when I add up the private working set for all 8 chrome.exe processes apparently needed to display this web page, they add up to 86M, only 1M less than my recently restarted firefox.exe session at 87M. (Comparing other memory usage numbers is awkward, as non-working-set memory isn't relevant, while non-private memory would be double-counted with Chrome.)

Monday, November 15, 2010

CrashPlan manual installation approach on Nexenta

I wrote an earlier post about installing CrashPlan on Nexenta. However, it seems that CrashPlan have changed their Linux installer's modus operandi, and it tries to install a (Linux) JRE of its own. Of course, that won't work; it needs the Nexenta JRE (but I get the whole JDK, as you never know when you'll need to brew up some Java):

$ sudo apt-get install sun-java6-jdk

Anyhow, I received some comments and emails about setting all this up, and figured I'd break it down a little more for people who still want to get it all working. It's a process of hacking things together, though, not a blind recipe; that's why I'm proceeding almost as if it's a debugging session.

Anyhow, once we have a JRE installed (from apt-get above), we can try and extract out the guts of the Linux installer, so rather than running its install.sh, we can set it up manually.

I went to the CrashPlan Linux download web page, and started downloading the installer in a scratch folder (this was the URL at the time of writing):

$ wget http://download.crashplan.com/installs/linux/install/CrashPlan/CrashPlan_2010-03-08_Linux.tgz

That's a gzipped tar archive, so I extracted it:

$ tar xzf CrashPlan_2010-03-08_Linux.tgz
$ cd CrashPlan-install # the dir it extracted
$ ls
CrashPlan_2010-03-08.cpi  EULA.txt  INSTALL  README  install.sh  scripts  uninstall.sh

I didn't know what the big CrashPlan*.cpi file was, so I checked:

$ file CrashPlan_2010-03-08.cpi
CrashPlan_2010-03-08.cpi: gzip compressed data, was "CrashPlan_2010-03-08.cpi", from Unix, max compression

So it's a gzipped file! I decompressed it, and tested the result:

$ gunzip < CrashPlan_2010-03-08.cpi > test
$ file test
test: ASCII cpio archive (pre-SVR4 or odc)

A cpio archive! I extracted that too, but in a separate directory:

$ mkdir t; cd t; cpio -i ../test
$ ls
bin  conf  doc  jniwrap.lic  lang  lib  libjniwrap.so  libjniwrap64.so  libjtux.so  libjtux64.so  log  skin  upgrade

Now this looks very close to the root listing of my actual current CrashPlan install on Nexenta, and by and large it is. Here's a rough difference between the two (listing.curr is my actual install, listing is extracted from the installer):

--- listing.curr        2010-11-15 06:32:12.548701734 +0000
+++ listing     2010-11-15 06:31:40.726884007 +0000
-CrashPlanEngine.pid
-bin/CrashPlanDesktop
-bin/CrashPlanEngine
-bin/CrashPlanEngine.lsb
-bin/run.conf
-bin/vars.sh
-conf/my.service.xml
-conf/service.login
-conf/service.model
-install.vars
-libjtux.so.lsb
-libjtux.so.sol

Let's go through those. This install tree is almost complete; it can be put in /usr/local/crashplan (that's where I have mine) or wherever you like, so long as the configuration bits are also hooked up appropriately.

CrashPlanEngine.pid is just the pidfile, a text file containing the process id of the currently running instance. I'm not sure why it's in the crashplan directory rather than somewhere like /var/run, but it is. I believe the crashplan service will create it.

CrashPlan* are all shell wrappers for Java applications; *.lsb (LSB for Linux Standard Base) are the original versions (I think I renamed them to this). I don't have an X server or libraries on my Nexenta install, so I'm uninterested in CrashPlanDesktop. CrashPlanEngine however is important; it's the main daemon file. Here's mine, which seems to work; I think I may have edited it to work with some GNU utils rather than Solaris utils, as I believe this came from the Solaris installer (or vice versa; actually that's probably more likely):

#!/bin/bash

TARGETDIR="`dirname ${0}`/.."

. ${TARGETDIR}/install.vars
. ${TARGETDIR}/bin/run.conf

cd ${TARGETDIR}

case $1 in
        start)
                PID=`/usr/bin/ps -Af -o pid,ppid,args | grep 'app=CrashPlanService' | grep -v grep | cut -f2 -d' '`
                if [ -n "$PID" ]; then
                  echo CrashPlan is already running with pid $PID
                  exit 1;
                fi
                echo "Starting CrashPlan Engine ... "
                nice -n 19 ${JAVACOMMON} ${SRV_JAVA_OPTS} -classpath "./lib/com.backup42.desktop.jar:./lang" com.backup42.service.CPService > ${TARGETDIR}/log/engine_output.log 2> ${TARGETDIR}/log/engine_error.log & 
                if [ $! -gt 0 ]; then
                        echo $! > ${TARGETDIR}/CrashPlanEngine.pid
                        echo "OK"
                else
                        echo "FAIL" 
                        exit 1
                fi
                ;;
        stop)
                echo "Stopping CrashPlan Engine ... "
                if [ -f ${TARGETDIR}/CrashPlanEngine.pid ] ; then
                  kill `cat ${TARGETDIR}/CrashPlanEngine.pid`
                  sleep 5
                fi
                PID=`/usr/bin/ps -Af -o pid,ppid,args | grep 'app=CrashPlanService' | grep -v grep | cut -f2 -d' '`
                if [ -n "$PID" ]; then
                  echo Still running, killing PID=$PID
                  kill -9 $PID
                fi
                rm -f ${TARGETDIR}/CrashPlanEngine.pid
                echo "OK"
                ;;
        *)      
                echo "$0 "
                exit 1
                ;;
esac

As you can see, that script sources (includes) a couple of other guys, bin/run.conf and install.vars. bin/run.conf looks like this:

SRV_JAVA_OPTS="-Dfile.encoding=UTF-8 -Dapp=CrashPlanService -DappBaseName=CrashPlan -Xms20m -Xmx512m -Dsun.net.inetaddr.ttl=300 -Dnetworkaddress.cache.ttl=300 -Dsun.net.inetaddr.negative.ttl=0 -Dnetworkaddress.cache.negative.ttl=0"
GUI_JAVA_OPTS="-Dfile.encoding=UTF-8 -Dapp=CrashPlanDesktop -DappBaseName=CrashPlan -Xms20m -Xmx512m -Dsun.net.inetaddr.ttl=300 -Dnetworkaddress.cache.ttl=300 -Dsun.net.inetaddr.negative.ttl=0 -Dnetworkaddress.cache.negative.ttl=0"

install.vars looks like this (for my current install):

TARGETDIR=/usr/local/crashplan
BINSDIR=/usr/local/bin
MANIFESTDIR=/tank/share/backup/crashplan
INITDIR=/etc/init.d
RUNLVLDIR=/etc/rc3.d
INSTALLDATE=20100310
APP_BASENAME=CrashPlan
DIR_BASENAME=crashplan
DOWNLOAD_HOST=download.crashplan.com
JAVACOMMON=/usr/bin/java

The MANIFESTDIR variable points to a directory I expect I provided when I ran the original install.sh, but there's nothing currently in it. The other paths are as you'd expect.

As far as I can make out, the conf/* files are created by CrashPlan itself in coordination with the configuration utility, which, as I linked in my previous post, can be run remotely from a Windows (or other OS) install of CrashPlan. There's a default.service.xml as a prototypical my.service.xml, while service.{login,model} appear to be login and encryption keys.

That leaves libjtux.so.*. These are renamed versions of the file libjtux.so which differs between Linux and Solaris. Primarily they differ in which C library they link against (note: running ldd on untrusted binaries is unsafe, but I trust these binaries):

$ ldd libjtux.so.*
warning: ldd: libjtux.so.lsb: is not executable
libjtux.so.lsb:
        librt.so.1 =>    /lib/librt.so.1
        libnsl.so.1 =>   /lib/libnsl.so.1
        libc.so.6 =>     (file not found)
        libmp.so.2 =>    /lib/libmp.so.2
        libmd.so.1 =>    /lib/libmd.so.1
        libscf.so.1 =>   /lib/libscf.so.1
        libc.so.1 =>     /lib/libc.so.1
        libuutil.so.1 =>         /lib/libuutil.so.1
        libgen.so.1 =>   /lib/libgen.so.1
        libm.so.2 =>     /lib/libm.so.2
warning: ldd: libjtux.so.sol: is not executable
libjtux.so.sol:
        librt.so.1 =>    /lib/librt.so.1
        libsocket.so.1 =>        /lib/libsocket.so.1
        libnsl.so.1 =>   /lib/libnsl.so.1
        libc.so.1 =>     /lib/libc.so.1
        libmp.so.2 =>    /lib/libmp.so.2
        libmd.so.1 =>    /lib/libmd.so.1
        libscf.so.1 =>   /lib/libscf.so.1
        libuutil.so.1 =>         /lib/libuutil.so.1
        libgen.so.1 =>   /lib/libgen.so.1
        libm.so.2 =>     /lib/libm.so.2

Anyhow, we need to get the right version of libjtux.so. It seems to be an open-source project and getting a version wouldn't be difficult, but the one in the CrashPlan Solaris installer works fine; let's get that one:

# current version when I ran this
$ wget http://download.crashplan.com/installs/solaris/install/CrashPlan/CrashPlan_2010-03-08_Solaris.tar.gz
$ tar xzf CrashPlan_2010-03-08_Solaris.tar.gz
$ cd CrashPlan/root/opt/sfw/crashplan
$ ls
bin  client-build.properties  conf  doc  installer  jniwrap.lic  lang  lib  libjtux.so  skin  upgrade

So here we have the entrails of a CrashPlan for Solaris install, and we can pick and choose which organs we need to transplant; and the thing that's particularly needed is libjtux.so; it needs to replace the Linux version.

All that should be left is getting CrashPlan to start at boot-up, which is easiest manually done in Nexenta with the init.d system. Here's the executable crashplan script I have in /etc/init.d:

#!/bin/bash
SCRIPTNAME=/usr/local/crashplan/bin/CrashPlanEngine

case "$1" in
start)
        $SCRIPTNAME start       
        ;;
stop)
        $SCRIPTNAME stop
        ;;
restart)
        $SCRIPTNAME restart
        ;;
force-reload)
        $SCRIPTNAME force-reload
        ;;
status)
        $SCRIPTNAME status
        ;;
*)      
        echo "Usage: $0 " >&2
        exit 3
        ;;
esac
exit 0

And the symlink in rc3.d:

$ ls -l /etc/rc3.d/S99crashplan 
lrwxrwxrwx 1 root root 21 Mar 10  2010 rc3.d/S99crashplan -> ../init.d/crashplan

That's it!

Wednesday, September 15, 2010

Non-Delphi: Postmodernism, Transformers: ROTF and Baudrillard

The simulacrum is never that which conceals the truth--it is the truth which conceals that there is none. The simulacrum is true. (Baudrillard, Simulacra and Simulation)

I've just watched Transformers: Revenge of the Fallen, and it got me thinking. It's perhaps the most post-modern movie I've seen to date.

T:ROTF isn't so much a movie as a string of soundbites, stereotypes and cliches arranged into a 2.5 hour trailer for a movie you'll never get to see, because it was never made. You turn up for the product the advertisement is selling, but it turns out the whole product was ad. There's no "there" there. I've never seen a better embodiment of Baudrillard's conception of simulacrum than this.

The meaningless symbols are so densely packed in this movie I need a convention. Everything in the movie is a stereotype or cliche: a stand-in, an intellectually lazy shorthand reference. I'll be lazy too, and mark stereotypes in my commentary with [brackets].

The beginning is indistinguishable from a trailer. A brief [dawn of man] scene, you know the kind, [stone age people silhouetted against a dawn sky somewhere in Africa], doing [caveman things with the spears and the facepaint], with a [rumbling voiceover] helpfully telling you that it's ["Earth, birthplace of the human race"]. If it were storytelling, it would be rushed, heavy-handed, and contemptuous of the viewer - both showing and telling. But I don't think it is storytelling. It's arranging some symbols (humans, decepticons) into a particular aspect required for later symbolic purposes. The decepticons portrayed in this ancient time are [evil] (with [King Kong-like grabbing] of a feeble human, albeit male), but there is no motive, no narrative. Why would such powerful machines pay any more attention to stone age humans than they would apes, or insects, which they can swat away with similar ease?

Next up: Shanghai, [disaster scene], with [disaster radio news chatter]. Cue [Pentagon command centre], explaining that some black hawks are moving in, while showing some black hawks moving in: Americans aircraft and troops entering Chinese territory, in complete suspension of geopolitical disbelief, no explanation considered necessary. Expository trailer voiceover says "new autobots", while expository camera shot shows new autobots, including [hot girl on bike], [fast car], and [military transport]. "Together, we form an alliance", explains voiceover, while showing human troops in [military transport] (which subsequently transforms). No attempt to explain why squishy soldiers with small arms are going up against fast-moving heavy machinery. What do they hope to achieve with their flying pieces of lead? Would they go up against even a human-engineered tank with such miserable munitions? Nor an explanation for the gunships flying with mere tens of feet clearance from the ground and the surrounding buildings that tower over them, completely negating the tactical advantages of a mobile, hovering cannon and missile platform.

But all is soon revealed. The squishy humans aren't going in to fight, they are going in to be squished, to symbolize human weakness against the machines. After a decepticon slams its fists into some concrete pipe sections, somehow creating a fiery explosion, gunships capable of engaging the enemy with missiles and canons from considerable distance approach low and close enough to be clobbered with a mere wave of mechanical arms. As an alleged depiction of a military engagement, it's beyond ludicrous, laughable on its face. Suspension of disbelief isn't possible: this isn't a battle; it isn't even a simulation of a battle. It's a simulation of battle simulation, an arrangement of symbols of battles. Here are our valiant heroes going into battle; here's our shockingly powerful foe, see how he easily puts our heroes on the back foot; but wait (!) here come our heroes again with reinforcements, to win the day with a bunch of soundbites: ["damn, I'm good!"], ["punk-ass decepticon"], ["any last words?"], "the fallen shall rise again", ["that doesn't sound good"], ["not today!", reload-click, bullet to the head].

That's just the first 8 minutes or so; it goes on for hours (!), with no variation in pacing that you wouldn't also expect in a 30-second movie trailer. Some other commentary roughly concurs with mine, though I didn't enjoy the spectacle or visual feast aspects, primarily because those spectacles are filmed too close to the action, and the subjects, transformed machines, have so many bits and bobs hanging out of them it's hard to tell where one begins and another ends, much like how camouflage breaks up outlines. Trying to figure out what's actually going on within the pace of the editing cuts would give me a headache. Besides, marvelling at the sheer density of signifiers and its generally jaw-dropping empty awfulness is more fun, in a perverse way.

Wednesday, September 01, 2010

Virtual method interception

Delphi XE has a new type in Rtti.pas called TVirtualMethodInterceptor. It was originally designed for use in DataSnap authentication scenarios (though I don't think it's currently being used there), but of course, making it only work for that would have been quite a limitation.

What does it do? Essentially, it creates a derived metaclass dynamically at runtime that overrides every virtual method in the ancestor, by creating a new virtual method table and populating it with stubs that intercepts calls and arguments. When the metaclass reference for any instance of the "ancestor" is replaced with this new metaclass, the user can then intercept virtual function calls, change arguments on the fly, change the return value, intercept and suppress exceptions or raise new exceptions, or entirely replace calling the underlying method. In concept, it's somewhat similar to dynamic proxies from .NET and Java. It's like being able to derive from a class at runtime, override methods (but not add new instance fields), and then change the runtime type of an instance to this new derived class.

Why would you want to do this? Two obvious purposes spring to mind: testing and remoting. Mock objects have been in vogue in the testing space in other languages for some time. By intercepting method calls, one may more easily verify that a particular subsystem is calling all the right methods, with the correct arguments, in the expected order; similarly, the subsystem can proceed with the return values from these method calls, without necessarily having to hit the database, the network, etc. for what should be a unit test. Remoting on the basis of method calls is somewhat less useful, especially when an unreliable and latency-prone network gets into the stack, but that's not the only usage point. The virtual method interceptor logic was originally implemented to be used as part of DataSnap authentication, so that a call that comes in from the network can still be checked as its code flow spreads throughout the graph of objects.

Anyhow, here's a simple example to get started:

uses SysUtils, Rtti;
{$apptype console}
type
  TFoo = class
    // Frob doubles x and returns the new x + 10
    function Frob(var x: Integer): Integer; virtual;
  end;

function TFoo.Frob(var x: Integer): Integer;
begin
  x := x * 2;
  Result := x + 10;
end;

procedure WorkWithFoo(Foo: TFoo);
var
  a, b: Integer;
begin
  a := 10;
  Writeln('  before: a = ', a);
  try
    b := Foo.Frob(a);
    Writeln('  Result = ', b);
    Writeln('  after:  a = ', a);
  except
    on e: Exception do
      Writeln('  Exception: ', e.ClassName);
  end;
end;

procedure P;
var
  foo: TFoo;
  vmi: TVirtualMethodInterceptor;
begin
  vmi := nil;
  foo := TFoo.Create;
  try
    Writeln('Before hackery:');
    WorkWithFoo(foo);
    
    vmi := TVirtualMethodInterceptor.Create(foo.ClassType);
    
    vmi.OnBefore := procedure(Instance: TObject; Method: TRttiMethod;
      const Args: TArray<TValue>; out DoInvoke: Boolean; out Result: TValue)
    var
      i: Integer;
    begin
      Write('[before] Calling ', Method.Name, ' with args: ');
      for i := 0 to Length(Args) - 1 do
        Write(Args[i].ToString, ' ');
      Writeln;
    end;
    
    // Change foo's metaclass pointer to our new dynamically derived
    // and intercepted descendant
    vmi.Proxify(foo);
    
    Writeln('After interception:');
    WorkWithFoo(foo);
  finally
    foo.Free;
    vmi.Free;
  end;
end;

begin
  P;
end.

Here's what it outputs:

Before hackery:
  before: a = 10
  Result = 30
  after:  a = 20
After interception:
  before: a = 10
[before] Calling Frob with args: 10 
  Result = 30
  after:  a = 20
[before] Calling BeforeDestruction with args: 
[before] Calling FreeInstance with args: 

You'll notice that it intercepts all the virtual methods, including those called during destruction, not just the one I declared. (The destructor itself is not included.)

We can get more ambitious with what it does. I can change the implementation entirely, and skip calling the underlying (i.e. inherited) method body:

procedure P;
var
  foo: TFoo;
  vmi: TVirtualMethodInterceptor;
  ctx: TRttiContext;
  m: TRttiMethod;
begin
  vmi := nil;
  foo := TFoo.Create;
  try
    Writeln('Before hackery:');
    WorkWithFoo(foo);
    
    vmi := TVirtualMethodInterceptor.Create(foo.ClassType);
    
    m := ctx.GetType(TFoo).GetMethod('Frob');
    vmi.OnBefore := procedure(Instance: TObject; Method: TRttiMethod;
      const Args: TArray; out DoInvoke: Boolean; out Result: TValue)
    begin
      if Method = m then
      begin
        DoInvoke := False;
        Result := 42;
        Args[0] := -Args[0].AsInteger;
      end;
    end;

Here, I inhibit the invocation and hard-code the result to 42, while negating the first argument. The proof is in the output:

Before hackery:
  before: a = 10
  Result = 30
  after:  a = 20
After interception:
  before: a = 10
  Result = 42
  after:  a = -10

I could have inhibited the call by raising an exception instead:

    vmi.OnBefore := procedure(Instance: TObject; Method: TRttiMethod;
      const Args: TArray; out DoInvoke: Boolean; out Result: TValue)
    begin
      if Method = m then
        raise Exception.Create('Aborting');
    end;

And output:

Before hackery:
  before: a = 10
  Result = 30
  after:  a = 20
After interception:
  before: a = 10
  Exception: Exception

It's not limited to interception before the logically inherited call, but also interception after the call, again with the opportunity to fiddle with arguments and return value:

    m := ctx.GetType(TFoo).GetMethod('Frob');
    vmi.OnAfter := procedure(Instance: TObject; Method: TRttiMethod;
      const Args: TArray; var Result: TValue)
    begin
      if Method = m then
        Result := Result.AsInteger + 1000000;
    end;

And output:

Before hackery:
  before: a = 10
  Result = 30
  after:  a = 20
After interception:
  before: a = 10
  Result = 1000030
  after:  a = 20

And if the inherited implementation raises an exception, that can even be squashed:

function TFoo.Frob(var x: Integer): Integer;
begin
  raise Exception.Create('Abort');
end;

// ...
    m := ctx.GetType(TFoo).GetMethod('Frob');
    vmi.OnException := procedure(Instance: TObject; Method: TRttiMethod;
      const Args: TArray; out RaiseException: Boolean;
      TheException: Exception; out Result: TValue)
    begin
      if Method = m then
      begin
        RaiseException := False;
        Args[0] := Args[0].AsInteger * 2;
        Result := Args[0].AsInteger + 10;
      end;
    end;

Output:

Before hackery:
  before: a = 10
  Exception: Exception
After interception:
  before: a = 10
  Result = 30
  after:  a = 20

One thing the TVirtualMethodInterceptor class doesn't have, however, is a way to unhook (unproxify) the object. If the object is never unhooked, it's important that the object doesn't outlive the interceptor, because the interceptor needs to allocate executable memory in order to create the little stubs with which it redirects method calls to the events. Fortunately, it's pretty trivial to do:

    PPointer(foo)^ := vmi.OriginalClass;

Another point: the class inheritance chain is changed by the hooking process. This can be shown easily:

//...
    Writeln('After interception:');
    WorkWithFoo(foo);
    
    Writeln('Inheritance chain while intercepted:');
    cls := foo.ClassType;
    while cls <> nil do
    begin
      Writeln(Format('  %s (%p)', [cls.ClassName, Pointer(cls)]));
      cls := cls.ClassParent;
    end;
    
    PPointer(foo)^ := vmi.OriginalClass;
    
    Writeln('After unhooking:');
    WorkWithFoo(foo);
    
    Writeln('Inheritance chain after unhooking:');
    cls := foo.ClassType;
    while cls <> nil do
    begin
      Writeln(Format('  %s (%p)', [cls.ClassName, Pointer(cls)]));
      cls := cls.ClassParent;
    end;
// ...

And output:

Before hackery:
  before: a = 10
  Exception: Exception
After interception:
  before: a = 10
  Result = 30
  after:  a = 20
Inheritance chain while intercepted:
  TFoo (01F34DA8)
  TFoo (0048BD84)
  TObject (004014F0)
After unhooking:
  before: a = 10
  Exception: Exception
Inheritance chain after unhooking:
  TFoo (0048BD84)
  TObject (004014F0)

The feature is primarily an infrastructure piece for advanced libraries, but hopefully you can see that it's not too difficult to get into.

Monday, August 30, 2010

Memory leaks, log analysis and Unix command-line tools

I had cause recently to tackle a fairly tricky memory leak bug in the Delphi compiler as hosted by the IDE. The test case for the bug was complicated: compiling and running a large portion of the IDE itself. After a compile, subsequent debug runs of the IDE kept on increasing the debugging IDE's memory usage.

The memory profiler in Automated QA's AQtime tool provided a very rough indication of the call stack that allocated the memory. The reason it was very rough, and frankly, not particularly useful, was because the Delphi compiler uses a memory suballocator for performance and complexity reasons. Dcc - the Delphi compiler - has roughly 3 different classes of memory allocation: memory associated with a unit, memory associated with a depth in the call stack, and temporary memory that will be done away with periodically, in particular, memory associated with parsing the syntax tree for any one procedure and generating code for it. What the stack from AQtime did tell me was that the memory was (at least initially) associated with unit allocations. Since the compiler reuses memory as units get freed, the initial allocation isn't necessarily the one that gets leaked

To get better insight into what happened around the allocator, I instrumented it. The Delphi compiler in its debug build has a simple logging system. Every logging point has a string associated with it, and by specifying a list of globs in an environment variable, the compiler will output more or less logging info, according to how those globs match the strings associated with each logging point. For example, a log point inside the memory allocator might look like "dcc.memory.alloc", and another like "dcc.memory.free". A glob like "dcc.memory.*" would then match both of these; alternatively, "*.free;*.alloc" would match both of them, but also other logical allocations and frees that use the same naming convention. Whether or not any logging point is matched by the log glob is usually cached in static variables at each logging point, so there's surprisingly little performance loss even from very aggressive logging.

I ended up with a log file which output every new unit allocation, along with all the unit frees, and some supplementary logging relating to how the compiler managed the state of units - information it used to decide whether or not to free the unit, or keep it around for reuse later. It ended up being about 2.4G in size. That's a little big to try and eyeball, so I used some tools to whittle down the interesting bits.

There's a general pattern I use; probably not the most efficient, but it works well for me because it uses the Unix-standard (Cygwin in my case) command-line tools I use most often, rather than other tools that might be more suited but I'm less familiar with. It generally goes like this:

egrep -o 'interesting regex' log | sed -r 's|.*boring (interesting).*|\1|'

That is, it extracts just the interesting lines out, then only the interesting bits of those lines, using extended regex with egrep, and extended regex with a capture in sed. The sed s command is a substitution; I substitute the entire line with the captured interesting bit.

Using this technique, I extracted every allocation address into one temporary output file and every deallocation address into another. I could then do set operations on them to discover allocations that aren't matched by deallocations:

sort allocs deallocs deallocs | uniq -u > leaks

Sort will bring all the allocations and deallocations together. Uniq, without any arguments, reduces consecutive duplicate lines down to a single line; but with -u, it removes all lines that are part of consecutive duplicates, and leaves the unique lines. By specifying deallocs twice, I guarantee that no deallocations with an unmatched allocation will appear in the output; only allocations with an unmatched deallocation.

Often this is sufficient. But in this case, the enormous amount of churn, thousands of units, meant that duplicate memory locations were showing up, associated with different units. How did I know this? I did this:

fgrep -f leaks log | egrep -o 'interesting etc.' ...

Fgrep searches for fixed strings rather than regexes, which is much faster. The -f argument tells fgrep to load the list of fixed strings from the subsequent file argument, one string per line. This is a quick way of whittling down a log file to only those lines containing one of possibly thousands of strings.

When slightly more processing is needed, a different approach can work, using bash:

for item in $(<leaks); do fgrep "$item" log | ...; done > output

That works well if you don't expect $item to contain spaces; bash breaks it up on each whitespace boundary, rather than each line. For lines, I prefer this:

cat leaks | while read -r line; do ...; done > output

I use the redundant cat at the start for readability and ease of use; bash would otherwise need the input redirector (<) at the end of the command. The bash command "read" loads a line from standard input into the specified variable. The -r argument prevents it from messing with the string, doing funny things with backslashes, which, in my experience, is almost always undesirable.

Anyhow, to get around the duplicate problem, I changed tack. I modified my filtering so that I had a simple text file consisting of lines like "alloc <address>" and "free <address>" in chronological order of the event, and then wrote a simple program which kept track of each address in a hash table, so that it wouldn't be fooled by reuse of addresses. This got me an unambiguous list of leaked allocations.

Once I had a reliable means of extracting the addresses of the leaks, I scripted the process (not much more than a list of command-lines in a bash script) so that I could easily experiment with deeper and more focused logging. This was the real meat of the process of fixing the bug, and involved a lot of scanning the log file with the "less" pager's search forward (/) and backward (?) commands, tracing the lifetime of various units. But the fact that I knew what to search for, and could whittle down the log file as necessary, was down to what had come earlier from the command-line tools.

I don't know how other programmers tackle these kinds of problems, but these are some of the tools in my toolbox, and they come in surprisingly useful every day. In many ways, debugging is the least talked-about discipline in software engineering, and I think it would be of great benefit to us all to share more of our experiences and techniques.

Tuesday, May 25, 2010

Android: Momentum and Apps

I'm liking the current momentum behind Android. I'm sure Apple will come out with their new phone soon, and the pendulum will swing a bit, but it's definitely a two horse race, and Apple isn't out of sight, not by a long shot.

I got my Nexus One from http://www.google.com/phone a few months ago, to replace my aging K800i (which still takes better photos than the Nexus One, or the iPhone, or indeed most phones). Since I got my phone directly from Google, it was easy to update to the latest Froyo, which got rid of some of my bigger annoyances on the device. Most tedious in previous versions was the need to individually select and authorize every application when an update showed up. Most days I see between two and five updated applications, and updating any given application takes around 30 seconds, so it was turning into a chore. Froyo is also much smoother in its transitions; it's the way Eclair should have been.

The Android app store (Marketplace) has gotten some criticism, but I don't know how truly meaningful those criticism are, as I've gotten applications primarily based on recommendations, or searching the store with intent, rather than browsing the store. I recently had occasion to count the number of apps I have installed, and if you judge by Android's app management screen, it's now up to 70 (I have 93 icons in my app icon scroller view). I thought I'd list a few of the best ones, as a way of aggregating my own perspective from the different recommended lists I've seen. Some of these applications are not free, but as a developer I don't baulk at paying the small amounts charged, especially since returning for a refund is built into the marketplace if you don't like the app.

ES File Explorer is probably the single application that's most responsible for me preferring Android to iPhone. It's a file browser, but more than that: it can browse Windows (SMB) network shares, as well as FTP and Bluetooth. That means I can copy music, movies etc. on and off my device to and from my NAS. I find iTunes to be a tedious waste of memory and processes, not to mention it wants to update with huge downloads practically every second week, incorporating media players and web browsers I don't want. Crapware. Being able to take control of the media upload and download experience is wonderful, and makes me feel like my phone is more than a locked-down toy.

I complement it with Astro File Manager, which has better support for photo management and a built-in image viewer (which supports image view intents, so it can actually be used by ES File Explorer to view photos). I'm not a big fan of Nexus One's stock Gallery app (by CoolIris) - lots of gloss, but slow to index photos if you've taken a bunch. Astro can also act as a file selection agent for other applications that browse for an image, such as image editors.

Another photo viewer is B&B Gallery. It was one of the first to support multi-touch, manually implemented before the OS got support. An advantage it has over the built-in Gallery is that it doesn't downsample loaded photos, so you can zoom in and check the details, rather than quickly getting lost in blur. As a gallery app, however, it's not particularly pretty. I find file management superior, especially as you don't have to wait as thumbnails from all over the place get loaded, but can classify into directories, etc.

Act 1 Video Player is an excellent replacement for the stock video player. It doesn't add any more decoding capabilities, but it has better affordances in its UI, especially with its touch-based seeking support. Best feature: swiping left and right in the center of the screen seeks back and forth through the video.

NewsRob is an offline RSS reader that synchronizes with Google Reader. It has configurable caching, so you can have it download images which are linked from the RSS (a problem with Google Reader's own offline version using Gears, ironically), up to caching the full web page associated with that RSS item. Excellent for public transport.

A major annoyance with the iPod Touch (and also the iPhone) for me was the auto-rotation. I almost never want to rotate the view, and it always ends up rotating when I'm lying down, or otherwise not in front of a computer. I was only able to solve this problem on the iPod Touch by jailbreaking it. Android has a setting for this, but for easier access to it I use the AutoRotate widget. This lets you put a 1x1 widget anywhere which toggles the auto-rotate setting on a tap.

Some games are useful for passing idle moments. Robo Defense is quite addictive tower defense with RPG-like elements; you earn what are essentially XP points, and can spend them on incremental upgrades, so there's a campaign-like aspect to the gameplay. Replica Island is a classic-style platformer which is particularly ergonomic on the Nexus One, using the scroll ball for directional control. As an aside, controls are one of the weakest elements of most iPhone games - it badly needs more physical buttons. And Nesoid, an NES emulator, is nice in principle, but a better control system is needed.

Artistic diversions: DoodleDroid is a finger-painting app with configurable brush dynamics, so with care, you can get some interesting impressionistic images out of it. Simpler, more like coloured markers than paint, is Draw!.

Of course, there are the bar code apps, like ShopSavvy, probably the most integrated when you have buying intent, though its local shop search isn't very localized, even when in London; ZXing Barcode Scanner, which runs more general web searches based on barcodes; Google Shopper and Google Goggles also do barcodes, but I feel they're weaker, and Goggles is mostly a gimmick (IMO).

Google Sky Map is pretty neat - the way it uses the accelerometer to overlay constellations etc. is probably the neatest augmented reality-style implementation I've seen, even though it doesn't overlay on a video image from the camera. Layar is the probably the canonical implementation, but I find it to be too gimmicky in practice, having to walk around like an idiot with a phone held out in front of you. At least with stars, you're normally standing still and looking into the sky.

Google Translate is another essential app. It's tantalizingly close to real-time speech to speech translation; as it is, you can speak into it and at a button press do text to speech on the translation, providing the speech recognition was good. My girlfriend tells me it can be overly literal for German, however.

Wifi Analyzer helped me get better channel placement on my home wifi access points. Really neat live view of signal strength for all the different APs in your area, even ones too faint to actually connect to.

Arity is a simple expression-based calculator which can graph simple non-parametric functions in two and three dimensions. By non-parametric, I mean you give it an expression using x, or x and y, and it plots the result of the expression as y, or z, in a 2D plane or 3D volume. You can't plot circles with it, for example.

ConnectBot is a SSH client, useful for remote administration when you're really stuck for connectivity. Doing anything serious on the command line without access to a keyboard is insanity, of course. When the job you're trying to do is simpler - a single command over SSH - ServerAssistant is a better approach.

If you're interested in programming your life, Locale can trigger events based on conditions. Conditions are one or more of location, time, orientation, calls by contacts and battery state. Settings include wallpaper, ringtone, screen brightness, wifi enabled or not, volume, bluetooth, but also actions published by third-party applications. For example, NewsRob can synchronize based on a Locale trigger. And if you've installed ASE, the Android Scripting Environment, you can run arbitrary scripts - bash, python, ruby, etc. - on a trigger. The sample scripts available for ASE include invoking text to speech to say the time and the current weather, toggling airplane mode, showing notifications, etc. Locale is a lot less useful if you have a more flexible schedule, but if you're tied in to a timetable, it makes a lot of sense.

Finally, a battery widget: Battery Left. I don't use task managers or killers; I've found that it's better to let Android do its thing and kill what it needs to kill, when it chooses to do it. I get about 46 hours on average battery, but I tend to recharge before 36 hours have gone past. You can drop this widget as a 1x1 (or 2x1) visual indicator of battery left, with configurable detailed textual data: estimated time before battery dead, estimated time of day of dead battery, estimated battery %, etc. It monitors battery performance, so it should straighten the curve that batteries self-report - I've often seen batteries say they have three-quarters battery for ages, and then run out the remainder quite suddenly, etc.

Obviously, I have many more applications installed than I've mentioned here, but they tend to be single-purpose location-based ones that have less general applicability, or ones I don't use as often and can't in good conscience recommend. But I can say that all of the above work pretty well for me, and it's notable that many of them would contravene Apple's developer policy, so for me at least, app availability for the iPhone isn't the killer advantage it's made out to be.

Tuesday, May 11, 2010

Locations vs Values: using RTTI to work with value types

Delphi's Rtti unit is designed in substantial part around TValue, a kind of hold-all record that should be capable of containing almost any Delphi value, along with type information for that value. However, this means that when you're working with value types, such as static arrays and records, modifying the values when stored in a TValue is modifying that copy, stored inside the TValue. If you want to manipulate a field (F1) of a record which is itself a field (F2) of another type, you need to first copy the F2 field's value out into a TValue, then modify F1 in the TValue, and then copy it back in to the original F2 field.

As an aside: TValue.MakeWithoutCopy does not relate to this copying behaviour, but is rather for managing reference counts with strings and interfaces and other managed types. This is particularly important when marshalling parameters to and from stack frames, where logical copies sometimes should be made, and sometimes not.

However, working with values in TValue all the time is not necessarily the most efficient technique. By adding another layer of indirection, we can improve things: instead of working with values, we can work with locations.

This can be encapsulated fairly trivially using the current RTTI support. I hacked up a TLocation type which represents a typed location analogously to how TValue represents a value:

type
  TLocation = record
  private
    FLocation: Pointer;
    FType: TRttiType;
  public
    class function FromValue(C: TRttiContext; const AValue: TValue): TLocation; static;
    class function FromAddress(ALocation: Pointer; AType: TRttiType): TLocation; static;
    function GetValue: TValue;
    procedure SetValue(const AValue: TValue);
    function Follow(const APath: string): TLocation;
    function Dereference: TLocation;
    function Index(n: Integer): TLocation;
    function FieldRef(const name: string): TLocation;
  end;

For ease of use, it uses TRttiType. If it were to be fully as flexible as TValue, it would use PTypeInfo instead, like TValue does. However, using the RTTI wrapper objects makes life a lot easier.

Here it is in use:

type
  TPoint = record
    X, Y: Integer;
  end;
  TArr = array[0..9] of TPoint;

  TFoo = class
  private
    FArr: TArr;
    constructor Create;
    function ToString: string; override;
  end;

{ TFoo }

constructor TFoo.Create;
var
  i: Integer;
begin
  for i := Low(FArr) to High(FArr) do
  begin
    FArr[i].X := i;
    FArr[i].Y := -i;
  end;
end;

function TFoo.ToString: string;
var
  i: Integer;
begin
  Result := '';
  for i := Low(FArr) to High(FArr) do
    Result := Result + Format('(%d, %d) ', [FArr[i].X, FArr[i].Y]);
end;

procedure P;
var
  obj: TFoo;
  loc: TLocation;
  ctx: TRttiContext;
begin
  obj := TFoo.Create;
  Writeln(obj.ToString);
  
  ctx := TRttiContext.Create;
  
  loc := TLocation.FromValue(ctx, obj);
  Writeln(loc.Follow('.FArr[2].X').GetValue.ToString);
  Writeln(obj.FArr[2].X);
  
  loc.Follow('.FArr[2].X').SetValue(42);
  Writeln(obj.FArr[2].X); // observe value changed
  
  // alternate syntax, not using path parser
  loc.FieldRef('FArr').Index(2).FieldRef('X').SetValue(24);
  Writeln(obj.FArr[2].X); // observe value changed again
  
  Writeln(obj.ToString);
end;

Here's most of the implementation:

{ TLocation }

type
  PPByte = ^PByte;

function TLocation.Dereference: TLocation;
begin
  if not (FType is TRttiPointerType) then
    raise Exception.CreateFmt('Non-pointer type %s can''t be dereferenced', [FType.Name]);
  Result.FLocation := PPointer(FLocation)^;
  Result.FType := TRttiPointerType(FType).ReferredType;
end;

function TLocation.FieldRef(const name: string): TLocation;
var
  f: TRttiField;
begin
  if FType is TRttiRecordType then
  begin
    f := FType.GetField(name);
    Result.FLocation := PByte(FLocation) + f.Offset;
    Result.FType := f.FieldType;
  end
  else if FType is TRttiInstanceType then
  begin
    f := FType.GetField(name);
    Result.FLocation := PPByte(FLocation)^ + f.Offset;
    Result.FType := f.FieldType;
  end
  else
    raise Exception.CreateFmt('Field reference applied to type %s, which is not a record or class',
      [FType.Name]);
end;

function TLocation.Follow(const APath: string): TLocation;
begin
  Result := GetPathLocation(APath, Self);
end;

class function TLocation.FromAddress(ALocation: Pointer;
  AType: TRttiType): TLocation;
begin
  Result.FLocation := ALocation;
  Result.FType := AType;
end;

class function TLocation.FromValue(C: TRttiContext; const AValue: TValue): TLocation;
begin
  Result.FType := C.GetType(AValue.TypeInfo);
  Result.FLocation := AValue.GetReferenceToRawData;
end;

function TLocation.GetValue: TValue;
begin
  TValue.Make(FLocation, FType.Handle, Result);
end;

function TLocation.Index(n: Integer): TLocation;
var
  sa: TRttiArrayType;
  da: TRttiDynamicArrayType;
begin
  if FType is TRttiArrayType then
  begin
    // extending this to work with multi-dimensional arrays and non-zero
    // based arrays is left as an exercise for the reader ... :)
    sa := TRttiArrayType(FType);
    Result.FLocation := PByte(FLocation) + sa.ElementType.TypeSize * n;
    Result.FType := sa.ElementType;
  end
  else if FType is TRttiDynamicArrayType then
  begin
    da := TRttiDynamicArrayType(FType);
    Result.FLocation := PPByte(FLocation)^ + da.ElementType.TypeSize * n;
    Result.FType := da.ElementType;
  end
  else
    raise Exception.CreateFmt('Index applied to non-array type %s', [FType.Name]);
end;

procedure TLocation.SetValue(const AValue: TValue);
begin
  AValue.Cast(FType.Handle).ExtractRawData(FLocation);
end;

To make it slightly easier to use, and slightly more fun for me to write, I also wrote a parser - the Follow method, which is implemented in terms of GetPathLocation:

function GetPathLocation(const APath: string; ARoot: TLocation): TLocation;

  { Lexer }
  
  function SkipWhite(p: PChar): PChar;
  begin
    while IsWhiteSpace(p^) do
      Inc(p);
    Result := p;
  end;

  function ScanName(p: PChar; out s: string): PChar;
  begin
    Result := p;
    while IsLetterOrDigit(Result^) do
      Inc(Result);
    SetString(s, p, Result - p);
  end;

  function ScanNumber(p: PChar; out n: Integer): PChar;
  var
    v: Integer;
  begin
    v := 0;
    while (p >= '0') and (p <= '9') do
    begin
      v := v * 10 + Ord(p^) - Ord('0');
      Inc(p);
    end;
    n := v;
    Result := p;
  end;

const
  tkEof = #0;
  tkNumber = #1;
  tkName = #2;
  tkDot = '.';
  tkLBracket = '[';
  tkRBracket = ']';
  
var
  cp: PChar;
  currToken: Char;
  nameToken: string;
  numToken: Integer;
  
  function NextToken: Char;
    function SetToken(p: PChar): PChar;
    begin
      currToken := p^;
      Result := p + 1;
    end;
  var
    p: PChar;
  begin
    p := cp;
    p := SkipWhite(p);
    if p^ = #0 then
    begin
      cp := p;
      currToken := tkEof;
      Exit(currToken);
    end;
    
    case p^ of
      '0'..'9':
      begin
        cp := ScanNumber(p, numToken);
        currToken := tkNumber;
      end;
      
      '^', '[', ']', '.': cp := SetToken(p);
      
    else
      cp := ScanName(p, nameToken);
      if nameToken = '' then
        raise Exception.Create('Invalid path - expected a name');
      currToken := tkName;
    end;
    
    Result := currToken;
  end;
  
  function Describe(tok: Char): string;
  begin
    case tok of
      tkEof: Result := 'end of string';
      tkNumber: Result := 'number';
      tkName: Result := 'name';
    else
      Result := '''' + tok + '''';
    end;
  end;
  
  procedure Expect(tok: Char);
  begin
    if tok <> currToken then
      raise Exception.CreateFmt('Expected %s but got %s', 
        [Describe(tok), Describe(currToken)]);
  end;

  { Semantic actions are methods on TLocation }
var
  loc: TLocation;
  
  { Driver and parser }
  
begin
  cp := PChar(APath);
  NextToken;
  
  loc := ARoot;
  
  // Syntax:
  // path ::= ( '.' <name> | '[' <num> ']' | '^' )+ ;;
  
  // Semantics:
  
  // '<name>' are field names, '[]' is array indexing, '^' is pointer
  // indirection.
  
  // Parser continuously calculates the address of the value in question, 
  // starting from the root.
  
  // When we see a name, we look that up as a field on the current type,
  // then add its offset to our current location if the current location is 
  // a value type, or indirect (PPointer(x)^) the current location before 
  // adding the offset if the current location is a reference type. If not
  // a record or class type, then it's an error.
  
  // When we see an indexing, we expect the current location to be an array
  // and we update the location to the address of the element inside the array.
  // All dimensions are flattened (multiplied out) and zero-based.
  
  // When we see indirection, we expect the current location to be a pointer,
  // and dereference it.
  
  while True do
  begin
    case currToken of
      tkEof: Break;
      
      '.':
      begin
        NextToken;
        Expect(tkName);
        loc := loc.FieldRef(nameToken);
        NextToken;
      end;
      
      '[':
      begin
        NextToken;
        Expect(tkNumber);
        loc := loc.Index(numToken);
        NextToken;
        Expect(']');
        NextToken;
      end;
      
      '^':
      begin
        loc := loc.Dereference;
        NextToken;
      end;
      
    else
      raise Exception.Create('Invalid path syntax: expected ".", "[" or "^"');
    end;
  end;
  
  Result := loc;
end;

The principle can be extended to other types and Delphi expression syntax, or TLocation may be changed to understand non-flat array indexing, etc.

This post was inspired by this question on Stack Overflow, and some similar questions to it that popped up over the past few weeks.

Monday, April 19, 2010

Programming font for VS2010

So, Visual Studio 2010 shipped. Now I have a problem: what font to use? My old standby, Dina, is a bitmap font so it doesn't work with the WPF text editor in VS2010.

Some things I look for in a programming font:

  • High information density - vertical height in particular. For example, I use Dina at 8pt.
  • Crisp, even lines
  • Strong distinction between bold and regular (for lexical highlighting)
  • Clearly unambiguous letters: l,1,0,O, and the like.
  • Balanced operators - good placement for <, >, {, }, etc.

Here's Dina 8pt in VS2008:

Here's Consolas at 8pt in VS2010:

(At least, I think that's 8pt. I have zoomed the editor in and out a little with Ctrl+Mouse Wheel, but there doesn't seem to be a zoom reset...

Here's Consolas 9pt in VS2010:

Consolas is quite constrained on the horizontal. It looks horrible at 8pt, and is still slightly taller than Dina 8pt. Even at 9pt though, to my eyes, it suffers quite badly with comparison to Dina. The bold of the comment and digits is almost unnoticeable. 'M' and 'm' are very weak, probably because of the narrowness. The vertical on the 'F' of 'Func' is weak, as is the vertical on 'r'. These are in part properties of ClearType subpixel rendering, which looks slightly different from monitor to monitor and depending on configuration, but even with the configuration selected for its heaviest rendering, it looks poor on my system. Overall, the effect is washed out and slightly blurry.

Proggy Fonts are another contender. These have TTF hinted versions of what amount to bitmap fonts, so they try and sneak older rendering in through the back door. Here's Proggy Clean Slashed Zero with Bold Punctuation (Visual Studio can bold operators (and I do - and make them white, to stand out even more), but it doesn't include [], {}, () as operators, unfortunately):

This is probably the most palatable option for me, but it's still not quite as good as Dina. It's the same height, probably because Dina was based on Proggy; but the things I don't like about it most are things that Dina fixed, in particular the oddly elongated < and >, the lazy looking 's', and the "gappy" look of the bold - contrast the comment in Proggy vs Dina. Other differences incude the leg on the 'R' and the general way the capitals are slightly overly broad. Another oddity is however ClearType has interacted with its hinting, the font has ended up tinted with green.

Proggy Clean SZBP is the font I'll stick with for the moment, until I see a better option, or a better translation of Dina to TTF than this one, which is tuned for 10pt Dina, not the size I use.

Tuesday, March 23, 2010

JEDI newsgroup change

I got word that the JEDI newsgroup changed after the old forums.talkto.net server died. The new server is at:

news.delphi-jedi.org

That's nntp on standard port 119. Please update your favorite news reader to this new server.

Saturday, March 13, 2010

CrashPlan for Backup on Nexenta

For some time, I've been using cron jobs with rdiff-backup on Cygwin for backups. The cron job runs on a Windows server I have on my home network and iteratively mounts remote Windows shares (if they're present) and runs rdiff-backup over them, with the destination also being local. Then the job runs rsync to mirror this backup to my Nexenta NAS running ZFS raidz.

This solution gives me a fair amount of local redundancy: two separate machines, with two copies of the backup data, plus the RAID-5-like redundancy and checksumming integrity that ZFS provides. Rdiff-backup is quite reassuring with respect to restores too: it stores files in the filesystem directly, along with differences (the rdiff bit) so you can go back in time. That means that restoring is as simple as copying the files straight out of the backup and deleting the rdiffs.

Of course, a backup strategy isn't solid without a remote copy. Today, I finished configuring CrashPlan, a really neat backup solution built using Java. You can read about the features etc. of CrashPlan on the website - I heard about it from the Java Posse podcast, episode 298. Initially, I opened an Amazon S3 account, and was considering mirroring my backups to S3 with s3sync, but after I evaluated CrashPlan, it looked like it made more sense than further pursuing my homegrown approach. Not only is it cheaper than S3 for my data (I guess they depend on overselling like ISPs), but CrashPlan has features that make it work well in my case.

The fact that it's primarily Java meant I could install it on my Nexenta box. CrashPlan don't have an installer for Nexenta, but they do have ones for Linux (LSB) and Solaris, and Nexenta is like a hybrid. I'm documenting the steps here in case I need to set it all up again, though hopefully not for a restore.

I'm running NexentaCore NCP 2, which doesn't come with Java. Installing a JDK (overkill, but I wanted to test the Java environment with a simple hello-world):

$ sudo apt-get install sun-java6-jdk

I was running a release candidate of Nexenta and I had to do an apt-get update and upgrade to resolve all necessary dependencies, but it was fairly stress free because Nexenta's apt-clone checkpointed the root file system with a ZFS snapshot. I did lose my almost half-year of uptime though - my Nexenta machine has been the most dependable of all my machines, in both hardware and software.

Installing CrashPlan itself needs bits from both the Linux and Solaris installers. I downloaded both, and unpacked both. In the Linux install, I ran the provided install.sh and followed the steps, putting it in /usr/local, with daemon init script in /etc/init.d and runlevel symlink in /etc/rc3.d. But the Linux install isn't enough. In particular, the init script assumes GNU ps, but Nexenta uses Solaris ps. So I swapped in CrashPlanEngine (the target of the init script symlink) from the Solaris installer in the place of the installed CrashPlanEngine that was here:

/usr/local/crashplan/bin/CrashPlanEngine

But that wasn't enough. CrashPlan loads a library called libjtux via JNI, a kind of POSIX API for Java programmers who want direct access to the OS. The libjtux.so from the Linux install was linking against libc.so.6, assuming GNU C library versioning. Replacing libjtux.so with the version from Solaris, linking against plain libc.so, solved this problem - here:

/usr/local/crashplan/libjtux.so

Finally, I had to configure CrashPlan. My Nexenta install is headless - all my interaction with it is either over Samba shares, HTTP for wiki servers etc., and of course ssh for everything else. Here's one of the really neat features of CrashPlan: the user interface for configuration is a client that depends on a single TCP connection with the local backup server. All I had to do is get the client to connect to a different local port, and tunnel that port to Nexenta using SSH, after enabling TCP forwarding in Nexenta in /etc/ssh/sshd_config. This bit is described on CrashPlan's site describing how to configure a headless client.

I'm running CrashPlan+, a for-pay version of the engine, on two machines - my main desktop and my Nexenta box - to get all the encryption, compression, deduplication etc. goodness. And considering that CrashPlan supports peer to peer backup, I may simply replace my existing ad-hoc rdiff-backup approach with local CrashPlan backups, as CrashPlan supports multiple destinations for backups, as well as receiving backups from other machines.

A key limitation of CrashPlan, and one that I found particularly annoying, is that the Windows client doesn't support network shares at all, in any shape or form - whether they're mapped to drive letters or not. The backup engine runs under the SYSTEM account, so it doesn't have network credentials, and also means it may not be able to access all the files you're trying to back up - especially EFS encrypted files. (CrashPlan doesn't seem to use Windows' EFS backup capability, e.g. ReadEncryptedFileRaw.) I changed the CrashPlan engine's service account to my own account to try and close this hole, but still no go on accessing network shares. This meant that I had to get CrashPlan running on my Nexenta box in order to store local backups on ZFS, should I so choose.

But apart from that, I've been impressed with CrashPlan's feature set and usability.

Wednesday, February 03, 2010

Multidimensional Separation of Concerns, or Matrix Inheritance

I just spotted a paper that I didn't know existed before. Abstract:

Done well, separation of concerns can provide many software engineering benefits, including reduced complexity, improved reusability, and simpler evolution. The choice of boundaries for separate concerns depends on both requirements on the system and on the kind(s) of decomposition and composition a given formalism supports. The predominant methodologies and formalisms available, however, support only orthogonal separations of concerns, along single dimensions of composition and decomposition. These characteristics lead to a number of well-known and difficult problems.

This paper describes a new paradigm for modeling and implementing software artifacts, one that permits separation of overlapping concerns along multiple dimensions of composition and decomposition. This approach addresses numerous problems throughout the software lifecycle in achieving well-engineered, evolvable, flexible software artifacts and traceability across artifacts.

I came up with a remarkably similar concept I called matrix inheritance in a previous job. Here's one description I've written up out there on the web:

[This is a description of a symptom of the problem.] Nested if-then-else can introduce non-essential complexity merely by having dead nested cases which are actually obviated by outer cases; looking at a complex version of this code, it can become quite difficult to see where exactly the code is supposed to flow under what circumstances, as there can seem to be conflicting assumptions in different areas.

Once upon a time I invented a scheme to solve this problem in the best way I thought possible, and called it "matrix inheritance". The problem with inheritance and subclassing is that it only handles a single dimension of customization. Suppose you have two dimensions, genre and type, such as [comedy, drama] and [movie, series]. If you were to try and classify any given thing under a classical type breakdown, you could subclass by one axes or the other, but you would need to duplicate subclassing for the remaining axes. So, you could end up with Programme, Comedy <: Programme, Drama <: Programme, but then you'd need ComedyMovie, ComedySeries, DramaMovie, DramaSeries, duplicating the kind axis in the two different branches.

The matrix inheritance concept basically takes the cartesian product of all option axes, essentially modelling it as an n-dimensional space, and then applies behaviour to sub-spaces of this space. So, you could apply conditions to [drama,*] and [*, series], with these two representing slices of the example 2-dimensional space described above. The advantage of modelling things this way is that it is declarative: you can analyse overlaps and identify un-covered space.

Here's the original post I made on comp.compilers when looking for related work. Here's a brief excerpt:

Object-oriented techniques have been rejected because they constrain me to using a fixed order of dimensions for reuse and overriding. A possible solution presents itself:

1. Creating a notional Cartesian-product of all dimensions of customization.

2. Overriding based on the application of rules whose scope is defined by sets described by a simple language: by example, '{a,b}' represents the set containing a and b, while '*' represents the set containing all the values for that dimension.

Friday, January 29, 2010

One-liner RAII in Delphi

Some C++ aficionados point at the ability, in C++, to create an object such that the creation allocates some kind of resource, and the destruction frees the resource, with much glee. The idiom is RAII: Resource Acquisition Is Initialization. The technique can be a useful, albeit somewhat opaque, way of acquiring a resource for the duration of a block of code and relinquishing the resource upon exiting the block. It's C++'s equivalent of Delphi / Java / C# / SEH's try / finally blocks (though of course C#'s "using" statement is an even closer analogue).

Delphi has a similar mechanism in the form of deterministic reference counting for interfaces. This can be (ab)used to functionally implement almost exactly the same technique as C++. The degree to which it is useful, however, is moderated by readability issues; RAII doesn't necessarily employ a visible nesting, and rather depends on the implied nesting of the scope in which it's used.

Anyway, here's a way of implementing RAII in Delphi, for people to whom the technique may not have occurred yet. It relies on defining a general-purpose class which implements an interface, and whose death will be triggered by scope exit:

type
  TScopeExitNotifier = class(TInterfacedObject)
  private
    FProc: TProc;
  public
    constructor Create(const AProc: TProc);
    destructor Destroy; override;
  end;

constructor TScopeExitNotifier.Create(const AProc: TProc);
begin
  FProc := AProc;
end;

destructor TScopeExitNotifier.Destroy;
begin
  if Assigned(FProc) then
    FProc;
  inherited;
end;

function MakeScopeExitNotifier(const AProc: TProc): IInterface;
begin
  Result := TScopeExitNotifier.Create(AProc);
end;

I would generally recommend hiding the actual class in the implementation-section of a unit - the functional interface of MakeScopeExitNotifier ought to be sufficient. Using this function, we can now implement a one-liner RAII equivalent of the following code:

  Writeln('Disable controls on ', {ASurface});
  try
    // (user code)
  finally
    Writeln('Enable controls on ', {ASurface});
  end;

Of course, I'm using strings etc. for simplicity, so that the demo translation is compilable. Here is the definition of the function which will be called to form the one-liner:

function DisableEnableControls(const ASurface: string): IInterface;
begin
  Writeln('Disable controls on ', ASurface);
  Result := MakeScopeExitNotifier(procedure
  begin
    Writeln('Enable controls on ', ASurface);
  end);
end;

And here it is in action:

procedure P;
begin
  DisableEnableControls('Foo');
  DisableEnableControls('Bar');
  Writeln('Controls on Foo and Bar are disabled now');
end;

Because of the way Delphi's deterministic reference counting and destruction works, Foo and Bar will be released in reverse order of construction, as expected for RAII. Functions that return an interface value in Delphi are implemented behind the scenes as procedures that take an out Result parameter; these are in effect hidden variables that keep the return values alive until the scope is exited.

The output of the program is as follows:

Disable controls on Foo
Disable controls on Bar
Controls on Foo and Bar are disabled now
Enable controls on Bar
Enable controls on Foo

Of course, it's not necessarily clear what DisableEnableControls(string) would do, and how it protects the scope. That's why I wouldn't necessarily recommend using this technique, except to counter C++-ite arguments. Using anonymous methods directly would seem to be a more readable approach in general, something along these lines:

procedure DisableEnableControls(const ASurface: string; const ACB: TProc);
begin
  Writeln('Disable controls on ', ASurface);
  try
    if Assigned(ACB) then
      ACB;
  finally
    Writeln('Enable controls on ', ASurface);
  end;
end;

procedure P;
begin
  DisableEnableControls('Foo', procedure
  begin
    DisableEnableControls('Bar', procedure
    begin
      Writeln('Controls are disabled now.');
    end);
  end);
end;

Output is the same as earlier. The extra complexity / verbosity of this abstraction over and above a literal try/finally would of course be more justified if the logic in the place of Enable and Disable sections were more involved, such as possibly setting up and rolling back transactions, etc.

However, the functionality of MakeScopeExitNotifier() routine is actually useful in the context of anonymous methods, because by capturing a variable containing the return value, you can get informed as to when the scope that created your anonymous method dies. This can be useful to e.g. free any objects that were allocated when the anonymous method was created:

function FreeOnExit(const Args: array of TObject): IInterface;
var
  toFree: TArray<TObject>;
begin
  // copy open array to toFree because open arrays can't be captured
  SetLength(toFree, Length(Args));
  Move(Args[0], toFree[0], Length(Args) * SizeOf(Args[0]));
  
  Result := MakeScopeExitNotifier(procedure
  var
    i: Integer;
  begin
    Writeln('FreeOnExit is freeing objects');
    for i := 0 to High(toFree) do
      toFree[i].Free;
  end);
end;

procedure Use(const x: IInterface);
begin
end;

function Q: TProc;
var
  onDie: IInterface;
  obj: TObject;
begin
  obj := nil;
  onDie := FreeOnExit([obj]);
  Writeln('Allocating an object');
  obj := TObject.Create;
  Result := procedure
  begin
    Use(onDie); // cause onDie to be captured
    Writeln(Format(
      'Inside callback, obj still alive: %p: %s',
      [Pointer(obj), obj.ToString]));
  end;
end;

procedure P;
var
  r: TProc;
begin
  Writeln('Before Q');
  r := Q();
  Writeln('Invoking result of Q');
  r;
  Writeln('Clearing result of Q');
  r := nil;
  Writeln('Result is cleared');
end;

The output is something like:

Before Q
Allocating an object
Invoking result of Q
Inside callback, obj still alive: 00970D20: TObject
Clearing result of Q
FreeOnExit is freeing objects
Result is cleared

This technique, capturing a variable of an interface or other deterministically lifetime-managed type, is essential for flexibility in cleaning up any resources that an anonymous method needs to for its lifetime. Note, however, that there can be a gotcha: the following alternative, using MakeScopeExitNotifier directly, will not work:

// <BROKEN CODE DO NOT USE>
function Q: TProc;
var
  onDie: IInterface;
begin
  onDie := MakeScopeExitNotifier(procedure
  begin
    Writeln('died!');
  end);
  Result := procedure
  begin
    Use(onDie);
  end;
end;
// </BROKEN CODE DO NOT USE>

The reason it's broken is that the anonymous method that's handed off to MakeScopeExitNotifier will itself keep the scope alive. So in order to make use of MakeScopeExitNotifier for this particular purpose, the anonymous method passed to it needs to be in a different scope.

Wednesday, January 20, 2010

Using anonymous methods in method pointers

Anonymous methods may have associated state. In particular, all variables that an anonymous method captures need to be kept alive so long as the anonymous method is callable. For this reason, anonymous methods are implemented with a lifetime management approach: anonymous methods are actually methods on an object which implements COM-style reference counted interfaces. Method references are interface references with a single method called Invoke.

If one badly wants to store an anonymous method in a method pointer, this information can be used to shoehorn it in. Here's how it can be done:

procedure MethRefToMethPtr(const MethRef; var MethPtr);
type
  TVtable = array[0..3] of Pointer;
  PVtable = ^TVtable;
  PPVtable = ^PVtable;
begin
  // 3 is offset of Invoke, after QI, AddRef, Release
  TMethod(MethPtr).Code := PPVtable(MethRef)^^[3];
  TMethod(MethPtr).Data := Pointer(MethRef);
end;

This procedure will take a method reference as the first argument, and then extract the two crucial pieces of data needed to put into a method pointer: the value to be passed as the first argument (the method, or interface, reference), and the code address to be called (the value of the Invoke entry in the interface's vtable).

The procedure can be used as follows. Note that the method reference still needs to be kept alive somewhere as long as the method pointer is callable to avoid premature disposal of the heap-allocated object:

type
  TMeth = procedure(x: Integer) of object;
  TMethRef = reference to procedure(x: Integer);
  
function MakeMethRef: TMethRef;
var
  y: Integer;
begin
  y := 30;
  Result := procedure(x: Integer)
  begin
    Writeln('x = ', x, ' and y = ', y);
  end;
end;

procedure P;
var
  x: TMethRef;
  m: TMeth;
begin
  x := MakeMethRef();
  MethRefToMethPtr(x, m);
  Writeln('Using x:');
  x(10);
  Writeln('Using m:');
  m(10);
end;

On the other hand, if the anonymous method's body never captures any variables, then it's not necessary to keep the method reference around. This loses much of the benefits of anonymous methods, but it does prove the point:

uses SysUtils, Forms, StdCtrls, Dialogs, classes;

procedure MethRefToMethPtr(const MethRef; var MethPtr);
type
  TVtable = array[0..3] of Pointer;
  PVtable = ^TVtable;
  PPVtable = ^PVtable;
begin
  // 3 is offset of Invoke, after QI, AddRef, Release
  TMethod(MethPtr).Code := PPVtable(MethRef)^^[3];
  TMethod(MethPtr).Data := Pointer(MethRef);
end;

type
  TNotifyRef = reference to procedure(Sender: TObject);

function MakeNotify(const ANotifyRef: TNotifyRef): TNotifyEvent;
begin
  MethRefToMethPtr(ANotifyRef, Result);
end;

procedure P;
var
  f: TForm;
  btn: TButton;
begin
  f := TForm.Create(nil);
  try
    btn := TButton.Create(f);
    btn.Parent := f;
    btn.Caption := 'Click Me!';
    btn.OnClick := MakeNotify(procedure(Sender: TObject)
      begin
        ShowMessage('Hello There!');
      end);
    f.ShowModal;
  finally
    f.Free;
  end;
end;

begin
  try
    P;
  except
    on e: Exception do
      ShowMessage(e.Message);
  end;
end.

Thursday, January 07, 2010

Delphi 2010 RTTI Contexts: how they work, and a usage note

Delphi 2010 includes extended support for RTTI, also known as run-time type info or reflection. Many design approaches that have previously only been possible with managed languages like C# and Java because of the code annotation and introspection they required should now be possible in the Delphi world.

Something somewhat interesting about how the RTTI works is its approach to object pools. Delphi isn't a garbage collected language, so users need to be careful to free objects when they're no longer needed, either explicitly, or by designing or using some kind of ownership scheme, such as that used by TComponent, where the Owner takes care of destruction.

Type information usage scenarios don't mesh particularly well with a TComponent-style of ownership. Typically, when working with RTTI, you want to do some kind of search for interesting objects, do something with them, and then go on your way. That means that many objects may get allocated for inspection, but not actually be used. Managing those objects' lifetimes independently would be tedious, so a different approach is used: there is a single global RTTI object pool. While there is at least one RTTI context active in the application, this object pool keeps all its objects alive. When the last context goes out of scope, the objects get freed up.

The pool management works by using a Delphi record that contains an interface reference. The first time any given RTTI context is used, it fills in this interface reference. It can't fill it in any later than first use, because Delphi records don't support default constructors, which besides have their own problems. For example, how do you handle exceptions in default constructors, in all the places they can occur? Allocating arrays, thread-local variables, global variables, global variables in packages, temporary objects created in expressions, etc. It can get ugly, and in C++ it sometimes does.

So, this first use allocates an interface, called a pool token. It acts as a reference-counted handle to the global object pool. For so long as this interface is alive, the the global object pool should stay alive. Even if the RTTI context is copied around, Delphi's built-in interface handling logic, designed along COM principles, will ensure that the interface doesn't gets disposed of prematurely or get its reference count muddled up. And when an RTTI context goes out of scope, either by being a local variable in a function that is exited, or a field in an object that is freed, the reference count is reduced. When it hits zero, the pool is emptied.

The biggest upside of this approach is that RTTI usage should feel reasonably cheap, conceptually speaking. Code need only declare a variable of the appropriate type, and start using it:

procedure Foo;
var
  ctx: TRttiContext;
  t: TRttiType;
begin
  t := ctx.GetType(TypeInfo(Integer));
  Writeln(t.Name);
end;

A downside, however, is that lazy initialization can create a gotcha. Imagine this scenario:

  1. Library A declares an RTTI context A.C
  2. User code B declares an RTTI context B.C
  3. B pulls some RTTI objects O out of B.C, in order to hand them to library A
  4. B.C goes out of scope
  5. Library A now tries to work with O, but discovers much to its surprise, that the objects have been prematurely disposed, even though A already has an RTTI context, A.C

The problem is that A never used A.C, so it never allocated a pool token. When B.C used its context, the pool came into being, and objects O were assigned to it; but after B.C went out of scope, the objects got freed.

The solution to this problem is for Library A, knowing that it has a long-lived RTTI context and it expects to communicate with third-party code which allocates objects from its own RTTI context and hands them back, it should ensure that the long-lived context's pool token is allocated. A trivial way to do this is like this:

type
  TFooManager = class
    FCtx: TRttiContext;
    // ...
    constructor Create;
    // ...
  end;

constructor TFooManager.Create;
begin
  FCtx.GetType(TypeInfo(Integer));
  // ...
end;

This will allocate only a bare minimum of RTTI objects, those needed to represent the type System.Integer, but more importantly, will ensure that FCtx has a pool token and will keep the global RTTI pool alive.

In future releases of Delphi, the static method TRttiContext.Create will make sure that its return value has a pool token allocated; currently, it does not. TRttiContext.Create was originally defined to make the TRttiContext record feel more like a class for people unfamiliar with the idiom of using interfaces for automated deterministic lifetime management. The corresponding TRttiContext.Free method disposes of the internal pool token, and should remain the same.