tag:blogger.com,1999:blog-101060322024-03-14T09:24:53.315+00:00Entropy OverloadBarry Kellyhttp://www.blogger.com/profile/10559947643606684495noreply@blogger.comBlogger102125tag:blogger.com,1999:blog-10106032.post-37500050501903981442013-01-14T05:32:00.002+00:002013-01-14T05:45:33.575+00:00Aaron Swartz<p>Aaron's death has had a larger effect on me than most things I read about on the web. This death is a little closer than most, and very sad.</p>
<p>I had a number of email conversations with him about his blog posts. His articles often annoyed me in their naivete, and what I perceived as their ignorance as to the way the world worked (especially from an economic perspective) and I have to say I flamed him a bit in that correspondence. We had a few to and fros, in the spirit of young guys who hadn't mastered tact online. My lack more than his.</p>
<p>But it was this naivete, this big disconnect between how he seemed to perceive the world and how it was (as I saw it), that drove him to do what he did. He was foolhardy but pure of heart, earnest, childlike. The little knocks that force most of us into alignment and compliance and conformity from an early age seemed not to have affected him, until the knocks got a lot harder, too hard, and yet he never bent.</p>
<p>The apparent tactics of US prosecution - to overcharge and then plea-bargain to extract guilty verdicts even from innocent parties, owing to the legal risk of one of the charges sticking - are sickening. I conjecture they're due to electoral incentives and the need to be seen to be tough on crime. Theatre to placate the mob, like Nero throwing Christians to the lions.</p>Barry Kellyhttp://www.blogger.com/profile/10559947643606684495noreply@blogger.com2tag:blogger.com,1999:blog-10106032.post-68743190360289729902012-09-04T01:18:00.000+01:002012-09-04T01:19:21.522+01:00Moving On<p>An update: I've left Embarcadero Technologies.</p>
<p>There's a lot of reasons. I was in the same position for more than six years, but I didn't necessarily want to be pigeonholed as a compiler guy. I was working remotely from London, with all the downsides that remote working entails: relative lack of camaraderie, out of sight / out of mind, disconnection from the actually quite vibrant local tech scene (I live about 3 miles from Old Street roundabout). But more than anything else, I'd fallen out of love with Delphi, and could no longer motivate myself to try and make it better - the gap I'd try to bridge would be a gap too far for the market to bear.</p>
<p>I'm currently taking stock, thinking about what I want to do next. No specific plans, but it's going to be fairly different than what went before!</p>Barry Kellyhttp://www.blogger.com/profile/10559947643606684495noreply@blogger.com55tag:blogger.com,1999:blog-10106032.post-31195486117913756642012-05-08T19:01:00.003+01:002012-05-08T19:01:47.947+01:00Motorcycling in the Spanish Pyrenees, and sport-touring tips<p>
I've just come back from an excellent holiday: a motorbike trip from London to southern Spain, via <a href="https://en.wikipedia.org/wiki/Ll%C3%ADvia">Llívia</a> and Andorra. The highlight of the trip was the roads in the Spanish Pyrenees. Many almost abandoned (frequently went for 10 to 30 minutes without seeing another vehicle), mostly good surface, with bends that go on for miles and miles. Not to forget the fantastic views. Largely good weather - we (myself and my girlfriend, on separate bikes) stuck mostly to the east coast of Spain, and only really got wet on a couple of days, when travelling through France. Not too hot, either, one of the primary reasons we went at the tail end of April.
</p>
<iframe width="560" height="315" src="http://www.youtube.com/embed/Qey0erjScrU" frameborder="0" allowfullscreen></iframe>
<p>
I want to note down some of the things I've learned over the past few touring trips I've taken, and of course distilled from year-round riding through English weather. Perhaps it will help someone else out there, save them from getting wet too many times, or buying more gear than necessary.
</p>
<p>
For context: I'm a sport-touring rider, with a bias of perhaps 3:1 sport to touring; that is, I prefer a light bike, no pillion - luggage strapped to the pillion seat - and am not too concerned about being super-comfortable on the motorway at motorway speeds; but I like a fairly upright seating position and some wind protection. I'll tolerate motorways if they're necessary to get to the really fun roads sooner, where I can throw the bike around more. I ride a Kawasaki ER6f; as a medium-capacity twin, it's got lots of torque at lower speeds without needing for a screaming engine, but is still fairly light and sporty for having fun, particularly when you keep the revs up around 6k.
</p>
<p>
In and around London though, I usually ride my scooter, an SH300. A bit of a sleeper, it often surprises guys on big bikes. It looks like a 125 - especially since maxi-scooters are fairly rare in the UK and are somewhat looked down upon - but has significant acceleration to about 40mph. It still rides like a scooter and is able to get in and around traffic with ease that's a lot harder and riskier on a big bike, primarily because of the low centre of gravity.
</p>
<p>
Motorcycle clothing for touring: my key tip is layers. Specifically, for a jacket, look for something relatively light with loads of ventilation options (i.e. zip vents, zip-off panels; this generally means textiles), but is still somewhat windproof when fully zipped up, and has a zip-in waterproof liner. Generally, people go touring in or towards good weather, so ventilation is very important, but good roads are usually in hilly areas, often shaded with trees, and times with least traffic often don't coincide with peak heat, so you need the windproofing and waterproofing to block out cold air. For extra warmth, pack a fleece, wool top, etc. No need to rely on a third liner for the jacket.
</p>
<p>
In steady rain, however, a zip-in waterproof liner isn't good enough. It'll probably keep you mostly dry (though in my experience the zip area always leaks from pools of water forming at folds and creases, combined with wind pressure), but there is another problem: the jacket outer will get soaked through, and it will carry the water down your arms and into your gloves. The waterproof liner generally only fits inside rather than outside your gloves, so it will guide the water into your gloves. So my solution is a motorbike rain jacket - i.e. one which is over-sized for its proportions so it fits on top of existing jackets. I have a high-vis fluorescent yellow Bering jacket, for extra safety in torrential downpours, which limit visibility much like fog, especially on the motorway with spray thrown up by tyres. It's very important that this jacket has cuffs with velcro or some similar positively adjustable system (not just elastic) that fits <b>over</b> your gloves (so water can't run into the gloves), but <b>tightly</b>, so that rain and wet air can't get up your sleeve.
</p>
<p>
For rain in London, I generally wear my winter jacket, which has a Gore Lock Out zip. This is a double-interlocking rubber zip, and is completely waterproof and windproof; it also needs lubrication with silicone oil once a month to keep it smooth.
</p>
<p>
For trousers, I get by with permanently ventilated textiles with zips for further ventilation. To stay warmer, I can wear jeans underneath; but the fairing on my bike keeps off most of the wind, unless it's very cold, and legs don't feel the cold nearly as much as the torso. Similarly to up top, I have waterproof over-trousers for rain, and in case it gets very cold. The weak area of waterproof trousers is the crotch. Water generally collects on your torso and runs down until it pools there. If the crotch isn't totally waterproof, it will leak. Most waterproof over-trousers don't have reinforced crotch areas, and as such they're good for perhaps 10 hours of light wear before they'll start leaking. If you can find them, get ones with a reinforced crotch and stay away from ones with simple seams right where your legs split; so far, I've had to get by with purchasing new over-trousers when the forecasts say I'll be riding through heavy rain.
</p>
<p>
Have at least two pairs of waterproof gloves for riding when expecting rain (and even sunny summer trips usually have a thunderstorm at some point). Even waterproof gloves will get really soggy leather on the outside, and damp and clammy inside. After the rain has let up, it's quite pleasant to pull over and put on some dry gloves. Even better if you have heated grips; dry and toasty fingers make all the difference, and help keep you sharp on the controls, even if you're a little wet elsewhere. I have found waterproof over-gloves quite dangerous; they're generally very slippy and hard to get on, much more trouble than they're worth. Same thing with waterproof over-boots, they're usually a poor fit and interfere or get caught on the gear lever; can be dangerous. Get quality waterproof boots; bring two pairs of boots if you don't think you can stand wearing waterproof non-ventilated boots in the expected weather.
</p>
<p>
Luggage: I get by with an oversized (Givi E55) top-box containing light things for quick access (visor sponge and wipes, spare gloves, spare earplugs, spare light bulbs, spare straps, extra rain gear), and for keeping my helmet and gloves in when parked up (this is why it's oversized) or shopping when staying at a self-catering place (exceeding the load limit for short distances at low speed); and a holdall / gym bag with rectangular base, strapped tightly to my pillion seat using two 220 daN (decanewton, ~220kg) luggage straps, so that it's completely rigid and won't move at all as the bike moves, even in vigorous cornering. I don't bother with a waterproof bag; I put my clothes and other things I want to keep dry in heavy-duty rubble sacks. At the bottom of the bag, I keep my laptop; currently a MacBook Air, on top of a long wooden cutting board approximately the shape of the bag, wrapped up together in a rubble sack. The idea here is to prevent too much bending force being applied to the laptop by the pillion seat. On top, there's clothes intermediating the force from the straps. So far my laptops have survived (in the past, I was using a Toshiba Portege) fine, except for a few minor marks on the screen from being pressed into the keyboard.
</p>
<p>
Touring by motorbike has been some of the most fun I've ever had in my life. Before I learned to ride, I didn't think much about bikes; at best, I dismissed riders as either adrenaline-junky boy-racers, or hairy old fogeys. The practicality of a scooter as a means of getting around London was my gateway drug. One thing lead to another, and now I'm hooked, for life, if I don't injure myself too badly along the way!
</p>Barry Kellyhttp://www.blogger.com/profile/10559947643606684495noreply@blogger.com3tag:blogger.com,1999:blog-10106032.post-15936196196832004092011-10-04T12:22:00.002+01:002011-10-04T12:56:13.778+01:00Delphi XE2 compiler performance<p><a href="http://www.embarcadero.com/products/delphi">Delphi XE2</a> introduced namespaces across the runtime library. This stressed unit name lookup inside the compiler, and led to some severe performance regressions in certain cases. So during the runup to the XE2 release, I fired up a profiler and started investigating. It turns out there were numerous situations where lookups were being performed repeatedly with the same arguments, and logically the results should have been consistent across these repeated calls. A relatively cheap and easy fix seemed to be <a href="http://en.wikipedia.org/wiki/Memoization">memoization</a>. So I added a caching infrastructure to the compiler and used the profiler to guide me where to inject it.</p>
<p>For the most part - particularly for full builds - the cache works really well. But I've had some reports of bugs that I suspected were caused by the cache hanging on to function results slightly too long, and upon investigation, this turned out to be true. The problem with caches is usually in invalidation; if you don't invalidate the cache soon enough and in all required situations, you end up serving stale results. So there are a few bugs lurking in the Delphi compiler here, which I'm seeking out and squashing.</p>
<p>Some good news, however; I had anticipated that this might be the case, so I added a super secret switch that enables diagnosing a probable cache failure: caches can be selectively disabled, and if a problem goes away with the cache disabled, it's probably because of stale results.</p>
<p>Caches can be disabled by setting an environment variable:</p>
<pre>
DCC_CACHE_DISABLE='SearchUnitNameInNS,FileSystem,UnitFindByAlias,GetUnitOf'
</pre>
<p>The above environment variable setting disables all the compiler's caches. By including fewer than the four separate cache names, the problem can iteratively be narrowed down to a specific cache.</p>
<p>I've just been fixing one bug caused by the cache that brought home how needed it is. The project is large; almost 2 million lines. An initial build with the cache enabled takes about a minute on my machine; the bug exhibits itself in later incremental compiles when modifying the source code and pressing F9, producing spurious errors. However, once I disabled the cache (or rather, I recompiled the compiler with cache sanity checking enabled, which still filled out the cache, but also invoked the underlying logic, and simply compared the results to verify the cache), the build time took nearly 3 hours!</p>
<p>Note: invalidation is most likely to be a problem on incremental compiles, rather than full rebuilds, especially from within the IDE. The reason is that the compiler may have lots of stale data for one half of an incremental compile that it later decides is out of date (e.g. a dependency changed); this can leave a bunch of stale entries in the cache for all the memoized function calls that occurred in the first half of the compile. The cache is strictly per-compile; it keeps no data across multiple compilations, even partial compilations.</p>Barry Kellyhttp://www.blogger.com/profile/10559947643606684495noreply@blogger.com11tag:blogger.com,1999:blog-10106032.post-17952347923644653622011-03-01T15:13:00.005+00:002011-03-01T15:27:14.538+00:00An ugly alternative to interface to object casting<p>I was answering a question on Stack Overflow, but the user didn't have the latest version of Delphi. My answer included converting an interface to an object instance, which is made possible with the <b>as</b> cast on interfaces in recent Delphi versions. But there is another way of doing it, exploiting the regularity Delphi interface vtable implementations:</p>
<pre>
{$apptype console}
function Intf2Obj(x: IInterface): TObject;
type
TStub = array[0..3] of Byte;
const
// ADD [ESP+$04], imm8; [ESP+$04] in stdcall is Self argument, after return address
add_esp_04_imm8: TStub = ($83, $44, $24, $04);
// ADD [ESP+$04], imm32
add_esp_04_imm32: TStub = ($81, $44, $24, $04);
function Match(L, R: PByte): Boolean;
var
i: Integer;
begin
for i := 0 to SizeOf(TStub) - 1 do
if L[i] <> R[i] then
Exit(False);
Result := True;
end;
var
p: PByte;
begin
p := PPointer(x)^; // get to vtable
p := PPointer(p)^; // load stub address from vtable
if Match(p, @add_esp_04_imm8) then
begin
Inc(p, SizeOf(TStub));
Result := TObject(PByte(Pointer(x)) + PShortint(p)^);
end
else if Match(p, @add_esp_04_imm32) then
begin
Inc(p, SizeOf(TStub));
Result := TObject(PByte(Pointer(x)) + PLongint(p)^);
end
else
raise Exception.Create('Not a Delphi interface implementation?');
end;
type
ITest = interface
procedure P;
end;
TTest = class(TInterfacedObject, ITest)
F: array[0..200] of Byte;
procedure P;
end;
procedure TTest.P;
begin
Writeln('Hello');
end;
procedure Go;
var
orig: TTest;
i: ITest;
o: TObject;
begin
orig := TTest.Create;
i := orig;
i.P;
o := Intf2Obj(i);
Writeln(o = orig);
end;
begin
Go;
end.
</pre>
<p>This approach is predicated on the idea that the stub code that Delphi produces for turning the implicit interface argument into an instance argument is predictable. It generally only has two forms, depending on how much of an adjustment it needs to make (which itself depends on how much instance data there is). It ought to work for almost all 32-bit Delphi interfaces that have been implemented by instances, where the vtable was created by the compiler. If not, other stub variations can be analyzed (in the IDE CPU view) and handled too. It ought to be pretty safe, as only this specific code is permitted. It could be made even safer by ensuring that the stub ends with a JMP and that the instance returned has a ClassType descending from TClass.</p>
<p>Update: After a Google search I note that Hallvard <a href="http://hallvards.blogspot.com/2004/07/hack-7-interface-to-object-in-delphi.html">also wrote about this</a> some time ago. His code is a little tighter than mine (using Integer constants rather than byte-by-byte comparison); in my defense, I only spent a few minutes on this...</p>Barry Kellyhttp://www.blogger.com/profile/10559947643606684495noreply@blogger.com11tag:blogger.com,1999:blog-10106032.post-29741433800939999562010-12-20T18:03:00.007+00:002010-12-20T18:27:20.693+00:00Scrolling: Chrome vs Firefox<p>I was using the Chrome browser on my laptop the other day (some browser compat problem). I don't normally use it; Firefox is my preferred browser. I couldn't help but notice how peculiarly laggy it felt to use. It came down to vertical scrolling: this highly common task - for any long article you'll be doing a lot of it - felt jerky and unpleasant. So I did a quick ad-hoc experiment: open up the <a href="http://www.artima.com/lejava/articles/azul_pauseless_gc.html">same web page</a> in both browsers with viewports of the same size, grab the vertical scroll bar, and wiggle it up and and down continuously so the browser is constantly redrawing text.</p>
<img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi6aY5CmM3XFgLCc3CW99qQvS5ToJ4wJdSQlKAt6orIJWsFDn93cpDV8waDlbnFKeNi10fymHT3pfwLOj1owf1c5hlJd-1u6znAi86XiOyeGHPUzDNSaPiIq_or0C48-HSd53Gp/s1600/chrome-vs-firefox.png" style="border: none" />
<p>This is a crop from Process monitor on my desktop machine, which has 8 logical cores, i.e. 4 physical cores with hyperthreading. That means that 100% CPU usage on one core shows up as 12.5% CPU usage. The big wide bump on the left is the CPU usage while I was fiddling with Chrome; the one on the right is Firefox. (The bump in the middle is sqlservr.exe, which wakes up and does meaningless busywork every 60 seconds.) You can see that Chrome uses perhaps 50% more CPU usage on this task than Firefox, with more time spent in the user process and (proportionately) less in the kernel (presumably shifting bits around). Eyeballing an average of CPU usage, Firefox ranged from 6% to 7.5%, while Chrome pegged at 12.5%. Chrome is simply less CPU-efficient redrawing while scrolling, and it's very obvious to my eyes. (I did the same experiment with IE, and it was in the middle, with about 9% CPU.)</p>
<p>I might hazard a guess that Firefox spends more memory on caching bitmaps of the web page, or some similar trick trading off space for time. In any case, it's one of the reasons Firefox is still my primary browser, and also why I'm completely unconcerned about its memory usage. I have yet to encounter significant paging because of Firefox, not least because it's a 32-bit process, limiting its maximum usage, and I have alternately 12G and 4G of memory in my primary desktop and laptop respectively.</p>
<p>Mind you, when I add up the private working set for all 8 chrome.exe processes apparently needed to display this web page, they add up to 86M, only 1M less than my recently restarted firefox.exe session at 87M. (Comparing other memory usage numbers is awkward, as non-working-set memory isn't relevant, while non-private memory would be double-counted with Chrome.)</p>Barry Kellyhttp://www.blogger.com/profile/10559947643606684495noreply@blogger.com7tag:blogger.com,1999:blog-10106032.post-7425147911559697322010-11-15T06:14:00.008+00:002010-11-15T07:32:25.932+00:00CrashPlan manual installation approach on Nexenta<p>I wrote <a href="http://blog.barrkel.com/2010/03/crashplan-for-backup-on-nexenta.html">an earlier post</a> about installing CrashPlan on Nexenta. However, it seems that CrashPlan have changed their Linux installer's modus operandi, and it tries to install a (Linux) JRE of its own. Of course, that won't work; it needs the Nexenta JRE (but I get the whole JDK, as you never know when you'll need to brew up some Java):</p>
<pre>$ sudo apt-get install sun-java6-jdk
</pre>
<p>Anyhow, I received some comments and emails about setting all this up, and figured I'd break it down a little more for people who still want to get it all working. It's a process of hacking things together, though, not a blind recipe; that's why I'm proceeding almost as if it's a debugging session.</p>
<p>Anyhow, once we have a JRE installed (from apt-get above), we can try and extract out the guts of the Linux installer, so rather than running its install.sh, we can set it up manually.</p>
<p>I went to the <a href="http://b5.crashplan.com/consumer/download.html?os=Linux">CrashPlan Linux download web page</a>, and started downloading the installer in a scratch folder (this was the URL at the time of writing):</p>
<pre>$ wget http://download.crashplan.com/installs/linux/install/CrashPlan/CrashPlan_2010-03-08_Linux.tgz
</pre>
<p>That's a gzipped tar archive, so I extracted it:</p>
<pre>$ tar xzf CrashPlan_2010-03-08_Linux.tgz
$ cd CrashPlan-install # the dir it extracted
$ ls
CrashPlan_2010-03-08.cpi EULA.txt INSTALL README install.sh scripts uninstall.sh
</pre>
<p>I didn't know what the big CrashPlan*.cpi file was, so I checked:</p>
<pre>$ file CrashPlan_2010-03-08.cpi
CrashPlan_2010-03-08.cpi: gzip compressed data, was "CrashPlan_2010-03-08.cpi", from Unix, max compression
</pre>
<p>So it's a gzipped file! I decompressed it, and tested the result:</p>
<pre>$ gunzip < CrashPlan_2010-03-08.cpi > test
$ file test
test: ASCII cpio archive (pre-SVR4 or odc)
</pre>
<p>A cpio archive! I extracted that too, but in a separate directory:</p>
<pre>$ mkdir t; cd t; cpio -i ../test
$ ls
bin conf doc jniwrap.lic lang lib libjniwrap.so libjniwrap64.so libjtux.so libjtux64.so log skin upgrade
</pre>
<p>Now this looks very close to the root listing of my actual current CrashPlan install on Nexenta, and by and large it is. Here's a rough difference between the two (listing.curr is my actual install, listing is extracted from the installer):</p>
<pre>
--- listing.curr 2010-11-15 06:32:12.548701734 +0000
+++ listing 2010-11-15 06:31:40.726884007 +0000
-CrashPlanEngine.pid
-bin/CrashPlanDesktop
-bin/CrashPlanEngine
-bin/CrashPlanEngine.lsb
-bin/run.conf
-bin/vars.sh
-conf/my.service.xml
-conf/service.login
-conf/service.model
-install.vars
-libjtux.so.lsb
-libjtux.so.sol
</pre>
<p>Let's go through those. This install tree is almost complete; it can be put in /usr/local/crashplan (that's where I have mine) or wherever you like, so long as the configuration bits are also hooked up appropriately.</p>
<p><b>CrashPlanEngine.pid</b> is just the pidfile, a text file containing the process id of the currently running instance. I'm not sure why it's in the crashplan directory rather than somewhere like /var/run, but it is. I believe the crashplan service will create it.</p>
<p><b>CrashPlan*</b> are all shell wrappers for Java applications; *.lsb (LSB for Linux Standard Base) are the original versions (I think I renamed them to this). I don't have an X server or libraries on my Nexenta install, so I'm uninterested in CrashPlanDesktop. CrashPlanEngine however is important; it's the main daemon file. Here's mine, which seems to work; I think I may have edited it to work with some GNU utils rather than Solaris utils, as I believe this came from the Solaris installer (or vice versa; actually that's probably more likely):</p>
<pre>
#!/bin/bash
TARGETDIR="`dirname ${0}`/.."
. ${TARGETDIR}/install.vars
. ${TARGETDIR}/bin/run.conf
cd ${TARGETDIR}
case $1 in
start)
PID=`/usr/bin/ps -Af -o pid,ppid,args | grep 'app=CrashPlanService' | grep -v grep | cut -f2 -d' '`
if [ -n "$PID" ]; then
echo CrashPlan is already running with pid $PID
exit 1;
fi
echo "Starting CrashPlan Engine ... "
nice -n 19 ${JAVACOMMON} ${SRV_JAVA_OPTS} -classpath "./lib/com.backup42.desktop.jar:./lang" com.backup42.service.CPService > ${TARGETDIR}/log/engine_output.log 2> ${TARGETDIR}/log/engine_error.log &
if [ $! -gt 0 ]; then
echo $! > ${TARGETDIR}/CrashPlanEngine.pid
echo "OK"
else
echo "FAIL"
exit 1
fi
;;
stop)
echo "Stopping CrashPlan Engine ... "
if [ -f ${TARGETDIR}/CrashPlanEngine.pid ] ; then
kill `cat ${TARGETDIR}/CrashPlanEngine.pid`
sleep 5
fi
PID=`/usr/bin/ps -Af -o pid,ppid,args | grep 'app=CrashPlanService' | grep -v grep | cut -f2 -d' '`
if [ -n "$PID" ]; then
echo Still running, killing PID=$PID
kill -9 $PID
fi
rm -f ${TARGETDIR}/CrashPlanEngine.pid
echo "OK"
;;
*)
echo "$0 <start|stop>"
exit 1
;;
esac
</pre>
<p>As you can see, that script sources (includes) a couple of other guys, <b>bin/run.conf and install.vars</b>. bin/run.conf looks like this:</p>
<pre>SRV_JAVA_OPTS="-Dfile.encoding=UTF-8 -Dapp=CrashPlanService -DappBaseName=CrashPlan -Xms20m -Xmx512m -Dsun.net.inetaddr.ttl=300 -Dnetworkaddress.cache.ttl=300 -Dsun.net.inetaddr.negative.ttl=0 -Dnetworkaddress.cache.negative.ttl=0"
GUI_JAVA_OPTS="-Dfile.encoding=UTF-8 -Dapp=CrashPlanDesktop -DappBaseName=CrashPlan -Xms20m -Xmx512m -Dsun.net.inetaddr.ttl=300 -Dnetworkaddress.cache.ttl=300 -Dsun.net.inetaddr.negative.ttl=0 -Dnetworkaddress.cache.negative.ttl=0"
</pre>
<p>install.vars looks like this (for my current install):</p>
<pre>TARGETDIR=/usr/local/crashplan
BINSDIR=/usr/local/bin
MANIFESTDIR=/tank/share/backup/crashplan
INITDIR=/etc/init.d
RUNLVLDIR=/etc/rc3.d
INSTALLDATE=20100310
APP_BASENAME=CrashPlan
DIR_BASENAME=crashplan
DOWNLOAD_HOST=download.crashplan.com
JAVACOMMON=/usr/bin/java
</pre>
<p>The MANIFESTDIR variable points to a directory I expect I provided when I ran the original install.sh, but there's nothing currently in it. The other paths are as you'd expect.</p>
<p>As far as I can make out, the <b>conf/*</b> files are created by CrashPlan itself in coordination with the configuration utility, which, as I linked in my previous post, can be <a href="http://support.crashplan.com/doku.php/how_to/configure_a_headless_client">run remotely from a Windows (or other OS) install of CrashPlan</a>. There's a default.service.xml as a prototypical my.service.xml, while service.{login,model} appear to be login and encryption keys.</p>
<p>That leaves libjtux.so.*. These are renamed versions of the file libjtux.so which differs between Linux and Solaris. Primarily they differ in which C library they link against (note: running ldd on untrusted binaries is unsafe, but I trust these binaries):</p>
<pre>$ ldd libjtux.so.*
warning: ldd: libjtux.so.lsb: is not executable
libjtux.so.lsb:
librt.so.1 => /lib/librt.so.1
libnsl.so.1 => /lib/libnsl.so.1
libc.so.6 => (file not found)
libmp.so.2 => /lib/libmp.so.2
libmd.so.1 => /lib/libmd.so.1
libscf.so.1 => /lib/libscf.so.1
libc.so.1 => /lib/libc.so.1
libuutil.so.1 => /lib/libuutil.so.1
libgen.so.1 => /lib/libgen.so.1
libm.so.2 => /lib/libm.so.2
warning: ldd: libjtux.so.sol: is not executable
libjtux.so.sol:
librt.so.1 => /lib/librt.so.1
libsocket.so.1 => /lib/libsocket.so.1
libnsl.so.1 => /lib/libnsl.so.1
libc.so.1 => /lib/libc.so.1
libmp.so.2 => /lib/libmp.so.2
libmd.so.1 => /lib/libmd.so.1
libscf.so.1 => /lib/libscf.so.1
libuutil.so.1 => /lib/libuutil.so.1
libgen.so.1 => /lib/libgen.so.1
libm.so.2 => /lib/libm.so.2
</pre>
<p>Anyhow, we need to get the right version of libjtux.so. It seems to be an open-source project and getting a version wouldn't be difficult, but the one in the CrashPlan Solaris installer works fine; let's get that one:</p>
<pre># current version when I ran this
$ wget http://download.crashplan.com/installs/solaris/install/CrashPlan/CrashPlan_2010-03-08_Solaris.tar.gz
$ tar xzf CrashPlan_2010-03-08_Solaris.tar.gz
$ cd CrashPlan/root/opt/sfw/crashplan
$ ls
bin client-build.properties conf doc installer jniwrap.lic lang lib libjtux.so skin upgrade
</pre>
<p>So here we have the entrails of a CrashPlan for Solaris install, and we can pick and choose which organs we need to transplant; and the thing that's particularly needed is libjtux.so; it needs to replace the Linux version.</p>
<p>All that should be left is getting CrashPlan to start at boot-up, which is easiest manually done in Nexenta with the init.d system. Here's the executable crashplan script I have in /etc/init.d:</p>
<pre>
#!/bin/bash
SCRIPTNAME=/usr/local/crashplan/bin/CrashPlanEngine
case "$1" in
start)
$SCRIPTNAME start
;;
stop)
$SCRIPTNAME stop
;;
restart)
$SCRIPTNAME restart
;;
force-reload)
$SCRIPTNAME force-reload
;;
status)
$SCRIPTNAME status
;;
*)
echo "Usage: $0 <start|stop|restart|force-reload|status>" >&2
exit 3
;;
esac
exit 0
</pre>
<p>And the symlink in rc3.d:</p>
<pre>$ ls -l /etc/rc3.d/S99crashplan
lrwxrwxrwx 1 root root 21 Mar 10 2010 rc3.d/S99crashplan -> ../init.d/crashplan
</pre>
<p>That's it!</p>
<a href="http://www.codinghorror.com/blog/2007/03/the-works-on-my-machine-certification-program.html"><img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgNUnUnRwu4JvfduH2dr9F1UDjYdDUsCO54z8IShQ7ITQB9WS5g6FV9ZT-PdKkoh98ewlgNWi2aNloUQq8E8Y0nnRUTrPrw_1Ox7lU3KCtcMff4YwPE-sswkuuFpTRDB6IHMu-O/s1600/works-on-my-machine-starburst.png" style="border:none"></a>Barry Kellyhttp://www.blogger.com/profile/10559947643606684495noreply@blogger.com11tag:blogger.com,1999:blog-10106032.post-80474560769647649032010-09-15T09:51:00.008+01:002010-09-16T00:01:53.642+01:00Non-Delphi: Postmodernism, Transformers: ROTF and Baudrillard<blockquote>
<p>
The simulacrum is never that which conceals the truth--it is the truth which conceals that there is none. The simulacrum is true. (<a href="
http://en.wikipedia.org/wiki/Simulacra_and_Simulation">Baudrillard, Simulacra and Simulation</a>)
</p>
</blockquote>
<p>I've just watched Transformers: Revenge of the Fallen, and it got me thinking. It's perhaps the most post-modern movie I've seen to date.</p>
<p>T:ROTF isn't so much a movie as a string of soundbites, stereotypes and cliches arranged into a 2.5 hour trailer for a movie you'll never get to see, because it was never made. You turn up for the product the advertisement is selling, but it turns out the whole product was ad. There's no "there" there. I've never seen a better embodiment of Baudrillard's conception of simulacrum than this.</p>
<p>The meaningless symbols are so densely packed in this movie I need a convention. Everything in the movie is a stereotype or cliche: a stand-in, an intellectually lazy shorthand reference. I'll be lazy too, and mark stereotypes in my commentary with [brackets].</p>
<p>The beginning is indistinguishable from a trailer. A brief [dawn of man] scene, you know the kind, [stone age people silhouetted against a dawn sky somewhere in Africa], doing [caveman things with the spears and the facepaint], with a [rumbling voiceover] helpfully telling you that it's ["Earth, birthplace of the human race"]. If it were storytelling, it would be rushed, heavy-handed, and contemptuous of the viewer - both showing <b>and</b> telling. But I don't think it is storytelling. It's arranging some symbols (humans, decepticons) into a particular aspect required for later symbolic purposes. The decepticons portrayed in this ancient time are [evil] (with [King Kong-like grabbing] of a feeble human, albeit male), but there is no motive, no narrative. Why would such powerful machines pay any more attention to stone age humans than they would apes, or insects, which they can swat away with similar ease?</p>
<p>Next up: Shanghai, [disaster scene], with [disaster radio news chatter]. Cue [Pentagon command centre], explaining that some black hawks are moving in, while showing some black hawks moving in: Americans aircraft and troops entering Chinese territory, in complete suspension of geopolitical disbelief, no explanation considered necessary. Expository trailer voiceover says "new autobots", while expository camera shot shows new autobots, including [hot girl on bike], [fast car], and [military transport]. "Together, we form an alliance", explains voiceover, while showing human troops in [military transport] (which subsequently transforms). No attempt to explain why squishy soldiers with small arms are going up against fast-moving heavy machinery. What do they hope to achieve with their flying pieces of lead? Would they go up against even a human-engineered tank with such miserable munitions? Nor an explanation for the gunships flying with mere tens of feet clearance from the ground and the surrounding buildings that tower over them, completely negating the tactical advantages of a mobile, hovering cannon and missile platform.</p>
<p>But all is soon revealed. The squishy humans aren't going in to fight, they are going in to be squished, to symbolize human weakness against the machines. After a decepticon slams its fists into some concrete pipe sections, somehow creating a fiery explosion, gunships capable of engaging the enemy with missiles and canons from considerable distance approach low and close enough to be clobbered with a mere wave of mechanical arms. As an alleged depiction of a military engagement, it's beyond ludicrous, laughable on its face. Suspension of disbelief isn't possible: this isn't a battle; it isn't even a simulation of a battle. It's a simulation of battle simulation, an arrangement of symbols of battles. Here are our valiant heroes going into battle; here's our shockingly powerful foe, see how he easily puts our heroes on the back foot; but wait (!) here come our heroes again with reinforcements, to win the day with a bunch of soundbites: ["damn, I'm good!"], ["punk-ass decepticon"], ["any last words?"], "the fallen shall rise again", ["that doesn't sound good"], ["not today!", reload-click, bullet to the head].</p>
<p>That's just the first 8 minutes or so; it goes on for hours (!), with no variation in pacing that you wouldn't also expect in a 30-second movie trailer. <a href="http://io9.com/5301898/michael-bay-finally-made-an-art-movie">Some</a> other <a href="http://scrawledinwax.com/2010/03/07/avatar-was-not-the-most-important-film-of-2009-transformers-2-was/">commentary</a> roughly concurs with mine, though I didn't enjoy the spectacle or visual feast aspects, primarily because those spectacles are filmed too close to the action, and the subjects, transformed machines, have so many bits and bobs hanging out of them it's hard to tell where one begins and another ends, much like how camouflage breaks up outlines. Trying to figure out what's actually going on within the pace of the editing cuts would give me a headache. Besides, marvelling at the sheer density of signifiers and its generally jaw-dropping empty awfulness is more fun, in a perverse way.</p>Barry Kellyhttp://www.blogger.com/profile/10559947643606684495noreply@blogger.com5tag:blogger.com,1999:blog-10106032.post-89780225690591536572010-09-01T12:00:00.004+01:002010-09-06T18:26:06.078+01:00Virtual method interception<p><a href="http://www.embarcadero.com/rad-studio-xe-preview">Delphi XE</a> has a new type in Rtti.pas called TVirtualMethodInterceptor. It was originally designed for use in DataSnap authentication scenarios (though I don't think it's currently being used there), but of course, making it only work for that would have been quite a limitation.</p>
<p>What does it do? Essentially, it creates a derived metaclass dynamically at runtime that overrides every virtual method in the ancestor, by creating a new virtual method table and populating it with stubs that intercepts calls and arguments. When the metaclass reference for any instance of the "ancestor" is replaced with this new metaclass, the user can then intercept virtual function calls, change arguments on the fly, change the return value, intercept and suppress exceptions or raise new exceptions, or entirely replace calling the underlying method. In concept, it's somewhat similar to dynamic proxies from <a href="http://www.codeproject.com/KB/dotnet/dynamicproxy.aspx">.NET</a> and <a href="http://www.devx.com/Java/Article/21463">Java</a>. It's like being able to derive from a class at runtime, override methods (but not add new instance fields), and then change the runtime type of an instance to this new derived class.</p>
<p>Why would you want to do this? Two obvious purposes spring to mind: testing and remoting. Mock objects have been in vogue in the testing space in other languages for some time. By intercepting method calls, one may more easily verify that a particular subsystem is calling all the right methods, with the correct arguments, in the expected order; similarly, the subsystem can proceed with the return values from these method calls, without necessarily having to hit the database, the network, etc. for what should be a unit test. Remoting on the basis of method calls is somewhat less useful, especially when an unreliable and latency-prone network gets into the stack, but that's not the only usage point. The virtual method interceptor logic was originally implemented to be used as part of DataSnap authentication, so that a call that comes in from the network can still be checked as its code flow spreads throughout the graph of objects.</p>
<p>Anyhow, here's a simple example to get started:</p>
<pre>
uses SysUtils, Rtti;
{$apptype console}
type
TFoo = class
// Frob doubles x and returns the new x + 10
function Frob(var x: Integer): Integer; virtual;
end;
function TFoo.Frob(var x: Integer): Integer;
begin
x := x * 2;
Result := x + 10;
end;
procedure WorkWithFoo(Foo: TFoo);
var
a, b: Integer;
begin
a := 10;
Writeln(' before: a = ', a);
try
b := Foo.Frob(a);
Writeln(' Result = ', b);
Writeln(' after: a = ', a);
except
on e: Exception do
Writeln(' Exception: ', e.ClassName);
end;
end;
procedure P;
var
foo: TFoo;
vmi: TVirtualMethodInterceptor;
begin
vmi := nil;
foo := TFoo.Create;
try
Writeln('Before hackery:');
WorkWithFoo(foo);
vmi := TVirtualMethodInterceptor.Create(foo.ClassType);
vmi.OnBefore := procedure(Instance: TObject; Method: TRttiMethod;
const Args: TArray<TValue>; out DoInvoke: Boolean; out Result: TValue)
var
i: Integer;
begin
Write('[before] Calling ', Method.Name, ' with args: ');
for i := 0 to Length(Args) - 1 do
Write(Args[i].ToString, ' ');
Writeln;
end;
// Change foo's metaclass pointer to our new dynamically derived
// and intercepted descendant
vmi.Proxify(foo);
Writeln('After interception:');
WorkWithFoo(foo);
finally
foo.Free;
vmi.Free;
end;
end;
begin
P;
end.
</pre>
<p>Here's what it outputs:</p>
<pre>
Before hackery:
before: a = 10
Result = 30
after: a = 20
After interception:
before: a = 10
[before] Calling Frob with args: 10
Result = 30
after: a = 20
[before] Calling BeforeDestruction with args:
[before] Calling FreeInstance with args:
</pre>
<p>You'll notice that it intercepts all the virtual methods, including those called during destruction, not just the one I declared. (The destructor itself is not included.)</p>
<p>We can get more ambitious with what it does. I can change the implementation entirely, and skip calling the underlying (i.e. inherited) method body:</p>
<pre>
procedure P;
var
foo: TFoo;
vmi: TVirtualMethodInterceptor;
ctx: TRttiContext;
m: TRttiMethod;
begin
vmi := nil;
foo := TFoo.Create;
try
Writeln('Before hackery:');
WorkWithFoo(foo);
vmi := TVirtualMethodInterceptor.Create(foo.ClassType);
m := ctx.GetType(TFoo).GetMethod('Frob');
vmi.OnBefore := procedure(Instance: TObject; Method: TRttiMethod;
const Args: TArray<TValue>; out DoInvoke: Boolean; out Result: TValue)
begin
if Method = m then
begin
DoInvoke := False;
Result := 42;
Args[0] := -Args[0].AsInteger;
end;
end;
</pre>
<p>Here, I inhibit the invocation and hard-code the result to 42, while negating the first argument. The proof is in the output:</p>
<pre>
Before hackery:
before: a = 10
Result = 30
after: a = 20
After interception:
before: a = 10
Result = 42
after: a = -10
</pre>
<p>I could have inhibited the call by raising an exception instead:</p>
<pre>
vmi.OnBefore := procedure(Instance: TObject; Method: TRttiMethod;
const Args: TArray<TValue>; out DoInvoke: Boolean; out Result: TValue)
begin
if Method = m then
raise Exception.Create('Aborting');
end;
</pre>
<p>And output:</p>
<pre>
Before hackery:
before: a = 10
Result = 30
after: a = 20
After interception:
before: a = 10
Exception: Exception
</pre>
<p>It's not limited to interception before the logically inherited call, but also interception after the call, again with the opportunity to fiddle with arguments and return value:</p>
<pre>
m := ctx.GetType(TFoo).GetMethod('Frob');
vmi.OnAfter := procedure(Instance: TObject; Method: TRttiMethod;
const Args: TArray<TValue>; var Result: TValue)
begin
if Method = m then
Result := Result.AsInteger + 1000000;
end;
</pre>
<p>And output:</p>
<pre>
Before hackery:
before: a = 10
Result = 30
after: a = 20
After interception:
before: a = 10
Result = 1000030
after: a = 20
</pre>
<p>And if the inherited implementation raises an exception, that can even be squashed:</p>
<pre>
function TFoo.Frob(var x: Integer): Integer;
begin
raise Exception.Create('Abort');
end;
// ...
m := ctx.GetType(TFoo).GetMethod('Frob');
vmi.OnException := procedure(Instance: TObject; Method: TRttiMethod;
const Args: TArray<TValue>; out RaiseException: Boolean;
TheException: Exception; out Result: TValue)
begin
if Method = m then
begin
RaiseException := False;
Args[0] := Args[0].AsInteger * 2;
Result := Args[0].AsInteger + 10;
end;
end;
</pre>
<p>Output:</p>
<pre>
Before hackery:
before: a = 10
Exception: Exception
After interception:
before: a = 10
Result = 30
after: a = 20
</pre>
<p>One thing the TVirtualMethodInterceptor class doesn't have, however, is a way to unhook (unproxify) the object. If the object is never unhooked, it's important that the object doesn't outlive the interceptor, because the interceptor needs to allocate executable memory in order to create the little stubs with which it redirects method calls to the events. Fortunately, it's pretty trivial to do:</p>
<pre>
PPointer(foo)^ := vmi.OriginalClass;
</pre>
<p>Another point: the class inheritance chain is changed by the hooking process. This can be shown easily:</p>
<pre>
//...
Writeln('After interception:');
WorkWithFoo(foo);
Writeln('Inheritance chain while intercepted:');
cls := foo.ClassType;
while cls <> nil do
begin
Writeln(Format(' %s (%p)', [cls.ClassName, Pointer(cls)]));
cls := cls.ClassParent;
end;
PPointer(foo)^ := vmi.OriginalClass;
Writeln('After unhooking:');
WorkWithFoo(foo);
Writeln('Inheritance chain after unhooking:');
cls := foo.ClassType;
while cls <> nil do
begin
Writeln(Format(' %s (%p)', [cls.ClassName, Pointer(cls)]));
cls := cls.ClassParent;
end;
// ...
</pre>
<p>And output:</p>
<pre>
Before hackery:
before: a = 10
Exception: Exception
After interception:
before: a = 10
Result = 30
after: a = 20
Inheritance chain while intercepted:
TFoo (01F34DA8)
TFoo (0048BD84)
TObject (004014F0)
After unhooking:
before: a = 10
Exception: Exception
Inheritance chain after unhooking:
TFoo (0048BD84)
TObject (004014F0)
</pre>
<p>The feature is primarily an infrastructure piece for advanced libraries, but hopefully you can see that it's not too difficult to get into.</p>Barry Kellyhttp://www.blogger.com/profile/10559947643606684495noreply@blogger.com16tag:blogger.com,1999:blog-10106032.post-57762106598726128512010-08-30T03:53:00.005+01:002010-08-30T05:04:48.755+01:00Memory leaks, log analysis and Unix command-line tools<p>I had cause recently to tackle a fairly tricky memory leak bug in the Delphi compiler as hosted by the IDE. The test case for the bug was complicated: compiling and running a large portion of the IDE itself. After a compile, subsequent debug runs of the IDE kept on increasing the debugging IDE's memory usage.</p>
<p>The memory profiler in <a href="http://www.automatedqa.com/products/aqtime/">Automated QA's AQtime tool</a> provided a very rough indication of the call stack that allocated the memory. The reason it was very rough, and frankly, not particularly useful, was because the Delphi compiler uses a memory suballocator for performance and complexity reasons. Dcc - the Delphi compiler - has roughly 3 different classes of memory allocation: memory associated with a unit, memory associated with a depth in the call stack, and temporary memory that will be done away with periodically, in particular, memory associated with parsing the syntax tree for any one procedure and generating code for it. What the stack from AQtime did tell me was that the memory was (at least initially) associated with unit allocations. Since the compiler reuses memory as units get freed, the initial allocation isn't necessarily the one that gets leaked</p>
<p>To get better insight into what happened around the allocator, I instrumented it. The Delphi compiler in its debug build has a simple logging system. Every logging point has a string associated with it, and by specifying a list of globs in an environment variable, the compiler will output more or less logging info, according to how those globs match the strings associated with each logging point. For example, a log point inside the memory allocator might look like "dcc.memory.alloc", and another like "dcc.memory.free". A glob like "dcc.memory.*" would then match both of these; alternatively, "*.free;*.alloc" would match both of them, but also other logical allocations and frees that use the same naming convention. Whether or not any logging point is matched by the log glob is usually cached in static variables at each logging point, so there's surprisingly little performance loss even from very aggressive logging.</p>
<p>I ended up with a log file which output every new unit allocation, along with all the unit frees, and some supplementary logging relating to how the compiler managed the state of units - information it used to decide whether or not to free the unit, or keep it around for reuse later. It ended up being about 2.4G in size. That's a little big to try and eyeball, so I used some tools to whittle down the interesting bits.</p>
<p>There's a general pattern I use; probably not the most efficient, but it works well for me because it uses the Unix-standard (Cygwin in my case) command-line tools I use most often, rather than other tools that might be more suited but I'm less familiar with. It generally goes like this:</p>
<pre>egrep -o 'interesting regex' log | sed -r 's|.*boring (interesting).*|\1|'
</pre>
<p>That is, it extracts just the interesting lines out, then only the interesting bits of those lines, using extended regex with egrep, and extended regex with a capture in sed. The sed s command is a substitution; I substitute the entire line with the captured interesting bit.</p>
<p>Using this technique, I extracted every allocation address into one temporary output file and every deallocation address into another. I could then do set operations on them to discover allocations that aren't matched by deallocations:</p>
<pre>sort allocs deallocs deallocs | uniq -u > leaks</pre>
<p>Sort will bring all the allocations and deallocations together. Uniq, without any arguments, reduces consecutive duplicate lines down to a single line; but with -u, it removes all lines that are part of consecutive duplicates, and leaves the unique lines. By specifying deallocs twice, I guarantee that no deallocations with an unmatched allocation will appear in the output; only allocations with an unmatched deallocation.</p>
<p>Often this is sufficient. But in this case, the enormous amount of churn, thousands of units, meant that duplicate memory locations were showing up, associated with different units. How did I know this? I did this:</p>
<pre>fgrep -f leaks log | egrep -o 'interesting etc.' ...</pre>
<p>Fgrep searches for fixed strings rather than regexes, which is much faster. The -f argument tells fgrep to load the list of fixed strings from the subsequent file argument, one string per line. This is a quick way of whittling down a log file to only those lines containing one of possibly thousands of strings.</p>
<p>When slightly more processing is needed, a different approach can work, using bash:</p>
<pre>for item in $(<leaks); do fgrep "$item" log | ...; done > output</pre>
<p>That works well if you don't expect $item to contain spaces; bash breaks it up on each whitespace boundary, rather than each line. For lines, I prefer this:</p>
<pre>cat leaks | while read -r line; do ...; done > output</pre>
<p>I use the redundant cat at the start for readability and ease of use; bash would otherwise need the input redirector (<) at the end of the command. The bash command "read" loads a line from standard input into the specified variable. The -r argument prevents it from messing with the string, doing funny things with backslashes, which, in my experience, is almost always undesirable.</p>
<p>Anyhow, to get around the duplicate problem, I changed tack. I modified my filtering so that I had a simple text file consisting of lines like "alloc <address>" and "free <address>" in chronological order of the event, and then wrote a simple program which kept track of each address in a hash table, so that it wouldn't be fooled by reuse of addresses. This got me an unambiguous list of leaked allocations.</p>
<p>Once I had a reliable means of extracting the addresses of the leaks, I scripted the process (not much more than a list of command-lines in a bash script) so that I could easily experiment with deeper and more focused logging. This was the real meat of the process of fixing the bug, and involved a lot of scanning the log file with the "less" pager's search forward (/) and backward (?) commands, tracing the lifetime of various units. But the fact that I knew what to search for, and could whittle down the log file as necessary, was down to what had come earlier from the command-line tools.</p>
<p>I don't know how other programmers tackle these kinds of problems, but these are some of the tools in my toolbox, and they come in surprisingly useful every day. In many ways, debugging is the least talked-about discipline in software engineering, and I think it would be of great benefit to us all to share more of our experiences and techniques.</p>Barry Kellyhttp://www.blogger.com/profile/10559947643606684495noreply@blogger.com6tag:blogger.com,1999:blog-10106032.post-18048523050050727632010-05-25T10:05:00.003+01:002010-05-25T12:39:20.727+01:00Android: Momentum and Apps<p>I'm liking the current momentum behind Android. I'm sure Apple will come out with their new phone soon, and the pendulum will swing a bit, but it's definitely a two horse race, and Apple isn't out of sight, not by a long shot.</p>
<p>I got my Nexus One from <a href="http://www.google.com/phone">http://www.google.com/phone</a> a few months ago, to replace my aging K800i (which still takes better photos than the Nexus One, or the iPhone, or indeed most phones). Since I got my phone directly from Google, it was easy to update to the latest Froyo, which got rid of some of my bigger annoyances on the device. Most tedious in previous versions was the need to individually select and authorize every application when an update showed up. Most days I see between two and five updated applications, and updating any given application takes around 30 seconds, so it was turning into a chore. Froyo is also much smoother in its transitions; it's the way Eclair should have been.</p>
<p>The Android app store (Marketplace) has gotten some criticism, but I don't know how truly meaningful those criticism are, as I've gotten applications primarily based on recommendations, or searching the store with intent, rather than browsing the store. I recently had occasion to count the number of apps I have installed, and if you judge by Android's app management screen, it's now up to 70 (I have 93 icons in my app icon scroller view). I thought I'd list a few of the best ones, as a way of aggregating my own perspective from the different recommended lists I've seen. Some of these applications are not free, but as a developer I don't baulk at paying the small amounts charged, especially since returning for a refund is built into the marketplace if you don't like the app.</p><p>
</p><p><a href="http://www.estrongs.com/en/products/file-explorer.html">ES File Explorer</a> is probably the single application that's most responsible for me preferring Android to iPhone. It's a file browser, but more than that: it can browse Windows (SMB) network shares, as well as FTP and Bluetooth. That means I can copy music, movies etc. on and off my device to and from <a href="http://blog.barrkel.com/2009/03/zfssolaris-as-nas.html">my NAS</a>. I find iTunes to be a <a href="http://blog.barrkel.com/2009/01/ipod-touch-itunes-and-unwanted.html">tedious waste of memory and processes</a>, not to mention it wants to update with huge downloads practically every second week, incorporating media players and web browsers I don't want. Crapware. Being able to take control of the media upload and download experience is wonderful, and makes me feel like my phone is more than a locked-down toy.
</p><p>I complement it with <a href="http://www.metago.net/astro/fm/">Astro File Manager</a>, which has better support for photo management and a built-in image viewer (which supports image view intents, so it can actually be used by ES File Explorer to view photos). I'm not a big fan of Nexus One's stock Gallery app (by <a href="http://www.cooliris.com/mobile/nexus-one/">CoolIris</a>) - lots of gloss, but slow to index photos if you've taken a bunch. Astro can also act as a file selection agent for other applications that browse for an image, such as image editors.</p>
<p>Another photo viewer is <a href="http://gallery.breadandbuttertool.com/">B&B Gallery</a>. It was one of the first to support multi-touch, manually implemented before the OS got support. An advantage it has over the built-in Gallery is that it doesn't downsample loaded photos, so you can zoom in and check the details, rather than quickly getting lost in blur. As a gallery app, however, it's not particularly pretty. I find file management superior, especially as you don't have to wait as thumbnails from all over the place get loaded, but can classify into directories, etc.</p>
<p><a href="http://www.hyperaware.com/android/video-player/">Act 1 Video Player</a> is an excellent replacement for the stock video player. It doesn't add any more decoding capabilities, but it has better affordances in its UI, especially with its touch-based seeking support. Best feature: swiping left and right in the center of the screen seeks back and forth through the video.</p>
<p><a href="http://newsrob.blogspot.com/">NewsRob</a> is an offline RSS reader that synchronizes with Google Reader. It has configurable caching, so you can have it download images which are linked from the RSS (a problem with Google Reader's own offline version using Gears, ironically), up to caching the full web page associated with that RSS item. Excellent for public transport.</p>
<p>A major annoyance with the iPod Touch (and also the iPhone) for me was the auto-rotation. I almost never want to rotate the view, and it always ends up rotating when I'm lying down, or otherwise not in front of a computer. I was only able to solve this problem on the iPod Touch by jailbreaking it. Android has a setting for this, but for easier access to it I use the <a href="http://curvefish.com/widgets/autorotate-onoff.htm">AutoRotate widget</a>. This lets you put a 1x1 widget anywhere which toggles the auto-rotate setting on a tap.</p>
<p>Some games are useful for passing idle moments. <a href="http://lupislabs.blogspot.com/">Robo Defense</a> is quite addictive tower defense with RPG-like elements; you earn what are essentially XP points, and can spend them on incremental upgrades, so there's a campaign-like aspect to the gameplay. <a href="http://www.replicaisland.net/">Replica Island</a> is a classic-style platformer which is particularly ergonomic on the Nexus One, using the scroll ball for directional control. As an aside, controls are one of the weakest elements of most iPhone games - it badly needs more physical buttons. And <a href="http://nesoid.com/">Nesoid</a>, an NES emulator, is nice in principle, but a better control system is needed.</p>
<p>Artistic diversions: <a href="http://www.doodledroid.net/">DoodleDroid</a> is a finger-painting app with configurable brush dynamics, so with care, you can get some interesting impressionistic images out of it. Simpler, more like coloured markers than paint, is <a href="http://www.thewongandonly.com/draw">Draw!</a>.</p>
<p>Of course, there are the bar code apps, like <a href="http://www.biggu.com/">ShopSavvy</a>, probably the most integrated when you have buying intent, though its local shop search isn't very localized, even when in London; <a href="http://code.google.com/p/zxing/">ZXing Barcode Scanner</a>, which runs more general web searches based on barcodes; Google Shopper and Google Goggles also do barcodes, but I feel they're weaker, and Goggles is mostly a gimmick (IMO).</p>
<p>Google Sky Map is pretty neat - the way it uses the accelerometer to overlay constellations etc. is probably the neatest augmented reality-style implementation I've seen, even though it doesn't overlay on a video image from the camera. Layar is the probably the canonical implementation, but I find it to be too gimmicky in practice, having to walk around like an idiot with a phone held out in front of you. At least with stars, you're normally standing still and looking into the sky.</p>
<p>Google Translate is another essential app. It's tantalizingly close to real-time speech to speech translation; as it is, you can speak into it and at a button press do text to speech on the translation, providing the speech recognition was good. My girlfriend tells me it can be overly literal for German, however.</p>
<p><a href="http://sites.google.com/site/farproc/wifi-analyzer">Wifi Analyzer</a> helped me get better channel placement on my home wifi access points. Really neat live view of signal strength for all the different APs in your area, even ones too faint to actually connect to.</p>
<p><a href="http://code.google.com/p/arity-calculator/">Arity</a> is a simple expression-based calculator which can graph simple non-parametric functions in two and three dimensions. By non-parametric, I mean you give it an expression using x, or x and y, and it plots the result of the expression as y, or z, in a 2D plane or 3D volume. You can't plot circles with it, for example.</p>
<p><a href="http://code.google.com/p/connectbot/">ConnectBot</a> is a SSH client, useful for remote administration when you're <b>really</b> stuck for connectivity. Doing anything serious on the command line without access to a keyboard is insanity, of course. When the job you're trying to do is simpler - a single command over SSH - <a href="http://projects.openobjectives.de/projects/serverassistant">ServerAssistant</a> is a better approach.</p>
<p>If you're interested in programming your life, <a href="http://www.twofortyfouram.com/">Locale</a> can trigger events based on conditions. Conditions are one or more of location, time, orientation, calls by contacts and battery state. Settings include wallpaper, ringtone, screen brightness, wifi enabled or not, volume, bluetooth, but also actions published by third-party applications. For example, NewsRob can synchronize based on a Locale trigger. And if you've installed <a href="http://code.google.com/p/android-scripting/wiki/UserGuide">ASE, the Android Scripting Environment</a>, you can run arbitrary scripts - bash, python, ruby, etc. - on a trigger. The sample scripts available for ASE include invoking text to speech to say the time and the current weather, toggling airplane mode, showing notifications, etc. Locale is a lot less useful if you have a more flexible schedule, but if you're tied in to a timetable, it makes a lot of sense.</p>
<p>Finally, a battery widget: <a href="http://preinvent.com/batteryleft">Battery Left</a>. I don't use task managers or killers; I've found that it's better to let Android do its thing and kill what it needs to kill, when it chooses to do it. I get about 46 hours on average battery, but I tend to recharge before 36 hours have gone past. You can drop this widget as a 1x1 (or 2x1) visual indicator of battery left, with configurable detailed textual data: estimated time before battery dead, estimated time of day of dead battery, estimated battery %, etc. It monitors battery performance, so it should straighten the curve that batteries self-report - I've often seen batteries say they have three-quarters battery for ages, and then run out the remainder quite suddenly, etc.</p>
<p>Obviously, I have many more applications installed than I've mentioned here, but they tend to be single-purpose location-based ones that have less general applicability, or ones I don't use as often and can't in good conscience recommend. But I can say that all of the above work pretty well for me, and it's notable that many of them would contravene Apple's developer policy, so for me at least, app availability for the iPhone isn't the killer advantage it's made out to be.</p>Barry Kellyhttp://www.blogger.com/profile/10559947643606684495noreply@blogger.com4tag:blogger.com,1999:blog-10106032.post-8816334362650052552010-05-11T18:21:00.003+01:002010-05-11T18:42:43.271+01:00Locations vs Values: using RTTI to work with value types<p>Delphi's Rtti unit is designed in substantial part around TValue, a kind of hold-all record that should be capable of containing almost any Delphi value, along with type information for that value. However, this means that when you're working with value types, such as static arrays and records, modifying the values when stored in a TValue is modifying that copy, stored inside the TValue. If you want to manipulate a field (F1) of a record which is itself a field (F2) of another type, you need to first copy the F2 field's value out into a TValue, then modify F1 in the TValue, and then copy it back in to the original F2 field.</p>
<p>As an aside: TValue.MakeWithoutCopy does not relate to this copying behaviour, but is rather for managing reference counts with strings and interfaces and other managed types. This is particularly important when marshalling parameters to and from stack frames, where logical copies sometimes should be made, and sometimes not.</p>
<p>However, working with values in TValue all the time is not necessarily the most efficient technique. By adding another layer of indirection, we can improve things: instead of working with values, we can work with locations.</p>
<p>This can be encapsulated fairly trivially using the current RTTI support. I hacked up a TLocation type which represents a typed location analogously to how TValue represents a value:</p>
<pre>
type
TLocation = record
private
FLocation: Pointer;
FType: TRttiType;
public
class function FromValue(C: TRttiContext; const AValue: TValue): TLocation; static;
class function FromAddress(ALocation: Pointer; AType: TRttiType): TLocation; static;
function GetValue: TValue;
procedure SetValue(const AValue: TValue);
function Follow(const APath: string): TLocation;
function Dereference: TLocation;
function Index(n: Integer): TLocation;
function FieldRef(const name: string): TLocation;
end;
</pre>
<p>For ease of use, it uses TRttiType. If it were to be fully as flexible as TValue, it would use PTypeInfo instead, like TValue does. However, using the RTTI wrapper objects makes life a lot easier.</p>
<p>Here it is in use:</p>
<pre>
type
TPoint = record
X, Y: Integer;
end;
TArr = array[0..9] of TPoint;
TFoo = class
private
FArr: TArr;
constructor Create;
function ToString: string; override;
end;
{ TFoo }
constructor TFoo.Create;
var
i: Integer;
begin
for i := Low(FArr) to High(FArr) do
begin
FArr[i].X := i;
FArr[i].Y := -i;
end;
end;
function TFoo.ToString: string;
var
i: Integer;
begin
Result := '';
for i := Low(FArr) to High(FArr) do
Result := Result + Format('(%d, %d) ', [FArr[i].X, FArr[i].Y]);
end;
procedure P;
var
obj: TFoo;
loc: TLocation;
ctx: TRttiContext;
begin
obj := TFoo.Create;
Writeln(obj.ToString);
ctx := TRttiContext.Create;
loc := TLocation.FromValue(ctx, obj);
Writeln(loc.Follow('.FArr[2].X').GetValue.ToString);
Writeln(obj.FArr[2].X);
loc.Follow('.FArr[2].X').SetValue(42);
Writeln(obj.FArr[2].X); // observe value changed
// alternate syntax, not using path parser
loc.FieldRef('FArr').Index(2).FieldRef('X').SetValue(24);
Writeln(obj.FArr[2].X); // observe value changed again
Writeln(obj.ToString);
end;
</pre>
<p>Here's most of the implementation:</p>
<pre>
{ TLocation }
type
PPByte = ^PByte;
function TLocation.Dereference: TLocation;
begin
if not (FType is TRttiPointerType) then
raise Exception.CreateFmt('Non-pointer type %s can''t be dereferenced', [FType.Name]);
Result.FLocation := PPointer(FLocation)^;
Result.FType := TRttiPointerType(FType).ReferredType;
end;
function TLocation.FieldRef(const name: string): TLocation;
var
f: TRttiField;
begin
if FType is TRttiRecordType then
begin
f := FType.GetField(name);
Result.FLocation := PByte(FLocation) + f.Offset;
Result.FType := f.FieldType;
end
else if FType is TRttiInstanceType then
begin
f := FType.GetField(name);
Result.FLocation := PPByte(FLocation)^ + f.Offset;
Result.FType := f.FieldType;
end
else
raise Exception.CreateFmt('Field reference applied to type %s, which is not a record or class',
[FType.Name]);
end;
function TLocation.Follow(const APath: string): TLocation;
begin
Result := GetPathLocation(APath, Self);
end;
class function TLocation.FromAddress(ALocation: Pointer;
AType: TRttiType): TLocation;
begin
Result.FLocation := ALocation;
Result.FType := AType;
end;
class function TLocation.FromValue(C: TRttiContext; const AValue: TValue): TLocation;
begin
Result.FType := C.GetType(AValue.TypeInfo);
Result.FLocation := AValue.GetReferenceToRawData;
end;
function TLocation.GetValue: TValue;
begin
TValue.Make(FLocation, FType.Handle, Result);
end;
function TLocation.Index(n: Integer): TLocation;
var
sa: TRttiArrayType;
da: TRttiDynamicArrayType;
begin
if FType is TRttiArrayType then
begin
// extending this to work with multi-dimensional arrays and non-zero
// based arrays is left as an exercise for the reader ... :)
sa := TRttiArrayType(FType);
Result.FLocation := PByte(FLocation) + sa.ElementType.TypeSize * n;
Result.FType := sa.ElementType;
end
else if FType is TRttiDynamicArrayType then
begin
da := TRttiDynamicArrayType(FType);
Result.FLocation := PPByte(FLocation)^ + da.ElementType.TypeSize * n;
Result.FType := da.ElementType;
end
else
raise Exception.CreateFmt('Index applied to non-array type %s', [FType.Name]);
end;
procedure TLocation.SetValue(const AValue: TValue);
begin
AValue.Cast(FType.Handle).ExtractRawData(FLocation);
end;
</pre>
<p>To make it slightly easier to use, and slightly more fun for me to write, I also wrote a parser - the Follow method, which is implemented in terms of GetPathLocation:</p>
<pre>
function GetPathLocation(const APath: string; ARoot: TLocation): TLocation;
{ Lexer }
function SkipWhite(p: PChar): PChar;
begin
while IsWhiteSpace(p^) do
Inc(p);
Result := p;
end;
function ScanName(p: PChar; out s: string): PChar;
begin
Result := p;
while IsLetterOrDigit(Result^) do
Inc(Result);
SetString(s, p, Result - p);
end;
function ScanNumber(p: PChar; out n: Integer): PChar;
var
v: Integer;
begin
v := 0;
while (p >= '0') and (p <= '9') do
begin
v := v * 10 + Ord(p^) - Ord('0');
Inc(p);
end;
n := v;
Result := p;
end;
const
tkEof = #0;
tkNumber = #1;
tkName = #2;
tkDot = '.';
tkLBracket = '[';
tkRBracket = ']';
var
cp: PChar;
currToken: Char;
nameToken: string;
numToken: Integer;
function NextToken: Char;
function SetToken(p: PChar): PChar;
begin
currToken := p^;
Result := p + 1;
end;
var
p: PChar;
begin
p := cp;
p := SkipWhite(p);
if p^ = #0 then
begin
cp := p;
currToken := tkEof;
Exit(currToken);
end;
case p^ of
'0'..'9':
begin
cp := ScanNumber(p, numToken);
currToken := tkNumber;
end;
'^', '[', ']', '.': cp := SetToken(p);
else
cp := ScanName(p, nameToken);
if nameToken = '' then
raise Exception.Create('Invalid path - expected a name');
currToken := tkName;
end;
Result := currToken;
end;
function Describe(tok: Char): string;
begin
case tok of
tkEof: Result := 'end of string';
tkNumber: Result := 'number';
tkName: Result := 'name';
else
Result := '''' + tok + '''';
end;
end;
procedure Expect(tok: Char);
begin
if tok <> currToken then
raise Exception.CreateFmt('Expected %s but got %s',
[Describe(tok), Describe(currToken)]);
end;
{ Semantic actions are methods on TLocation }
var
loc: TLocation;
{ Driver and parser }
begin
cp := PChar(APath);
NextToken;
loc := ARoot;
// Syntax:
// path ::= ( '.' <name> | '[' <num> ']' | '^' )+ ;;
// Semantics:
// '<name>' are field names, '[]' is array indexing, '^' is pointer
// indirection.
// Parser continuously calculates the address of the value in question,
// starting from the root.
// When we see a name, we look that up as a field on the current type,
// then add its offset to our current location if the current location is
// a value type, or indirect (PPointer(x)^) the current location before
// adding the offset if the current location is a reference type. If not
// a record or class type, then it's an error.
// When we see an indexing, we expect the current location to be an array
// and we update the location to the address of the element inside the array.
// All dimensions are flattened (multiplied out) and zero-based.
// When we see indirection, we expect the current location to be a pointer,
// and dereference it.
while True do
begin
case currToken of
tkEof: Break;
'.':
begin
NextToken;
Expect(tkName);
loc := loc.FieldRef(nameToken);
NextToken;
end;
'[':
begin
NextToken;
Expect(tkNumber);
loc := loc.Index(numToken);
NextToken;
Expect(']');
NextToken;
end;
'^':
begin
loc := loc.Dereference;
NextToken;
end;
else
raise Exception.Create('Invalid path syntax: expected ".", "[" or "^"');
end;
end;
Result := loc;
end;
</pre>
<p>The principle can be extended to other types and Delphi expression syntax, or TLocation may be changed to understand non-flat array indexing, etc.</p>
<p>This post was inspired by <a href="http://stackoverflow.com/questions/2802864/rtti-accessing-fields-and-properties-in-complex-data-structures">this question</a> on Stack Overflow, and some similar questions to it that popped up over the past few weeks.</p>Barry Kellyhttp://www.blogger.com/profile/10559947643606684495noreply@blogger.com5tag:blogger.com,1999:blog-10106032.post-37055452810776073342010-04-19T11:19:00.008+01:002010-04-19T12:51:53.558+01:00Programming font for VS2010<p>So, Visual Studio 2010 shipped. Now I have a problem: what font to use? My old standby, <a href="http://www.donationcoder.com/Software/Jibz/Dina/">Dina</a>, is a bitmap font so it doesn't work with the WPF text editor in VS2010.</p>
<p>Some things I look for in a programming font:</p>
<ul>
<li>High information density - vertical height in particular. For example, I use Dina at 8pt.
<li>Crisp, even lines
<li>Strong distinction between bold and regular (for lexical highlighting)
<li>Clearly unambiguous letters: l,1,0,O, and the like.
<li>Balanced operators - good placement for <, >, {, }, etc.
</ul>
<p>Here's Dina 8pt in VS2008:</p>
<img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjFgy4BMMmIbynYOVUzd69EeRWfHHcOBdu_MeY41c96dR15aZ6rX81j-p_gJZ5Xgx1J1ZmTdJ0XHqcBys9MwdRPD3l4UV2JH4PIkOpoOMkZpBs0OLsOjPIdpDfvd4z1MeVWShjw/s1600/dina.png">
<p>Here's Consolas at 8pt in VS2010:</p>
<img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgVRoTcfPELjEMd_pJEulC099Av_D2nABzR98WuX1IuieuVqlKy3XimtfRxSySJv5r5J1d08K8LqXxfKTo2ZnxbVlE55Eo-uqnvFVuAlVdNueMslWhm2_brKAL8fyCVSOUvHZ-N/s1600/consolas-8.png">
<p>(At least, I <b>think</b> that's 8pt. I have zoomed the editor in and out a little with Ctrl+Mouse Wheel, but there doesn't seem to be a zoom reset...</p>
<p>Here's Consolas 9pt in VS2010:</p>
<img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjTGHfZkwD43vvB_Dsfa4YZgzJ1PAsfySLAH5cgYSBC5kZjH8uFTKD8EVhanrULAt4SANWI-Lcmg7z_XMaoTgtFJCuQjt6MuzMzUHald_3j3nFhuIMcnPIW0FdN_sepY5RsLtBy/s1600/consolas.png">
<p>Consolas is quite constrained on the horizontal. It looks horrible at 8pt, and is still slightly taller than Dina 8pt. Even at 9pt though, to my eyes, it suffers quite badly with comparison to Dina. The bold of the comment and digits is almost unnoticeable. 'M' and 'm' are very weak, probably because of the narrowness. The vertical on the 'F' of 'Func' is weak, as is the vertical on 'r'. These are in part properties of ClearType subpixel rendering, which looks slightly different from monitor to monitor and depending on configuration, but even with the configuration selected for its heaviest rendering, it looks poor on my system. Overall, the effect is washed out and slightly blurry.</p>
<p><a href="http://www.proggyfonts.com/index.php?menu=download">Proggy Fonts</a> are another contender. These have TTF hinted versions of what amount to bitmap fonts, so they try and sneak older rendering in through the back door. Here's Proggy Clean Slashed Zero with Bold Punctuation (Visual Studio can bold operators (and I do - and make them white, to stand out even more), but it doesn't include [], {}, () as operators, unfortunately):</p>
<img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgTO4SxMa2Dl3mqpzk6PKKGXSgYIP96xIWANS9Ba-Mnt6-ttuAdKYXAlo8FAPCR2oGMXY2J3QNzlmB34I-4mO9p05D-On9tR15jblFzIDcrFMjO4YrG_dyFwAWVQ1F28OWCe_Oh/s1600/proggy-clean-szbp.png">
<p>This is probably the most palatable option for me, but it's still not quite as good as Dina. It's the same height, probably because Dina was based on Proggy; but the things I don't like about it most are things that Dina fixed, in particular the oddly elongated < and >, the lazy looking 's', and the "gappy" look of the bold - contrast the comment in Proggy vs Dina. Other differences incude the leg on the 'R' and the general way the capitals are slightly overly broad. Another oddity is however ClearType has interacted with its hinting, the font has ended up tinted with green.</p>
<p>Proggy Clean SZBP is the font I'll stick with for the moment, until I see a better option, or a better translation of Dina to TTF than <a href="http://www.geenat.com/?p=66">this one</a>, which is tuned for 10pt Dina, not the size I use.</p>Barry Kellyhttp://www.blogger.com/profile/10559947643606684495noreply@blogger.com16tag:blogger.com,1999:blog-10106032.post-61281128796062214902010-03-23T22:10:00.002+00:002010-03-23T22:15:50.963+00:00JEDI newsgroup change<p>I got word that the JEDI newsgroup changed after the old forums.talkto.net server died. The new server is at:</p>
<pre>news.delphi-jedi.org
</pre>
<p>That's nntp on standard port 119. Please update your favorite news reader to this new server.</p>Barry Kellyhttp://www.blogger.com/profile/10559947643606684495noreply@blogger.com0tag:blogger.com,1999:blog-10106032.post-85273705189431529632010-03-13T17:35:00.004+00:002010-03-13T18:31:09.179+00:00CrashPlan for Backup on Nexenta<p>For some time, I've been using <a href="http://en.wikipedia.org/wiki/Cron">cron</a> jobs with <a href="http://rdiff-backup.nongnu.org/">rdiff-backup</a> on Cygwin for backups. The cron job runs on a Windows server I have on my home network and iteratively mounts remote Windows shares (if they're present) and runs rdiff-backup over them, with the destination also being local. Then the job runs rsync to mirror this backup to my <a href="http://blog.barrkel.com/2009/03/zfssolaris-as-nas.html">Nexenta NAS running ZFS raidz</a>.</p>
<p>This solution gives me a fair amount of local redundancy: two separate machines, with two copies of the backup data, plus the RAID-5-like redundancy and checksumming integrity that ZFS provides. Rdiff-backup is quite reassuring with respect to restores too: it stores files in the filesystem directly, along with differences (the rdiff bit) so you can go back in time. That means that restoring is as simple as copying the files straight out of the backup and deleting the rdiffs.</p>
<p>Of course, a backup strategy isn't solid without a remote copy. Today, I finished configuring <a href="http://www.crashplan.com/">CrashPlan</a>, a really neat backup solution built using Java. You can read about the features etc. of CrashPlan on the website - I heard about it from the <a href="http://javaposse.com/index.php?post_id=586968">Java Posse podcast, episode 298</a>. Initially, I opened an <a href="https://s3.amazonaws.com/">Amazon S3</a> account, and was considering mirroring my backups to S3 with <a href="http://s3sync.net/">s3sync</a>, but after I evaluated CrashPlan, it looked like it made more sense than further pursuing my homegrown approach. Not only is it cheaper than S3 for my data (I guess they depend on overselling like ISPs), but CrashPlan has features that make it work well in my case.</p>
<p>The fact that it's primarily Java meant I could install it on my Nexenta box. CrashPlan don't have an installer for Nexenta, but they do have ones for Linux (LSB) and Solaris, and Nexenta is like a hybrid. I'm documenting the steps here in case I need to set it all up again, though hopefully not for a restore.</p>
<p>I'm running NexentaCore NCP 2, which doesn't come with Java. Installing a JDK (overkill, but I wanted to test the Java environment with a simple hello-world):</p>
<pre>$ sudo apt-get install sun-java6-jdk
</pre>
<p>I was running a release candidate of Nexenta and I had to do an apt-get update and upgrade to resolve all necessary dependencies, but it was fairly stress free because Nexenta's apt-clone checkpointed the root file system with a ZFS snapshot. I did lose my almost half-year of uptime though - my Nexenta machine has been the most dependable of all my machines, in both hardware and software.</p>
<p>Installing CrashPlan itself needs bits from both the Linux and Solaris installers. I downloaded both, and unpacked both. In the Linux install, I ran the provided install.sh and followed the steps, putting it in /usr/local, with daemon init script in /etc/init.d and runlevel symlink in /etc/rc3.d. But the Linux install isn't enough. In particular, the init script assumes GNU ps, but Nexenta uses Solaris ps. So I swapped in CrashPlanEngine (the target of the init script symlink) from the Solaris installer in the place of the installed CrashPlanEngine that was here:</p>
<pre>/usr/local/crashplan/bin/CrashPlanEngine
</pre>
<p>But that wasn't enough. CrashPlan loads a library called <a href="http://basepath.com/aup/jtux/">libjtux</a> via <a href="http://en.wikipedia.org/wiki/Java_Native_Interface">JNI</a>, a kind of POSIX API for Java programmers who want direct access to the OS. The libjtux.so from the Linux install was linking against libc.so.6, assuming GNU C library versioning. Replacing libjtux.so with the version from Solaris, linking against plain libc.so, solved this problem - here:
<pre>/usr/local/crashplan/libjtux.so
</pre>
<p>Finally, I had to configure CrashPlan. My Nexenta install is headless - all my interaction with it is either over Samba shares, HTTP for wiki servers etc., and of course ssh for everything else. Here's one of the really neat features of CrashPlan: the user interface for configuration is a client that depends on a single TCP connection with the local backup server. All I had to do is get the client to connect to a different local port, and tunnel that port to Nexenta using SSH, after enabling TCP forwarding in Nexenta in /etc/ssh/sshd_config. This bit is described on CrashPlan's site describing <a href="http://support.crashplan.com/doku.php/how_to/configure_a_headless_client">how to configure a headless client</a>.</p>
<p>I'm running CrashPlan+, a for-pay version of the engine, on two machines - my main desktop and my Nexenta box - to get all the encryption, compression, deduplication etc. goodness. And considering that CrashPlan supports peer to peer backup, I may simply replace my existing ad-hoc rdiff-backup approach with local CrashPlan backups, as CrashPlan supports multiple destinations for backups, as well as receiving backups from other machines.</p>
<p>A key limitation of CrashPlan, and one that I found particularly annoying, is that the Windows client doesn't support network shares at all, in any shape or form - whether they're mapped to drive letters or not. The backup engine runs under the SYSTEM account, so it doesn't have network credentials, and also means it may not be able to access all the files you're trying to back up - especially EFS encrypted files. (CrashPlan doesn't seem to use Windows' EFS backup capability, e.g. <a href="http://msdn.microsoft.com/en-us/library/aa365466(VS.85).aspx">ReadEncryptedFileRaw</a>.) I changed the CrashPlan engine's service account to my own account to try and close this hole, but still no go on accessing network shares. This meant that I <b>had</b> to get CrashPlan running on my Nexenta box in order to store local backups on ZFS, should I so choose.</p>
<p>But apart from that, I've been impressed with CrashPlan's feature set and usability.</p>Barry Kellyhttp://www.blogger.com/profile/10559947643606684495noreply@blogger.com5tag:blogger.com,1999:blog-10106032.post-24085170988875268452010-02-03T19:35:00.003+00:002010-02-03T19:53:42.480+00:00Multidimensional Separation of Concerns, or Matrix Inheritance<p>I just spotted <a href="http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.43.7026">a paper</a> that I didn't know existed before. Abstract:</p>
<blockquote>
<p>Done well, separation of concerns can provide many software engineering benefits, including reduced complexity, improved reusability, and simpler evolution. The choice of boundaries for separate concerns depends on both requirements on the system and on the kind(s) of decomposition and composition a given formalism supports. The predominant methodologies and formalisms available, however, support only orthogonal separations of concerns, along single dimensions of composition and decomposition. These characteristics lead to a number of well-known and difficult problems.</p>
<p>This paper describes a new paradigm for modeling and implementing software artifacts, one that permits separation of overlapping concerns along multiple dimensions of composition and decomposition. This approach addresses numerous problems throughout the software lifecycle in achieving well-engineered, evolvable, flexible software artifacts and traceability across artifacts.</p>
</blockquote>
<p>I came up with a remarkably similar concept I called matrix inheritance in a previous job. Here's one description I've written up out there on the web:</p>
<blockquote>
<p>[This is a description of a symptom of the problem.] Nested if-then-else can introduce non-essential complexity merely by having dead nested cases which are actually obviated by outer cases; looking at a complex version of this code, it can become quite difficult to see where exactly the code is supposed to flow under what circumstances, as there can seem to be conflicting assumptions in different areas.</p>
<p>Once upon a time I invented a scheme to solve this problem in the best way I thought possible, and called it "matrix inheritance". The problem with inheritance and subclassing is that it only handles a single dimension of customization. Suppose you have two dimensions, genre and type, such as [comedy, drama] and [movie, series]. If you were to try and classify any given thing under a classical type breakdown, you could subclass by one axes or the other, but you would need to duplicate subclassing for the remaining axes. So, you could end up with Programme, Comedy <: Programme, Drama <: Programme, but then you'd need ComedyMovie, ComedySeries, DramaMovie, DramaSeries, duplicating the kind axis in the two different branches.</p>
<p>The matrix inheritance concept basically takes the cartesian product of all option axes, essentially modelling it as an n-dimensional space, and then applies behaviour to sub-spaces of this space. So, you could apply conditions to [drama,*] and [*, series], with these two representing slices of the example 2-dimensional space described above. The advantage of modelling things this way is that it is declarative: you can analyse overlaps and identify un-covered space.</p>
</blockquote>
<p><a href="http://compilers.iecc.com/comparch/article/04-08-117">Here's the original post</a> I made on comp.compilers when looking for related work. Here's a brief excerpt:</p>
<blockquote>
<p>Object-oriented techniques have been rejected because they constrain
me to using a fixed order of dimensions for reuse and overriding. A
possible solution presents itself:</p>
<p>1. Creating a notional Cartesian-product of all dimensions of
customization.</p>
<p>2. Overriding based on the application of rules whose scope is defined
by sets described by a simple language: by example, '{a,b}' represents
the set containing a and b, while '*' represents the set containing
all the values for that dimension.</p>
</blockquote>Barry Kellyhttp://www.blogger.com/profile/10559947643606684495noreply@blogger.com4tag:blogger.com,1999:blog-10106032.post-80422714449274315172010-01-29T19:18:00.006+00:002010-01-29T22:04:35.491+00:00One-liner RAII in Delphi<p>Some C++ aficionados point at the ability, in C++, to create an object such that the creation allocates some kind of resource, and the destruction frees the resource, with much glee. The idiom is RAII: Resource Acquisition Is Initialization. The technique can be a useful, albeit somewhat opaque, way of acquiring a resource for the duration of a block of code and relinquishing the resource upon exiting the block. It's C++'s equivalent of Delphi / Java / C# / SEH's try / finally blocks (though of course C#'s "using" statement is an even closer analogue).</p>
<p>Delphi has a similar mechanism in the form of deterministic reference counting for interfaces. This can be (ab)used to functionally implement almost exactly the same technique as C++. The degree to which it is useful, however, is moderated by readability issues; RAII doesn't necessarily employ a visible nesting, and rather depends on the implied nesting of the scope in which it's used.</p>
<p>Anyway, here's a way of implementing RAII in Delphi, for people to whom the technique may not have occurred yet. It relies on defining a general-purpose class which implements an interface, and whose death will be triggered by scope exit:</p>
<pre>
type
TScopeExitNotifier = class(TInterfacedObject)
private
FProc: TProc;
public
constructor Create(const AProc: TProc);
destructor Destroy; override;
end;
constructor TScopeExitNotifier.Create(const AProc: TProc);
begin
FProc := AProc;
end;
destructor TScopeExitNotifier.Destroy;
begin
if Assigned(FProc) then
FProc;
inherited;
end;
function MakeScopeExitNotifier(const AProc: TProc): IInterface;
begin
Result := TScopeExitNotifier.Create(AProc);
end;
</pre>
<p>I would generally recommend hiding the actual class in the implementation-section of a unit - the functional interface of MakeScopeExitNotifier ought to be sufficient. Using this function, we can now implement a one-liner RAII equivalent of the following code:</p>
<pre>
Writeln('Disable controls on ', {ASurface});
try
// (user code)
finally
Writeln('Enable controls on ', {ASurface});
end;
</pre>
<p>Of course, I'm using strings etc. for simplicity, so that the demo translation is compilable. Here is the definition of the function which will be called to form the one-liner:</p>
<pre>
function DisableEnableControls(const ASurface: string): IInterface;
begin
Writeln('Disable controls on ', ASurface);
Result := MakeScopeExitNotifier(procedure
begin
Writeln('Enable controls on ', ASurface);
end);
end;
</pre>
<p>And here it is in action:</p>
<pre>
procedure P;
begin
DisableEnableControls('Foo');
DisableEnableControls('Bar');
Writeln('Controls on Foo and Bar are disabled now');
end;
</pre>
<p>Because of the way Delphi's deterministic reference counting and destruction works, Foo and Bar will be released in reverse order of construction, as expected for RAII. Functions that return an interface value in Delphi are implemented behind the scenes as procedures that take an out Result parameter; these are in effect hidden variables that keep the return values alive until the scope is exited.</p>
<p>The output of the program is as follows:</p>
<pre>
Disable controls on Foo
Disable controls on Bar
Controls on Foo and Bar are disabled now
Enable controls on Bar
Enable controls on Foo
</pre>
<p>Of course, it's not necessarily clear what DisableEnableControls(string) would do, and how it protects the scope. That's why I wouldn't necessarily recommend using this technique, except to counter C++-ite arguments. Using anonymous methods directly would seem to be a more readable approach in general, something along these lines:</p>
<pre>
procedure DisableEnableControls(const ASurface: string; const ACB: TProc);
begin
Writeln('Disable controls on ', ASurface);
try
if Assigned(ACB) then
ACB;
finally
Writeln('Enable controls on ', ASurface);
end;
end;
procedure P;
begin
DisableEnableControls('Foo', procedure
begin
DisableEnableControls('Bar', procedure
begin
Writeln('Controls are disabled now.');
end);
end);
end;
</pre>
<p>Output is the same as earlier. The extra complexity / verbosity of this abstraction over and above a literal try/finally would of course be more justified if the logic in the place of Enable and Disable sections were more involved, such as possibly setting up and rolling back transactions, etc.</p>
<p>However, the functionality of MakeScopeExitNotifier() routine is actually useful in the context of anonymous methods, because by capturing a variable containing the return value, you can get informed as to when the scope that created your anonymous method dies. This can be useful to e.g. free any objects that were allocated when the anonymous method was created:</p>
<pre>
function FreeOnExit(const Args: array of TObject): IInterface;
var
toFree: TArray<TObject>;
begin
// copy open array to toFree because open arrays can't be captured
SetLength(toFree, Length(Args));
Move(Args[0], toFree[0], Length(Args) * SizeOf(Args[0]));
Result := MakeScopeExitNotifier(procedure
var
i: Integer;
begin
Writeln('FreeOnExit is freeing objects');
for i := 0 to High(toFree) do
toFree[i].Free;
end);
end;
procedure Use(const x: IInterface);
begin
end;
function Q: TProc;
var
onDie: IInterface;
obj: TObject;
begin
obj := nil;
onDie := FreeOnExit([obj]);
Writeln('Allocating an object');
obj := TObject.Create;
Result := procedure
begin
Use(onDie); // cause onDie to be captured
Writeln(Format(
'Inside callback, obj still alive: %p: %s',
[Pointer(obj), obj.ToString]));
end;
end;
procedure P;
var
r: TProc;
begin
Writeln('Before Q');
r := Q();
Writeln('Invoking result of Q');
r;
Writeln('Clearing result of Q');
r := nil;
Writeln('Result is cleared');
end;
</pre>
<p>The output is something like:</p>
<pre>
Before Q
Allocating an object
Invoking result of Q
Inside callback, obj still alive: 00970D20: TObject
Clearing result of Q
FreeOnExit is freeing objects
Result is cleared
</pre>
<p>This technique, capturing a variable of an interface or other deterministically lifetime-managed type, is essential for flexibility in cleaning up any resources that an anonymous method needs to for its lifetime. Note, however, that there can be a gotcha: the following alternative, using MakeScopeExitNotifier directly, will <b>not</b> work:</p>
<pre>
// <BROKEN CODE DO NOT USE>
function Q: TProc;
var
onDie: IInterface;
begin
onDie := MakeScopeExitNotifier(procedure
begin
Writeln('died!');
end);
Result := procedure
begin
Use(onDie);
end;
end;
// </BROKEN CODE DO NOT USE>
</pre>
<p>The reason it's broken is that the anonymous method that's handed off to MakeScopeExitNotifier <b>will itself keep the scope alive</b>. So in order to make use of MakeScopeExitNotifier for this particular purpose, the anonymous method passed to it needs to be in a different scope.</p>Barry Kellyhttp://www.blogger.com/profile/10559947643606684495noreply@blogger.com14tag:blogger.com,1999:blog-10106032.post-28592108871479563982010-01-20T18:12:00.002+00:002010-01-20T18:31:08.649+00:00Using anonymous methods in method pointers<p>Anonymous methods may have associated state. In particular, all variables that an anonymous method captures need to be kept alive so long as the anonymous method is callable. For this reason, anonymous methods are implemented with a lifetime management approach: anonymous methods are actually methods on an object which implements COM-style reference counted interfaces. Method references are interface references with a single method called Invoke.</p>
<p>If one badly wants to store an anonymous method in a method pointer, this information can be used to shoehorn it in. Here's how it can be done:</p>
<pre>
procedure MethRefToMethPtr(const MethRef; var MethPtr);
type
TVtable = array[0..3] of Pointer;
PVtable = ^TVtable;
PPVtable = ^PVtable;
begin
// 3 is offset of Invoke, after QI, AddRef, Release
TMethod(MethPtr).Code := PPVtable(MethRef)^^[3];
TMethod(MethPtr).Data := Pointer(MethRef);
end;
</pre>
<p>This procedure will take a method reference as the first argument, and then extract the two crucial pieces of data needed to put into a method pointer: the value to be passed as the first argument (the method, or interface, reference), and the code address to be called (the value of the Invoke entry in the interface's vtable).<p>
<p>The procedure can be used as follows. Note that the method reference still needs to be kept alive somewhere as long as the method pointer is callable to avoid premature disposal of the heap-allocated object:</p>
<pre>
type
TMeth = procedure(x: Integer) of object;
TMethRef = reference to procedure(x: Integer);
function MakeMethRef: TMethRef;
var
y: Integer;
begin
y := 30;
Result := procedure(x: Integer)
begin
Writeln('x = ', x, ' and y = ', y);
end;
end;
procedure P;
var
x: TMethRef;
m: TMeth;
begin
x := MakeMethRef();
MethRefToMethPtr(x, m);
Writeln('Using x:');
x(10);
Writeln('Using m:');
m(10);
end;
</pre>
<p>On the other hand, if the anonymous method's body never captures any variables, then it's not necessary to keep the method reference around. This loses much of the benefits of anonymous methods, but it does prove the point:</p>
<pre>
uses SysUtils, Forms, StdCtrls, Dialogs, classes;
procedure MethRefToMethPtr(const MethRef; var MethPtr);
type
TVtable = array[0..3] of Pointer;
PVtable = ^TVtable;
PPVtable = ^PVtable;
begin
// 3 is offset of Invoke, after QI, AddRef, Release
TMethod(MethPtr).Code := PPVtable(MethRef)^^[3];
TMethod(MethPtr).Data := Pointer(MethRef);
end;
type
TNotifyRef = reference to procedure(Sender: TObject);
function MakeNotify(const ANotifyRef: TNotifyRef): TNotifyEvent;
begin
MethRefToMethPtr(ANotifyRef, Result);
end;
procedure P;
var
f: TForm;
btn: TButton;
begin
f := TForm.Create(nil);
try
btn := TButton.Create(f);
btn.Parent := f;
btn.Caption := 'Click Me!';
btn.OnClick := MakeNotify(procedure(Sender: TObject)
begin
ShowMessage('Hello There!');
end);
f.ShowModal;
finally
f.Free;
end;
end;
begin
try
P;
except
on e: Exception do
ShowMessage(e.Message);
end;
end.
</pre>Barry Kellyhttp://www.blogger.com/profile/10559947643606684495noreply@blogger.com2tag:blogger.com,1999:blog-10106032.post-85205294545170456512010-01-07T19:34:00.002+00:002010-01-07T19:59:54.937+00:00Delphi 2010 RTTI Contexts: how they work, and a usage noteDelphi 2010 includes extended support for RTTI, also known as run-time type info or reflection. Many design approaches that have previously only been possible with managed languages like C# and Java because of the code annotation and introspection they required should now be possible in the Delphi world.</p>
<p>Something somewhat interesting about how the RTTI works is its approach to object pools. Delphi isn't a garbage collected language, so users need to be careful to free objects when they're no longer needed, either explicitly, or by designing or using some kind of ownership scheme, such as that used by TComponent, where the Owner takes care of destruction.</p>
<p>Type information usage scenarios don't mesh particularly well with a TComponent-style of ownership. Typically, when working with RTTI, you want to do some kind of search for interesting objects, do something with them, and then go on your way. That means that many objects may get allocated for inspection, but not actually be used. Managing those objects' lifetimes independently would be tedious, so a different approach is used: there is a single global RTTI object pool. While there is at least one <b>RTTI context</b> active in the application, this object pool keeps all its objects alive. When the last context goes out of scope, the objects get freed up.</p>
<p>The pool management works by using a Delphi record that contains an interface reference. The first time any given RTTI context is used, it fills in this interface reference. It can't fill it in any later than first use, because Delphi records don't support default constructors, which besides have their own problems. For example, how do you handle exceptions in default constructors, in all the places they can occur? Allocating arrays, thread-local variables, global variables, global variables in packages, temporary objects created in expressions, etc. It can get ugly, and in C++ it sometimes does.</p>
<p>So, this first use allocates an interface, called a pool token. It acts as a reference-counted handle to the global object pool. For so long as this interface is alive, the the global object pool should stay alive. Even if the RTTI context is copied around, Delphi's built-in interface handling logic, designed along COM principles, will ensure that the interface doesn't gets disposed of prematurely or get its reference count muddled up. And when an RTTI context goes out of scope, either by being a local variable in a function that is exited, or a field in an object that is freed, the reference count is reduced. When it hits zero, the pool is emptied.</p>
<p>The biggest upside of this approach is that RTTI usage should feel reasonably cheap, conceptually speaking. Code need only declare a variable of the appropriate type, and start using it:</p>
<pre>
procedure Foo;
var
ctx: TRttiContext;
t: TRttiType;
begin
t := ctx.GetType(TypeInfo(Integer));
Writeln(t.Name);
end;
</pre>
<p>A downside, however, is that lazy initialization can create a gotcha. Imagine this scenario:</p>
<ol>
<li>Library A declares an RTTI context A.C
<li>User code B declares an RTTI context B.C
<li>B pulls some RTTI objects O out of B.C, in order to hand them to library A
<li>B.C goes out of scope
<li>Library A now tries to work with O, but discovers much to its surprise, that the objects have been prematurely disposed, even though A already has an RTTI context, A.C
</ol>
<p>The problem is that A never used A.C, so it never allocated a pool token. When B.C used its context, the pool came into being, and objects O were assigned to it; but after B.C went out of scope, the objects got freed.</p>
<p>The solution to this problem is for Library A, knowing that it has a long-lived RTTI context and it expects to communicate with third-party code which allocates objects from its own RTTI context and hands them back, it should ensure that the long-lived context's pool token is allocated. A trivial way to do this is like this:</p>
<pre>
type
TFooManager = class
FCtx: TRttiContext;
// ...
constructor Create;
// ...
end;
constructor TFooManager.Create;
begin
FCtx.GetType(TypeInfo(Integer));
// ...
end;
</pre>
<p>This will allocate only a bare minimum of RTTI objects, those needed to represent the type System.Integer, but more importantly, will ensure that FCtx has a pool token and will keep the global RTTI pool alive.</p>
<p>In future releases of Delphi, the static method TRttiContext.Create will make sure that its return value has a pool token allocated; currently, it does not. TRttiContext.Create was originally defined to make the TRttiContext record feel more like a class for people unfamiliar with the idiom of using interfaces for automated deterministic lifetime management. The corresponding TRttiContext.Free method disposes of the internal pool token, and should remain the same.</p>Barry Kellyhttp://www.blogger.com/profile/10559947643606684495noreply@blogger.com1tag:blogger.com,1999:blog-10106032.post-19928244714126270632009-12-29T03:48:00.002+00:002009-12-29T04:45:46.966+00:00Playing DVDs in Windows 7<p>Oh, how I pine for the blissful days of Windows XP, when menus were still in fashion, the start menu didn't contain a tree view, applications didn't require a half-dozen clicks to launch (to expand the tree view, you see), and the user's colour theme choices were respected. Better days, when the customer was still king.</p>
<p>Lately I've been having difficulty playing back DVDs on my Windows 7 PC. My preferred media player, mplayerc aka Media Player Classic - and its close cousin, the more recently maintained Media Player Classic Home Cinema, failed with the error message "DVD: copy-protection failed". (I quote the message and versions in detail to increase the chances a search will find this page.) Windows Media Player has similar issues, though since they've hidden the menus and generally "simplified" the WMP UI, I can no longer figure out how to use it.</p>
<p>From my research and experimentation, it appears that ever since Windows Vista, RPC1 (aka region-free) DVD drives are no longer welcome. Windows will detect the drive as being region-free, and prevent DVD video playback. The only commonly-found player that still works is VLC, if you can stand its broken UI. I cannot: for example, try full-screen playback (F11) with hidden controls (Ctrl+H) in VLC; you'll find that hiding the controls breaks the full-screen toggle, F11! I've had to flash my DVD drive with a region-locked firmware just to play DVDs. Workarounds exist, such as using auto-reset firmwares if they exist for the drive, or what I'm probably going to do, use a separate DVD drive for each region, since DVD drives are exceedingly cheap - less than 20 GBP - and I only need two regions, Europe (2) and North America (1). (I note with slight puzzlement that DVD drives seem to be slightly cheaper in the UK than in the US; not normally the case.)</p>
<p>One thing does really puzzle me, though. All the error messages say that <b>copy protection</b> failed, when as far as I know region locking has nothing whatsoever to do with copy protection, but is rather about inhibiting global trade. With some irony, one of the workarounds I used to watch DVDs was to rip them and play from the file server across the network.</p>Barry Kellyhttp://www.blogger.com/profile/10559947643606684495noreply@blogger.com12tag:blogger.com,1999:blog-10106032.post-34359423757191932342009-12-14T16:01:00.005+00:002009-12-14T17:42:26.128+00:00Commonly Confused Tidbits re .NET Garbage Collector<p>I recently read a less than informed blog post purporting to describe the .NET garbage collector, but when pressed to point to a better one, I couldn't find a single concise article that covered everything I thought should have been covered.</p>
<h4>Type and Memory Safety</h4>
<p>We all know what the gist of GC is about: you allocate new objects, but don't have to worry about freeing them. Combined with type safety in the language, this can give you another kind of guarantee, <a href="http://en.wikipedia.org/wiki/Memory_safety">memory safety</a>: that it's impossible to create a dangling pointer (a pointer to freed memory). Type safety's "safety" is compromised in the absence of memory safety. For example, in a type-safe yet non-GC language, if you allocate an object, make a copy of the reference, then free the object, you're left with a dangling pointer. If you then reallocate another object with a similar size, depending on the memory allocator's implementation, you may have actually created a type hole - the dangling pointer may now point to the newly allocated object but with the wrong type. <b>This extra safety is one of the reasons why GC programs often have less bugs than ones using manual allocation.</b></p>
<h4>Precise compacting GC</h4>
<p>.NET has a precise compacting GC. It's precise because .NET deals with type-safe and memory-safe references into the managed heap, it can know the precise layout of every allocated object. The opposite is conservative GC; conservative GC doesn't necessarily know the types of variables stored on the stack or in registers, or the layouts of those types, so if a random number on the stack or in a register or the middle of an object happens to coincide with a valid object address, it can't collect that object. It's compacting because during a collection, it moves live (non-garbage) objects closer together, squeezing out the gaps left by dead, old objects which no longer have any references to them. This eliminates the problem of memory fragmentation, at least as far as the managed heap is concerned.</p>
<p>However, thinking of a precise compacting GC as "collecting garbage" is not always the best way. When looking at the asymptotic performance of GC, it's helpful to turn things around, and consider that the GC collects live objects and brings them together. Garbage is just the stuff that got overwritten during moving, or the stuff left at the end after everything has been compacted together. Why is this important? Because the cost of a precise compacting garbage collection is proportional to the number of <b>live</b> objects it has to trace through and move, rather than the amount of garbage it needs to collect. When compared with manual allocation, where the cost of freeing memory is usually proportional to the number of objects freed (i.e. the amount of garbage), something should become clear: the longer you can delay GC - i.e. the more garbage you can build up - the faster and cheaper, in per-garbage terms, GC is than manual allocation.</p>
<p>An example. Imagine a single linked list with 1000 elements, with a single variable in the program referring to the head of the list. Consider the problem of truncating the list, and leaving only the head behind. With manual memory allocation, all 999 trailing items will need to be disposed of, with bookkeeping for each one. With garbage collection, only the first element need be moved to the start of the heap, and the free pointer (where new allocations begin) set to begin directly after it. The 999 other elements got "collected" for free.</p>
<p>This is why <b>the performance of precise compacting GC is proportional to the amount of garbage it is allowed to build up between collections</b>, and hence proportional to the amount of memory it's allowed to consume. It's also the reason why GC programs tend to use more memory than manual allocation ones. It's also the reason why small single-use objects, such as for function return values, are very cheap and shouldn't be unduly avoided in GC languages. Congruent with this, code in a programming language with GC is often more naturally written, easier to read and easier to write because of the freedom with which one can return freshly allocated objects without concern for who is going to collect them, or the performance ramifications. <b>Understanding this point is key to understanding why GC is so productivity-enhancing.</b></p>
<h4>Resources</h4>
<p><b>The garbage collector is for memory, not resources.</b> <a href="http://blog.barrkel.com/2009/01/jeff-is-wrong-and-dont-listen-to-him.html">I've written about this at length in the past</a>, so I shall quote myself:</p>
<blockquote>
<p>Disposal of resources in a long-running application using GC is not a performance issue. It's a correctness issue.</p>
<p>Garbage collection knows about memory. With most collectors, GC is only invoked when the GC is asked for more memory than is immediately available, taking specific tuning parameters into account. In other words, the GC is <b>only sensitive to memory pressure</b>. It doesn't know about resources that it sees only as pointer-sized handles, it doesn't know how much they "cost", and indeed those resources might be on a different machine or even spread across many different machines.</p>
<p>More critically, garbage collection gets its performance surplus over and above manual garbage collection by <b>not collecting</b> until as late as reasonably possible</p>
</blockquote>
<p>In .NET, the deterministic disposal mechanism is via the IDisposable interface, and in languages like C#, the <b>using</b> keyword.</p>
<h4>Finalizer vs IDisposable</h4>
<p>Finalizers in .NET are overrides of the method System.Object::Finalize(void). In C#, the syntax looks like a C++ destructor, but other .NET languages have other syntaxes. If this method is overridden, the .NET GC treats the object specially. After it has become garbage, rather than just overwriting it during compaction phase of GC, it will delay collection until the Finalize method of the instance has been executed by the finalizer thread. The GC makes a best effort to call the method; if the finalizer thread is blocked (e.g. some other finalizer hung), then the finalizer might not get called. This is another reason why deterministic disposal is preferred to relying on finalizers.</p>
<p>Finalizers should only be used with objects that directly encapsulate a non-managed resource, such as a file handle, a window handle, a socket, etc. A file stream, or a window object, or a network stream, should not normally implement a finalizer. IDisposable is a different matter. IDisposable should be implemented by every class with a finalizer, and every class that logically owns (transitively) any other object that implements IDisposable. Calling IDisposable on the root of the ownership tree should ripple through and call IDisposable on everything beneath it; ownership trees and other deterministic lifetime approaches still exist with GC precisely because GC is just for memory, not resources. Most objects neither own nor encapsulate resources, however, so much of the burden is still lifted.</p>
<h4>Generations</h4>
<p>I've read many posts that describe how generational GC works: very loosely, you have three generations, gen0 through gen2; allocations come from gen0; gen0 is collected if it's full; gen1 is collected if gen0 collection didn't free enough space; gen2 is collected if gen1 didn't free enough; more memory is allocated if gen2 didn't free enough; if not enough memory, then out of memory exception.</p>
<p>They also describe the hypothesis behind generational collections: that young objects are more likely to die (become garbage) sooner rather than later, while old objects are likely to survive longer.</p>
<p>But this, however, doesn't capture the intuition behind why there are 3 generations, rather than 4, or 5, or some other number. Let me explain.</p>
<p>Most applications have a roughly constant, steady state of total allocated memory. It's made up of two kinds of objects: objects associated with currently executing code (let's call them ephemeral objects), and objects linked into a global object graph which is shared across multiple instances and iterations of executing code (let's call them persistent objects). We want to avoid tracing through the persistent objects as much as possible, as it's unlikely that any objects which have survived for a long time have become garbage. <b>Actually, making old objects garbage is a pathology to be avoided in .NET programs, and <a href="http://blogs.msdn.com/ricom/archive/2003/12/04/41281.aspx">has been described as the mid-life crisis problem</a>.</b></p>
<p>The ephemeral objects, on the other hand, are likely to have a high proportion of garbage. But we have a problem: when we perform a garbage collection on the ephemeral objects, there'll be some live objects left over, currently being used by executing code - but we still don't want them in with the persistent objects. To solve this problem, there needs to be a generation in the middle, a half-way house, before resorting to increasing the persistent object set.</p>
<p>This combination of three generations is almost ideally suited to server applications which have relatively little variance in total per-request allocation and tend towards an average per-request allocation which is less than the size of gen0+gen1 per thread. If one could time per-thread collections of gen0 and gen1 correctly, in between requests, then in theory GC would be almost completely free, as there would be almost no live objects. There are complications in practice (we can't manually invoke a GC for just a single thread, for one thing), but when designing applications and servers, it's very much worth keeping the generational design of the GC in mind and working with its assumptions, rather than against them.</p>
<p>For example, this can mean that caching objects can turn out to be a pessimization rather than an optimization, because keeping them around moves them into gen2, where it is more expensive to collect them when they get evicted from the cache than it was to create them in the first place. Depending on how expensive the objects are to create vs cache, they may need to be cached permanently and updated as needed, or referred to through a WeakReference which won't keep them alive, or at worst managed manually one way or another, such as via unmanaged memory, or slots or ranges in an array, etc.</p>Barry Kellyhttp://www.blogger.com/profile/10559947643606684495noreply@blogger.com14tag:blogger.com,1999:blog-10106032.post-54279116445296112112009-11-10T16:59:00.007+00:002009-11-10T17:54:08.710+00:00On the Difficulty of setting the Windows 7 Desktop Wallpaper<p>You need to be a software engineer to precisely and correctly set the Windows background in Windows 7.</p>
<p>I have a multi-monitor setup, and I use a very particular method for laying out windows on my secondary screen: I have them cascaded, from the top-left corner of the screen towards the bottom right, such that a little of the bottom-left corner of every window is visible on-screen at all times, like this:</p>
<img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjqkoBGliw9J63_cB3wUZoLU6BlXPY1ah12gHv3v6fZFVaXjt4YK9lu6Ubw4ch7CJSMgFo0KZHqxNQDjKqxjPTqdBwvRrHhPnDbGTd_-wRXOudh6hShHBMYlu-js0-7ltsN6Xit/s1600/Cascade.png">
<p>The corners form a mechanism for switching between windows based on spatial layout. Certain windows, such as terminals, browsers, email apps, etc. have "natural" dimensions. Browsers generally like to be page-shaped; terminal windows are quite squat; documentation browsers are somewhere in between, with a treeview on the left and a content pane on the right. These natural shapes guide their spatial positioning. So, all my terminal windows end up cascaded one after the other on at the top left of the screen, followed by Firefox's downloads window, then my bug tracker client, then Thunderbird, then MSDN documentation, with Firefox at the bottom-right, filling the full height of the screen.</p>
<p>Switching between applications based on their spatial layout turns out to usually be more efficient than almost any other scheme, particularly when one's hands aren't on the keyboard. Even alt-tab doesn't distinguish very clearly between different instances of e.g. a terminal app, when in fact each one may be logged in via ssh to different machines and thus it matters very much which one I select.</p>
<p>This scheme works well enough, but the bottom-left corners can get a little ragged. To make aligning the corners easier, I have a special wallpaper which consists of a line carefully placed so that it shows up on the bottom-left corner of my second monitor. When the wallpaper mode is set to "tile", a wallpaper which is big enough to cover the entire desktop area will show up at the expected offset, and everything works out OK.</p>
<p>This wallpaper, a 16-colour bitmap, is so small when compressed that I can embed it here uuencoded directly:</p>
<pre>
begin 644 wallpaper2.bmp.bz2
M0EIH.3%!629365%3W2<`1PST[.0U0`!`0!`"`0!``!``0`!$``,(L`#9`0JC
M4#1HR::%&C(&C3(T"E4TAM30`&=?M"6=)(E^T$2WA+9$L*JDF-E$NG?'AI55
M0/>6^L(CB*T(550`Q-.DJ".(B"O,*JH`8F(WU0$<@^P((\>[801S_$;<:YG'
:LR**SC<GG'.8JJ@!M,13_%W)%.%"045/=)P`
`
end
</pre>
<p>If you save that to a file with a '.uue' extension, many common archivers, such as WinZip or WinRar, will be able to open it up. Here's what the meat of it looks like, anyway:</p>
<img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjTBUY4nUsfCVnDB-naHn1BnCEIfF51TdE2Qch_C9kWTxXoF0GBQKVuaMhFub-McIglI5_v_jJD8g_nmTEYqfAN5SPFMyJQGYGf2LUqHputJnlZcgq76_8ViIoKOH-I0tkJih_B/s1600/Cancade-Line.png">
<p>However, it turns out, if you try to set this background using the "Personalization" Control Panel applet in Windows 7, it does some helpful "conversions" on it: it appears to round-trip the damn bitmap through JPG, resulting in artefacts. I've enhanced a zoom of the line after Windows has butchered it:</p>
<img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhcDy6yAaRnMqfKD0dTBo85DqxQcFIYLi_Ja-KXx1PRV6NWpzvfZRpYYO_AdkKCI2GSXWlOPu6yZYLfwaKJ5V66sske6QIAZDC2ZueNav0q193XTicToiWHgI5eD2fm3AP0DFbx/s1600/Crap-Highlight.png">
<p>The line looks unclean, and in particular, it ends up with wave of intensity throughout its length, which is quite distracting and not the desired effect.</p>
<p>Things can get worse though; if you try to use Windows Photo Viewer to set the background using the original bitmap, it will actually resize the thing down to fit the dimensions of your primary monitor (!!!), all but zooming the line out of existence. This drove me up the wall for about 20 minutes, as I tried to figure out where my line had gone when I first transitioned from XP to Windows 7.</p>
<p>The easiest way I currently know to correctly set the Windows 7 desktop wallpaper is to write a program that uses the <a href="http://msdn.microsoft.com/en-us/library/ms724947%28VS.85%29.aspx">SystemParametersInfo function</a> directly, to ensure the bits get through unmolested.</p>
<p>I have a lot more Windows 7 annoyances where this came from, but this one cost me quite a bit of time, so it sticks out. My verdict on Windows 7 is pretty mixed. Many applications, particularly games, are incompatible, and overall I dislike the shell compared to XP, especially Windows Explorer and the Vista-esque Start Menu. About the best things I can say in favour of the move is having less concern about future compatibility, and having a 64-bit mode that mostly works, despite all the horrific hacks going on beneath the covers to make it happen. Oh, and I'm running with UAC disabled. I can only take so many access denied errors as I poke around the system before I crack - and command-line apps generally just give you the error, rather than popping up an elevation prompt.</p>Barry Kellyhttp://www.blogger.com/profile/10559947643606684495noreply@blogger.com5tag:blogger.com,1999:blog-10106032.post-20055435949027780222009-10-24T23:27:00.002+01:002009-10-25T00:28:05.662+01:00Bad week<p>It's been a bad week. My main hard drive crashed last Monday or so, while today my scooter is gone, presumed stolen. On the bright side, the final component of my new PC arrived and I was able to get Windows 7 up and running.</p>
<p>The new machine has an i7 920, so things like %NUMBER_OF_PROCESSORS% returns 8 - it's a quad core with 2x hyperthreading. Having so many logical cores has some downsides: maxed-out single-threaded processes only take up 12.5% CPU, so it can look like the machine is not particularly busy even though a process is working as hard as it can. Other specs include 12GB RAM, AMD 5870 graphics, 80GB Intel SSD and 1TB Samsung F3.</p>
<p>Maximizing the value of the small boot drive has been interesting. Windows 7 all by itself takes up a good 18G of space, what with the side-by-side (SxS) feature keeping lots of versions of the same APIs, and a big hiberfil.sys in the root - I moved the page file to the big drive. Because of the acute space constraints, I've been strongly motivated to keep almost all the junk that apps usually dump into the file system off the boot drive. By and large, I achieve that by creating directories on the 1TB drive and pointing to them with junctions on the boot drive. That way, apps think they are installing onto the C drive but they don't steal too much space. My prior drive - the one that died - was a 10K WD Raptor, so I've had some experience with this, and I wrote a script to automate the process:</p>
<pre>
#!/bin/bash
. _die
function usage
{
echo "usage: $(basename $0) <directory>..."
echo "Replace directories with junctions pointing to /bulk, after moving contents."
exit 1
}
test -n "$1" || usage
for f in "$@"; do
test -d "$f" || die "'$f' isn't a directory"
dest="$(abs-path "$f")"
dest=/bulk/mathom/"${dest//\//_}"
test -d "$dest" && die "'$dest' already exists"
# Try to copy stuff over; if an error, delete target and give up.
cp -r "$f" "$dest" || {
rm -rf "$dest"
die "couldn't copy '$f' to '$dest'"
}
# Rename source, in case something can't be deleted.
aside="$(move-aside "$f")" || die "failed in move-aside"
# Create junction.
junction "$dest" "$f" || die "failed to create junction"
# Remove the aside.
rm -rf "$aside" || {
echo "warning: couldn't delete '$aside'"
}
done
</pre>
<p>This script uses a number of little wrapper utilities I wrote - at this point I really ought to be thinking of putting them in an online repository. Here's a summary so you can follow along:
<p>
<ul>
<li><b>abs-path:</b> Canonicalizes and makes an absolute path by using cygpath twice, once to Windows format and once again back to Unix format, and prepending $PWD if necessary.</li>
<li><b>_die:</b> Defines a <b>die</b> function which prints out appropriate error and exits.</li>
<li><b>move-aside:</b> Renames a file or directory so that a failed attempt at creating a junction can be rolled back. Also helps with locked files - generally these can be copied and renamed, but not deleted. The new name is printed out on standard output.</li>
<li><b>junction:</b> This wraps <a href="http://technet.microsoft.com/en-us/sysinternals/bb896768.aspx">SysInternals junction</a> to create a junction in the TARGET LINK_NAME argument order familiar from <b>ln</b>, and permits using Cygwin paths.</li>
</ul>
</p>
<p>With this script, and the big drive mounted at /bulk with a directory called <b>mathom</b> in its root, any given directory tree can be moved across readily enough. However, things in Windows 7 x64 are slightly more complicated.</p>
<p>Some of the directories I moved over to /bulk/mathom included Program Files, Program Files (x86), and ProgramData. Windows 7 itself has some junctions set up (the rats nest here is a proper horrorshow, I should document it at some point just for my own use, if nothing else). For example, "ProgramData\Application Data" points at ProgramData itself, and a simple cmd /c dir will fall into the infinite recursion and print out errors when the file path gets too long. So, in moving these directories across, I had to examine them for existing junctions and replace these as necessary. The Cygwin tools of Unix heritage are more robust against cycles in the file system hierarchy - for example, <b>find</b> keeps track of inodes to avoid infinite recursion.</p>
<p>All this checking of existing junctions wasn't sufficient to get everything working nicely, though. The permissions generally also need tweaking, particularly for folders that get modified by installers. These generally perform operations using either the NT SERVICE\TrustedInstaller or NT AUTHORITY\SYSTEM accounts - and sometimes both, in a dance of setup.exe and msiexec.exe. So, by process of checking, trial and error, using <a href="http://technet.microsoft.com/en-us/sysinternals/bb896645.aspx">SysInternals Process Monitor</a> to track installation failures, I put together a new script to ensure Program Files, ProgramData and folders of their ilk have the right set of permissions:</p>
<pre>
#!/bin/bash
function usage
{
echo "usage: $(basename $0) <file/directory>..."
echo "Resets files and directories to Windows 7 general install permissions."
echo "Essentialy this is: SYSTEM/TrustedInstaller/Admin: full; AuthUsers: Modify; Users: Read"
exit 1
}
test -e "$1" || usage
for arg; do
# First, take ownership so we can definitely set all ACLs.
# /F <file>; /R - recurse; /D <Y/N> - take ownership if no directory listing
takeown /F "$(cygpath -w "$arg")" /R /D Y > /dev/null
# Reset ACLs recursively to inherit from parent
icacls "$(cygpath -w "$arg")" /reset /t /c /q > /dev/null
# Set appropriate ACL on root, to progate recursively
icacls "$(cygpath -w "$arg")" \
/grant "Administrators:(OI)(CI)F" \
/grant "SYSTEM:(OI)(CI)F" \
/grant 'NT SERVICE\TrustedInstaller:(OI)(CI)F' \
/grant 'Authenticated Users:(OI)(CI)M' \
/grant 'Users:(OI)(CI)RX' \
/inheritance:r /c /q > /dev/null
done
</pre>
<p>The logic is applied in three phases:</p>
<ol>
<li>Take ownership, with takeown</li>
<li>Reset permissions recursively so that they inherit from parent</li>
<li>Set ACL on the root, so children will inherit from it, but make sure the root itself doesn't inherit permissions.</li>
</ol>
<p>I tried for a while to get it all into one icacls invocation, but I couldn't get the two effects working together: enabling inheritance in the children, but disabling it in the root. This is something you can do in one "Apply" step using the Security tab in Explorer, though it's quite tedious picking out all the users and setting the appropriate checkboxes.</p>
<p>As an aside, the Security tab is quite confused by junctions; when permissions have been inherited from a directory on a different drive, because the current file / directory is in a subtree pointed to by a junction, the Security tab will search in vain through the parents looking for what provided the given inherited permission. Sometimes it gets lucky, and finds a random parent that coincidentally has the same permission; otherwise, it bails out with "Parent Object" as the indicated "Inherited From" column. But my proper list of Windows 7 issues will have to wait until I really feel the need to rant, which will no doubt happen sooner or later.</p>
<p>I should add that all the drastic file system surgery I write about here was performed with UAC effectively out of the picture, and all Cygwin tools running with the administrative privilege bits set. It's been my experience that command-line apps don't pop up the required UAC prompt; instead, they fail with an access denied message, which is rather vexing when you're trying to automate things.</p>Barry Kellyhttp://www.blogger.com/profile/10559947643606684495noreply@blogger.com1tag:blogger.com,1999:blog-10106032.post-13559937187392940752009-07-15T13:15:00.002+01:002009-07-15T13:21:20.305+01:00Public interest rule of thumb<p><a href="http://www.techcrunch.com/2009/07/15/our-reaction-to-your-reactions-on-the-twitter-confidential-documents-post/">TechCrunch</a> is having a crisis of conscience over what they'll do with internal Twitter documents they've received.</p>
<p>My rule of thumb is pretty simple. The documents are stolen. Unless the documents reveal wrongdoing greater than stealing, such that the public interest served in publishing is greater than the private harm, then it's unethical to publish. Idle musings about "they are going to be published somewhere on the Internet" is specious; the morality of one's actions don't depend on whether someone else is "doing it", much less everyone.</p>
<p>If TechCrunch goes ahead and publishes these documents, it'll only further cement my already pretty low opinion of Arrington. (I'm not a habitual reader of TechCrunch, or otherwise I'd waste away over all the food I'd be vomiting up.)</p>Barry Kellyhttp://www.blogger.com/profile/10559947643606684495noreply@blogger.com6tag:blogger.com,1999:blog-10106032.post-63994632085543117512009-07-14T19:45:00.002+01:002009-07-14T19:55:00.552+01:00StackOverflow and religious language in programming<p>
I was reading <a href="http://unhandled-exceptions.com/blog/index.php/2009/07/13/why-i-dont-listen-to-the-stackoverflow-podcast-any-more/">Why I Don’t Listen to the StackOverflow Podcast any More</a>, and in reading the comments, I found it interesting to see the amount of religious language.
</p>
<blockquote>
#21 (sbohlen) I’d have to agree that they do indeed appear successful, but success / failure isn’t a boolean [...]
The world is full of people who succeed due to all kinds of factors including dumb-luck, [...]
</blockquote>
<p>
This reminds me intensely of "the race is not always to the swift" etc. <a href="http://www.biblegateway.com/passage/?search=Ecclesiastes%209:11&version=9;">Eccl 9:11</a>.
</p>
<p>I changed adopters to believers to accentuate the effect:
<blockquote> "#16: [...] I have also experienced that the non-[believers] aren’t (gnerally) non-[believers] out of choice but are instead non-[believers] largely out of ignorance and inexperience — they just don’t know what they don’t know [...]"
</blockquote>
<blockquote>
#16: "the [types] who aren’t interested in self-improvement; there’s not a damned thing I can suggest to help these people — I cannot MAKE someone want to better themselves, that’s gotta come from within"
</blockquote>
<blockquote>
#16: "I [...] think that Jeff and Joel are indeed preaching [...] they are (IMO) abdicating the unofficial professional responsibility that comes with having a pulpit from which to preach."
</blockquote>
<blockquote>
#14: "I am a strong believer"
</blockquote>
<blockquote>
#13: "@sbohlen. I fully agree with the sentiment that “up and running” is not a valid metric"
</blockquote>
<p>
I'm not going to go on with this, though I could - the <a href="http://unhandled-exceptions.com/blog/index.php/about/">about page for the blog</a> is particularly ripe for further examples of religious language - but I do want to make a further point.
</p>
<p>
Could it be that latent religious feelings are responsible for a good portion of the minor angst on display in these comments? Could it be that people are unconsciously annoyed, not with StackOverflow's success, but rather its success even in the face of quite mild contempt at the unit-testing / SOLID religion?
</p>
<p>
My own position: I'm a "believer" in evolution and the market. Neither evolution nor the market have any moral content (they can't say what "ought" to be), and both only measure success by effects rather than processes. In a phrase, <b>"up and running" is not just a valid metric - it's the only metric, until it stops running</b>. If SO continues to thrive, and doesn't die under the weight of its lack of unit testing (which seems to be the logical end result of the beliefs of the believers in the comments), where then is the truth in the proclamations of doom from the prophets of the true religion?
</p>
<p>
I don't think unit-testing is the only way to avoid long-term bit-rot. I think it's a good way, but not the only way; moreover, I don't think unit testing is especially useful in the very early stages of a project, where the design changes often and rewriting unit tests becomes a disincentive to larger refactoring and redesign. I think every one of the concepts embodied in S.O.L.I.D. is debatable in certain scenarios (except perhaps LSP).
</p>
<p>
In my experience, premature abstraction has been responsible for a similar number of ills to premature optimization: abstractions chosen at the wrong boundaries, for the wrong reasons, because designers thought they could foretell the future and anticipate where changes would come. Thus I am wary of merchants of abstractions and patterns, principles and practices. I'll use them when appropriate, but never with religious zeal.
</p>Barry Kellyhttp://www.blogger.com/profile/10559947643606684495noreply@blogger.com2