Android: Momentum and Apps

I'm liking the current momentum behind Android. I'm sure Apple will come out with their new phone soon, and the pendulum will swing a bit, but it's definitely a two horse race, and Apple isn't out of sight, not by a long shot.

I got my Nexus One from http://www.google.com/phone a few months ago, to replace my aging K800i (which still takes better photos than the Nexus One, or the iPhone, or indeed most phones). Since I got my phone directly from Google, it was easy to update to the latest Froyo, which got rid of some of my bigger annoyances on the device. Most tedious in previous versions was the need to individually select and authorize every application when an update showed up. Most days I see between two and five updated applications, and updating any given application takes around 30 seconds, so it was turning into a chore. Froyo is also much smoother in its transitions; it's the way Eclair should have been.

The Android app store (Marketplace) has gotten some criticism, but I don't know how truly meaningful those criticism are, as I've gotten applications primarily based on recommendations, or searching the store with intent, rather than browsing the store. I recently had occasion to count the number of apps I have installed, and if you judge by Android's app management screen, it's now up to 70 (I have 93 icons in my app icon scroller view). I thought I'd list a few of the best ones, as a way of aggregating my own perspective from the different recommended lists I've seen. Some of these applications are not free, but as a developer I don't baulk at paying the small amounts charged, especially since returning for a refund is built into the marketplace if you don't like the app.

ES File Explorer is probably the single application that's most responsible for me preferring Android to iPhone. It's a file browser, but more than that: it can browse Windows (SMB) network shares, as well as FTP and Bluetooth. That means I can copy music, movies etc. on and off my device to and from my NAS. I find iTunes to be a tedious waste of memory and processes, not to mention it wants to update with huge downloads practically every second week, incorporating media players and web browsers I don't want. Crapware. Being able to take control of the media upload and download experience is wonderful, and makes me feel like my phone is more than a locked-down toy.

I complement it with Astro File Manager, which has better support for photo management and a built-in image viewer (which supports image view intents, so it can actually be used by ES File Explorer to view photos). I'm not a big fan of Nexus One's stock Gallery app (by CoolIris) - lots of gloss, but slow to index photos if you've taken a bunch. Astro can also act as a file selection agent for other applications that browse for an image, such as image editors.

Another photo viewer is B&B Gallery. It was one of the first to support multi-touch, manually implemented before the OS got support. An advantage it has over the built-in Gallery is that it doesn't downsample loaded photos, so you can zoom in and check the details, rather than quickly getting lost in blur. As a gallery app, however, it's not particularly pretty. I find file management superior, especially as you don't have to wait as thumbnails from all over the place get loaded, but can classify into directories, etc.

Act 1 Video Player is an excellent replacement for the stock video player. It doesn't add any more decoding capabilities, but it has better affordances in its UI, especially with its touch-based seeking support. Best feature: swiping left and right in the center of the screen seeks back and forth through the video.

NewsRob is an offline RSS reader that synchronizes with Google Reader. It has configurable caching, so you can have it download images which are linked from the RSS (a problem with Google Reader's own offline version using Gears, ironically), up to caching the full web page associated with that RSS item. Excellent for public transport.

A major annoyance with the iPod Touch (and also the iPhone) for me was the auto-rotation. I almost never want to rotate the view, and it always ends up rotating when I'm lying down, or otherwise not in front of a computer. I was only able to solve this problem on the iPod Touch by jailbreaking it. Android has a setting for this, but for easier access to it I use the AutoRotate widget. This lets you put a 1x1 widget anywhere which toggles the auto-rotate setting on a tap.

Some games are useful for passing idle moments. Robo Defense is quite addictive tower defense with RPG-like elements; you earn what are essentially XP points, and can spend them on incremental upgrades, so there's a campaign-like aspect to the gameplay. Replica Island is a classic-style platformer which is particularly ergonomic on the Nexus One, using the scroll ball for directional control. As an aside, controls are one of the weakest elements of most iPhone games - it badly needs more physical buttons. And Nesoid, an NES emulator, is nice in principle, but a better control system is needed.

Artistic diversions: DoodleDroid is a finger-painting app with configurable brush dynamics, so with care, you can get some interesting impressionistic images out of it. Simpler, more like coloured markers than paint, is Draw!.

Of course, there are the bar code apps, like ShopSavvy, probably the most integrated when you have buying intent, though its local shop search isn't very localized, even when in London; ZXing Barcode Scanner, which runs more general web searches based on barcodes; Google Shopper and Google Goggles also do barcodes, but I feel they're weaker, and Goggles is mostly a gimmick (IMO).

Google Sky Map is pretty neat - the way it uses the accelerometer to overlay constellations etc. is probably the neatest augmented reality-style implementation I've seen, even though it doesn't overlay on a video image from the camera. Layar is the probably the canonical implementation, but I find it to be too gimmicky in practice, having to walk around like an idiot with a phone held out in front of you. At least with stars, you're normally standing still and looking into the sky.

Google Translate is another essential app. It's tantalizingly close to real-time speech to speech translation; as it is, you can speak into it and at a button press do text to speech on the translation, providing the speech recognition was good. My girlfriend tells me it can be overly literal for German, however.

Wifi Analyzer helped me get better channel placement on my home wifi access points. Really neat live view of signal strength for all the different APs in your area, even ones too faint to actually connect to.

Arity is a simple expression-based calculator which can graph simple non-parametric functions in two and three dimensions. By non-parametric, I mean you give it an expression using x, or x and y, and it plots the result of the expression as y, or z, in a 2D plane or 3D volume. You can't plot circles with it, for example.

ConnectBot is a SSH client, useful for remote administration when you're really stuck for connectivity. Doing anything serious on the command line without access to a keyboard is insanity, of course. When the job you're trying to do is simpler - a single command over SSH - ServerAssistant is a better approach.

If you're interested in programming your life, Locale can trigger events based on conditions. Conditions are one or more of location, time, orientation, calls by contacts and battery state. Settings include wallpaper, ringtone, screen brightness, wifi enabled or not, volume, bluetooth, but also actions published by third-party applications. For example, NewsRob can synchronize based on a Locale trigger. And if you've installed ASE, the Android Scripting Environment, you can run arbitrary scripts - bash, python, ruby, etc. - on a trigger. The sample scripts available for ASE include invoking text to speech to say the time and the current weather, toggling airplane mode, showing notifications, etc. Locale is a lot less useful if you have a more flexible schedule, but if you're tied in to a timetable, it makes a lot of sense.

Finally, a battery widget: Battery Left. I don't use task managers or killers; I've found that it's better to let Android do its thing and kill what it needs to kill, when it chooses to do it. I get about 46 hours on average battery, but I tend to recharge before 36 hours have gone past. You can drop this widget as a 1x1 (or 2x1) visual indicator of battery left, with configurable detailed textual data: estimated time before battery dead, estimated time of day of dead battery, estimated battery %, etc. It monitors battery performance, so it should straighten the curve that batteries self-report - I've often seen batteries say they have three-quarters battery for ages, and then run out the remainder quite suddenly, etc.

Obviously, I have many more applications installed than I've mentioned here, but they tend to be single-purpose location-based ones that have less general applicability, or ones I don't use as often and can't in good conscience recommend. But I can say that all of the above work pretty well for me, and it's notable that many of them would contravene Apple's developer policy, so for me at least, app availability for the iPhone isn't the killer advantage it's made out to be.

Locations vs Values: using RTTI to work with value types

Delphi's Rtti unit is designed in substantial part around TValue, a kind of hold-all record that should be capable of containing almost any Delphi value, along with type information for that value. However, this means that when you're working with value types, such as static arrays and records, modifying the values when stored in a TValue is modifying that copy, stored inside the TValue. If you want to manipulate a field (F1) of a record which is itself a field (F2) of another type, you need to first copy the F2 field's value out into a TValue, then modify F1 in the TValue, and then copy it back in to the original F2 field.

As an aside: TValue.MakeWithoutCopy does not relate to this copying behaviour, but is rather for managing reference counts with strings and interfaces and other managed types. This is particularly important when marshalling parameters to and from stack frames, where logical copies sometimes should be made, and sometimes not.

However, working with values in TValue all the time is not necessarily the most efficient technique. By adding another layer of indirection, we can improve things: instead of working with values, we can work with locations.

This can be encapsulated fairly trivially using the current RTTI support. I hacked up a TLocation type which represents a typed location analogously to how TValue represents a value:

type
  TLocation = record
  private
    FLocation: Pointer;
    FType: TRttiType;
  public
    class function FromValue(C: TRttiContext; const AValue: TValue): TLocation; static;
    class function FromAddress(ALocation: Pointer; AType: TRttiType): TLocation; static;
    function GetValue: TValue;
    procedure SetValue(const AValue: TValue);
    function Follow(const APath: string): TLocation;
    function Dereference: TLocation;
    function Index(n: Integer): TLocation;
    function FieldRef(const name: string): TLocation;
  end;

For ease of use, it uses TRttiType. If it were to be fully as flexible as TValue, it would use PTypeInfo instead, like TValue does. However, using the RTTI wrapper objects makes life a lot easier.

Here it is in use:

type
  TPoint = record
    X, Y: Integer;
  end;
  TArr = array[0..9] of TPoint;

  TFoo = class
  private
    FArr: TArr;
    constructor Create;
    function ToString: string; override;
  end;

{ TFoo }

constructor TFoo.Create;
var
  i: Integer;
begin
  for i := Low(FArr) to High(FArr) do
  begin
    FArr[i].X := i;
    FArr[i].Y := -i;
  end;
end;

function TFoo.ToString: string;
var
  i: Integer;
begin
  Result := '';
  for i := Low(FArr) to High(FArr) do
    Result := Result + Format('(%d, %d) ', [FArr[i].X, FArr[i].Y]);
end;

procedure P;
var
  obj: TFoo;
  loc: TLocation;
  ctx: TRttiContext;
begin
  obj := TFoo.Create;
  Writeln(obj.ToString);
  
  ctx := TRttiContext.Create;
  
  loc := TLocation.FromValue(ctx, obj);
  Writeln(loc.Follow('.FArr[2].X').GetValue.ToString);
  Writeln(obj.FArr[2].X);
  
  loc.Follow('.FArr[2].X').SetValue(42);
  Writeln(obj.FArr[2].X); // observe value changed
  
  // alternate syntax, not using path parser
  loc.FieldRef('FArr').Index(2).FieldRef('X').SetValue(24);
  Writeln(obj.FArr[2].X); // observe value changed again
  
  Writeln(obj.ToString);
end;

Here's most of the implementation:

{ TLocation }

type
  PPByte = ^PByte;

function TLocation.Dereference: TLocation;
begin
  if not (FType is TRttiPointerType) then
    raise Exception.CreateFmt('Non-pointer type %s can''t be dereferenced', [FType.Name]);
  Result.FLocation := PPointer(FLocation)^;
  Result.FType := TRttiPointerType(FType).ReferredType;
end;

function TLocation.FieldRef(const name: string): TLocation;
var
  f: TRttiField;
begin
  if FType is TRttiRecordType then
  begin
    f := FType.GetField(name);
    Result.FLocation := PByte(FLocation) + f.Offset;
    Result.FType := f.FieldType;
  end
  else if FType is TRttiInstanceType then
  begin
    f := FType.GetField(name);
    Result.FLocation := PPByte(FLocation)^ + f.Offset;
    Result.FType := f.FieldType;
  end
  else
    raise Exception.CreateFmt('Field reference applied to type %s, which is not a record or class',
      [FType.Name]);
end;

function TLocation.Follow(const APath: string): TLocation;
begin
  Result := GetPathLocation(APath, Self);
end;

class function TLocation.FromAddress(ALocation: Pointer;
  AType: TRttiType): TLocation;
begin
  Result.FLocation := ALocation;
  Result.FType := AType;
end;

class function TLocation.FromValue(C: TRttiContext; const AValue: TValue): TLocation;
begin
  Result.FType := C.GetType(AValue.TypeInfo);
  Result.FLocation := AValue.GetReferenceToRawData;
end;

function TLocation.GetValue: TValue;
begin
  TValue.Make(FLocation, FType.Handle, Result);
end;

function TLocation.Index(n: Integer): TLocation;
var
  sa: TRttiArrayType;
  da: TRttiDynamicArrayType;
begin
  if FType is TRttiArrayType then
  begin
    // extending this to work with multi-dimensional arrays and non-zero
    // based arrays is left as an exercise for the reader ... :)
    sa := TRttiArrayType(FType);
    Result.FLocation := PByte(FLocation) + sa.ElementType.TypeSize * n;
    Result.FType := sa.ElementType;
  end
  else if FType is TRttiDynamicArrayType then
  begin
    da := TRttiDynamicArrayType(FType);
    Result.FLocation := PPByte(FLocation)^ + da.ElementType.TypeSize * n;
    Result.FType := da.ElementType;
  end
  else
    raise Exception.CreateFmt('Index applied to non-array type %s', [FType.Name]);
end;

procedure TLocation.SetValue(const AValue: TValue);
begin
  AValue.Cast(FType.Handle).ExtractRawData(FLocation);
end;

To make it slightly easier to use, and slightly more fun for me to write, I also wrote a parser - the Follow method, which is implemented in terms of GetPathLocation:

function GetPathLocation(const APath: string; ARoot: TLocation): TLocation;

  { Lexer }
  
  function SkipWhite(p: PChar): PChar;
  begin
    while IsWhiteSpace(p^) do
      Inc(p);
    Result := p;
  end;

  function ScanName(p: PChar; out s: string): PChar;
  begin
    Result := p;
    while IsLetterOrDigit(Result^) do
      Inc(Result);
    SetString(s, p, Result - p);
  end;

  function ScanNumber(p: PChar; out n: Integer): PChar;
  var
    v: Integer;
  begin
    v := 0;
    while (p >= '0') and (p <= '9') do
    begin
      v := v * 10 + Ord(p^) - Ord('0');
      Inc(p);
    end;
    n := v;
    Result := p;
  end;

const
  tkEof = #0;
  tkNumber = #1;
  tkName = #2;
  tkDot = '.';
  tkLBracket = '[';
  tkRBracket = ']';
  
var
  cp: PChar;
  currToken: Char;
  nameToken: string;
  numToken: Integer;
  
  function NextToken: Char;
    function SetToken(p: PChar): PChar;
    begin
      currToken := p^;
      Result := p + 1;
    end;
  var
    p: PChar;
  begin
    p := cp;
    p := SkipWhite(p);
    if p^ = #0 then
    begin
      cp := p;
      currToken := tkEof;
      Exit(currToken);
    end;
    
    case p^ of
      '0'..'9':
      begin
        cp := ScanNumber(p, numToken);
        currToken := tkNumber;
      end;
      
      '^', '[', ']', '.': cp := SetToken(p);
      
    else
      cp := ScanName(p, nameToken);
      if nameToken = '' then
        raise Exception.Create('Invalid path - expected a name');
      currToken := tkName;
    end;
    
    Result := currToken;
  end;
  
  function Describe(tok: Char): string;
  begin
    case tok of
      tkEof: Result := 'end of string';
      tkNumber: Result := 'number';
      tkName: Result := 'name';
    else
      Result := '''' + tok + '''';
    end;
  end;
  
  procedure Expect(tok: Char);
  begin
    if tok <> currToken then
      raise Exception.CreateFmt('Expected %s but got %s', 
        [Describe(tok), Describe(currToken)]);
  end;

  { Semantic actions are methods on TLocation }
var
  loc: TLocation;
  
  { Driver and parser }
  
begin
  cp := PChar(APath);
  NextToken;
  
  loc := ARoot;
  
  // Syntax:
  // path ::= ( '.' <name> | '[' <num> ']' | '^' )+ ;;
  
  // Semantics:
  
  // '<name>' are field names, '[]' is array indexing, '^' is pointer
  // indirection.
  
  // Parser continuously calculates the address of the value in question, 
  // starting from the root.
  
  // When we see a name, we look that up as a field on the current type,
  // then add its offset to our current location if the current location is 
  // a value type, or indirect (PPointer(x)^) the current location before 
  // adding the offset if the current location is a reference type. If not
  // a record or class type, then it's an error.
  
  // When we see an indexing, we expect the current location to be an array
  // and we update the location to the address of the element inside the array.
  // All dimensions are flattened (multiplied out) and zero-based.
  
  // When we see indirection, we expect the current location to be a pointer,
  // and dereference it.
  
  while True do
  begin
    case currToken of
      tkEof: Break;
      
      '.':
      begin
        NextToken;
        Expect(tkName);
        loc := loc.FieldRef(nameToken);
        NextToken;
      end;
      
      '[':
      begin
        NextToken;
        Expect(tkNumber);
        loc := loc.Index(numToken);
        NextToken;
        Expect(']');
        NextToken;
      end;
      
      '^':
      begin
        loc := loc.Dereference;
        NextToken;
      end;
      
    else
      raise Exception.Create('Invalid path syntax: expected ".", "[" or "^"');
    end;
  end;
  
  Result := loc;
end;

The principle can be extended to other types and Delphi expression syntax, or TLocation may be changed to understand non-flat array indexing, etc.

This post was inspired by this question on Stack Overflow, and some similar questions to it that popped up over the past few weeks.

Programming font for VS2010

So, Visual Studio 2010 shipped. Now I have a problem: what font to use? My old standby, Dina, is a bitmap font so it doesn't work with the WPF text editor in VS2010.

Some things I look for in a programming font:

• High information density - vertical height in particular. For example, I use Dina at 8pt.
• Crisp, even lines
• Strong distinction between bold and regular (for lexical highlighting)
• Clearly unambiguous letters: l,1,0,O, and the like.
• Balanced operators - good placement for <, >, {, }, etc.

Here's Dina 8pt in VS2008:

Here's Consolas at 8pt in VS2010:

(At least, I think that's 8pt. I have zoomed the editor in and out a little with Ctrl+Mouse Wheel, but there doesn't seem to be a zoom reset...

Here's Consolas 9pt in VS2010:

Consolas is quite constrained on the horizontal. It looks horrible at 8pt, and is still slightly taller than Dina 8pt. Even at 9pt though, to my eyes, it suffers quite badly with comparison to Dina. The bold of the comment and digits is almost unnoticeable. 'M' and 'm' are very weak, probably because of the narrowness. The vertical on the 'F' of 'Func' is weak, as is the vertical on 'r'. These are in part properties of ClearType subpixel rendering, which looks slightly different from monitor to monitor and depending on configuration, but even with the configuration selected for its heaviest rendering, it looks poor on my system. Overall, the effect is washed out and slightly blurry.

Proggy Fonts are another contender. These have TTF hinted versions of what amount to bitmap fonts, so they try and sneak older rendering in through the back door. Here's Proggy Clean Slashed Zero with Bold Punctuation (Visual Studio can bold operators (and I do - and make them white, to stand out even more), but it doesn't include [], {}, () as operators, unfortunately):

This is probably the most palatable option for me, but it's still not quite as good as Dina. It's the same height, probably because Dina was based on Proggy; but the things I don't like about it most are things that Dina fixed, in particular the oddly elongated < and >, the lazy looking 's', and the "gappy" look of the bold - contrast the comment in Proggy vs Dina. Other differences incude the leg on the 'R' and the general way the capitals are slightly overly broad. Another oddity is however ClearType has interacted with its hinting, the font has ended up tinted with green.

Proggy Clean SZBP is the font I'll stick with for the moment, until I see a better option, or a better translation of Dina to TTF than this one, which is tuned for 10pt Dina, not the size I use.

JEDI newsgroup change

I got word that the JEDI newsgroup changed after the old forums.talkto.net server died. The new server is at:

news.delphi-jedi.org

That's nntp on standard port 119. Please update your favorite news reader to this new server.

CrashPlan for Backup on Nexenta

For some time, I've been using cron jobs with rdiff-backup on Cygwin for backups. The cron job runs on a Windows server I have on my home network and iteratively mounts remote Windows shares (if they're present) and runs rdiff-backup over them, with the destination also being local. Then the job runs rsync to mirror this backup to my Nexenta NAS running ZFS raidz.

This solution gives me a fair amount of local redundancy: two separate machines, with two copies of the backup data, plus the RAID-5-like redundancy and checksumming integrity that ZFS provides. Rdiff-backup is quite reassuring with respect to restores too: it stores files in the filesystem directly, along with differences (the rdiff bit) so you can go back in time. That means that restoring is as simple as copying the files straight out of the backup and deleting the rdiffs.

Of course, a backup strategy isn't solid without a remote copy. Today, I finished configuring CrashPlan, a really neat backup solution built using Java. You can read about the features etc. of CrashPlan on the website - I heard about it from the Java Posse podcast, episode 298. Initially, I opened an Amazon S3 account, and was considering mirroring my backups to S3 with s3sync, but after I evaluated CrashPlan, it looked like it made more sense than further pursuing my homegrown approach. Not only is it cheaper than S3 for my data (I guess they depend on overselling like ISPs), but CrashPlan has features that make it work well in my case.

The fact that it's primarily Java meant I could install it on my Nexenta box. CrashPlan don't have an installer for Nexenta, but they do have ones for Linux (LSB) and Solaris, and Nexenta is like a hybrid. I'm documenting the steps here in case I need to set it all up again, though hopefully not for a restore.

I'm running NexentaCore NCP 2, which doesn't come with Java. Installing a JDK (overkill, but I wanted to test the Java environment with a simple hello-world):

$ sudo apt-get install sun-java6-jdk

I was running a release candidate of Nexenta and I had to do an apt-get update and upgrade to resolve all necessary dependencies, but it was fairly stress free because Nexenta's apt-clone checkpointed the root file system with a ZFS snapshot. I did lose my almost half-year of uptime though - my Nexenta machine has been the most dependable of all my machines, in both hardware and software.

Installing CrashPlan itself needs bits from both the Linux and Solaris installers. I downloaded both, and unpacked both. In the Linux install, I ran the provided install.sh and followed the steps, putting it in /usr/local, with daemon init script in /etc/init.d and runlevel symlink in /etc/rc3.d. But the Linux install isn't enough. In particular, the init script assumes GNU ps, but Nexenta uses Solaris ps. So I swapped in CrashPlanEngine (the target of the init script symlink) from the Solaris installer in the place of the installed CrashPlanEngine that was here:

/usr/local/crashplan/bin/CrashPlanEngine

But that wasn't enough. CrashPlan loads a library called libjtux via JNI, a kind of POSIX API for Java programmers who want direct access to the OS. The libjtux.so from the Linux install was linking against libc.so.6, assuming GNU C library versioning. Replacing libjtux.so with the version from Solaris, linking against plain libc.so, solved this problem - here:

/usr/local/crashplan/libjtux.so

Finally, I had to configure CrashPlan. My Nexenta install is headless - all my interaction with it is either over Samba shares, HTTP for wiki servers etc., and of course ssh for everything else. Here's one of the really neat features of CrashPlan: the user interface for configuration is a client that depends on a single TCP connection with the local backup server. All I had to do is get the client to connect to a different local port, and tunnel that port to Nexenta using SSH, after enabling TCP forwarding in Nexenta in /etc/ssh/sshd_config. This bit is described on CrashPlan's site describing how to configure a headless client.

I'm running CrashPlan+, a for-pay version of the engine, on two machines - my main desktop and my Nexenta box - to get all the encryption, compression, deduplication etc. goodness. And considering that CrashPlan supports peer to peer backup, I may simply replace my existing ad-hoc rdiff-backup approach with local CrashPlan backups, as CrashPlan supports multiple destinations for backups, as well as receiving backups from other machines.

A key limitation of CrashPlan, and one that I found particularly annoying, is that the Windows client doesn't support network shares at all, in any shape or form - whether they're mapped to drive letters or not. The backup engine runs under the SYSTEM account, so it doesn't have network credentials, and also means it may not be able to access all the files you're trying to back up - especially EFS encrypted files. (CrashPlan doesn't seem to use Windows' EFS backup capability, e.g. ReadEncryptedFileRaw.) I changed the CrashPlan engine's service account to my own account to try and close this hole, but still no go on accessing network shares. This meant that I had to get CrashPlan running on my Nexenta box in order to store local backups on ZFS, should I so choose.

But apart from that, I've been impressed with CrashPlan's feature set and usability.