Public interest rule of thumb

TechCrunch is having a crisis of conscience over what they'll do with internal Twitter documents they've received.

My rule of thumb is pretty simple. The documents are stolen. Unless the documents reveal wrongdoing greater than stealing, such that the public interest served in publishing is greater than the private harm, then it's unethical to publish. Idle musings about "they are going to be published somewhere on the Internet" is specious; the morality of one's actions don't depend on whether someone else is "doing it", much less everyone.

If TechCrunch goes ahead and publishes these documents, it'll only further cement my already pretty low opinion of Arrington. (I'm not a habitual reader of TechCrunch, or otherwise I'd waste away over all the food I'd be vomiting up.)

StackOverflow and religious language in programming

I was reading Why I Don’t Listen to the StackOverflow Podcast any More, and in reading the comments, I found it interesting to see the amount of religious language.

#21 (sbohlen) I’d have to agree that they do indeed appear successful, but success / failure isn’t a boolean [...] The world is full of people who succeed due to all kinds of factors including dumb-luck, [...]

This reminds me intensely of "the race is not always to the swift" etc. Eccl 9:11.

I changed adopters to believers to accentuate the effect:

"#16: [...] I have also experienced that the non-[believers] aren’t (gnerally) non-[believers] out of choice but are instead non-[believers] largely out of ignorance and inexperience — they just don’t know what they don’t know [...]"

#16: "the [types] who aren’t interested in self-improvement; there’s not a damned thing I can suggest to help these people — I cannot MAKE someone want to better themselves, that’s gotta come from within"

#16: "I [...] think that Jeff and Joel are indeed preaching [...] they are (IMO) abdicating the unofficial professional responsibility that comes with having a pulpit from which to preach."

#14: "I am a strong believer"

#13: "@sbohlen. I fully agree with the sentiment that “up and running” is not a valid metric"

I'm not going to go on with this, though I could - the about page for the blog is particularly ripe for further examples of religious language - but I do want to make a further point.

Could it be that latent religious feelings are responsible for a good portion of the minor angst on display in these comments? Could it be that people are unconsciously annoyed, not with StackOverflow's success, but rather its success even in the face of quite mild contempt at the unit-testing / SOLID religion?

My own position: I'm a "believer" in evolution and the market. Neither evolution nor the market have any moral content (they can't say what "ought" to be), and both only measure success by effects rather than processes. In a phrase, "up and running" is not just a valid metric - it's the only metric, until it stops running. If SO continues to thrive, and doesn't die under the weight of its lack of unit testing (which seems to be the logical end result of the beliefs of the believers in the comments), where then is the truth in the proclamations of doom from the prophets of the true religion?

I don't think unit-testing is the only way to avoid long-term bit-rot. I think it's a good way, but not the only way; moreover, I don't think unit testing is especially useful in the very early stages of a project, where the design changes often and rewriting unit tests becomes a disincentive to larger refactoring and redesign. I think every one of the concepts embodied in S.O.L.I.D. is debatable in certain scenarios (except perhaps LSP).

In my experience, premature abstraction has been responsible for a similar number of ills to premature optimization: abstractions chosen at the wrong boundaries, for the wrong reasons, because designers thought they could foretell the future and anticipate where changes would come. Thus I am wary of merchants of abstractions and patterns, principles and practices. I'll use them when appropriate, but never with religious zeal.

WPF / VS2010 Font rendering: blurryville

I downloaded and installed VS2010 Beta 1 to see how it's looking. It's not pleasant though - it makes me feel like I'm looking at the screen through a layer of coke-bottle plastic:

Compared to the old rendering, where my preferred fixed-width code / terminal font, Dina, is correctly chosen, as opposed to falling back to what looks like Courier New:

The new rendering hurts my eyes (they keep trying to focus more, but it doesn't help!) and makes VS2010 Beta unusable, as far as I'm concerned.

The "recommended solution" for text clarity issues in WPF is "use the largest font size possible". That's a bit of a cop-out for the working programmer, where displaying as much code on-screen as is comfortably legible is usually the best option.

FWIW, I registered a feedback item on this.

Sharing a printer, the hard way: writing your own spooler

I have an ultra-cheap HP Color Laserjet 1600 printer, the kind that costs less than a full set of replacement toner cartridges, and the ones that are shipped in the box are only 20% full. From the driver publisher info it appears the I/O protocol logic of the printer was provided by a company called Zenographics, which appears to now be either a part of or owned by Marvell. It works well enough when you have it directly connected to a machine via the USB cable, but sharing it is a nightmare.

In particular, the drivers are very unstable when targeting shared printers. Especially problematic is printing from Vista to XP, or XP to XP, or XP to Vista. With some combinations, the print spooler crashes; with others, it simply hangs, and the spooler service needs to be restarted to get the document printing at all. The folks on the HP forums seem to imply (a) that this is by design, and (b) sharing the printer effectively is impossible. I find this hard to swallow; since the printer works when directly connected, there is no valid reason why printing across the network should not also work, since the physical printer doesn't come into the equation. Bits are bits; if the driver can't consistently send some bits across the network, either there's some atrocious QA going on inside Zenographics, or worse, malice.

I had despaired of ever getting it working nicely as a small workgroup printer, but I saw a faint glimmer of hope when setting up my home NAS, based on Solaris. CUPS is available for Solaris, and there's a Foomatic printer driver available for this specific model as well.

It turns out that I couldn't get CUPS configured with this printer and driver setup. CUPS uses a web page interface for configuration, but it was desperately slow, and every time I tried to print a test page, it informed me that the printer had just gone offline. I could see all the background processes it had started up to print, and the printer device at /dev/printers/0 was indeed opened for writing, but it never got anywhere.

But that didn't matter. The foo2hp driver provided the only real tools I needed: a PostScript to ZjStream (the printer protocol) filter, which simply reads the raw PostScript and outputs the raw print data. Windows ships with a generic PostScript driver; I use the PScript5-based "HP Color Laserjet PS". A simple shell pipeline from source to printer device would do.

To share the printer, I needed some kind of queue for buffering. I decided on a simple filesystem-based approach, which seems to me to be in the Unix tradition. I created a share on my Solaris box specifically for files to be printed. Windows clients can use the aforementioned generic printer driver with a FILE: target, and place the output in this shared directory. The spooler script I wrote polls this directory periodically, and attempts to print whatever it finds.

The printing logic is pretty simple, at its core:

foo2hp2600-wrapper -p 9 input.ps > /dev/printers/0

The actual code is a mite more sophisticated, of course; it does logging, moves documents to be printed into a directory categorized by date and time, etc. And in future it can be extended to other document types than PS, e.g. printing PDFs or photos simply by copying them to a directory.

Another advantage of this approach is that I can shell out to ps2pdf to create an archival version of the printed document. The PDF is usually far smaller than the PS and is viewable from Windows without installing a PostScript viewer. This archival version is useful if I ever need to print it again. Consider online flight checkins, for example - some websites (such as EasyJet) prevent printing the boarding pass more than three times, which is a little inconvenient if you're having printer trouble.

On the downside, printer options that can't be expressed directly in the PostScript with the generic driver can't be configured. For example, the way I have things configured I'm limited to printing monochrome on A4 by default. In practice this isn't really a problem, since the vast majority of print jobs fit this template; I could create other spooler directories for different defaults.

I expect that with enough research, I'll be able to get CUPS configured correctly, or to get Samba to look like a PostScript printer that dumps the raw network data into my spooler directory. However, I'm pretty happy with my hack for now...

ZFS/Solaris as a NAS

I've finally got a semblance of a Solaris system up and running with a nice fat ZFS storage pool. It was neither trivial nor pleasant, however.

Here's the executive summary of what I learned:

• Don't use OpenSolaris 2008.11; use at least NexentaCore 2 instead; others may work.
• Don't use an Intel or Realtek network adapter; if you must, use Realtek with an older version of the gani driver.
• Don't use Solaris's built-in CIFS implementation, smb/server, aka hereafter "cifs" (to contrast with "samba").
• Don't let ZFS to default to 128KB max block size unless all your files are big (greater than about 2MB) or tiny (less than about 64KB).

OpenSolaris 2008.11, the version I tried, is not production ready unless you have extremely specific hardware and quite narrow requirements. I finally ended up using NexentaCore Unstable v2.0 Beta 2.

The prime requirements for a working NAS are:

1. Lots of attachment points for disks
2. Good physical network connectivity
3. Good network protocol implementation
4. Good filesystem implementation

There is no one of these that I didn't have issues with.

The first one, attachment points, was a nice-to-know gotcha that it took me a while to figure out. My motherboard (an old Abit AW9D-MAX) has 7 internal SATA attachment points, as well as 1 PATA channel for two extra PATA devices. This being insufficient, I procured an extra PCIe two-port SATA adapter. I was up and running fine with 5 disks attached (1 boot drive and 4 drives in a raidz ZFS pool - effectively RAID 5 without the need for NVRAM), and in preparation for creating my second raidz vdev (virtual device in ZFS lingo), I started attaching more drives, one by one so that I could label them (and thereby know which one failed when it fails - which it will, eventually). When I had 7 drives attached, the OS occasionally failed to boot, but would reliably hang upon execution of the 'format' command (the handiest way of figuring out the drive device name under Solaris).

The gotcha, however, is that the BIOS defaults to IDE emulation mode, which, for this particular BIOS, supports a maximum of 6 devices. It packs the first 2 hard drives into pretend-PATA (complete with "master" and "slave") even if they aren't actually PATA, and then labels the next 4 drives as SATA 1 through 4. I had to change the BIOS's mode to AHCI to get it to support more drives. Luckily, my boot drive is an old PATA one, so it didn't need compatibility to stay in the same place (master on the first PATA channel).

Physical network connectivity was a far, far harder task, and one that still isn't quite complete. My motherboard has two physical 1Gb Ethernet adapters built in. Unfortunately, OpenSolaris defaults to a driver it calls rge for the chipset (RTL8111/8168B PCIe), but this driver has big issues, that I suspect are related somehow to duplex operations. Big transfers using big buffers using cifs (the Windows-compatible protocol Solaris ships with) will work with reasonable performance (30MB/sec or so), but streaming video (small reads that expect low latency) performs abysmally. Even worse, at non-deterministically random times (as far as I could ascertain), the rge driver would fall into a mode where transfers were extremely slow, as in less than 1MB/sec slow. I took to keeping a ssh session open to the Solaris machine and simply echoing '.' back and forth (echo '.' loop piped to ssh cat -), and mysteriously enough, so long as there was small back and forth traffic, transfer performance rocketed back up again.

This being unsatisfactory, I tried to use the gani driver. I never could get gani-2.6.3 to work with my chipset, however. I didn't know if it was a problem with the driver, a conflict with my chipset, or my poor configuration skills with Solaris. All I know is that I tried every technique I could find, up to and including patching driver_aliases and running sys-unconfig to start things off from a fresh basis.

So, I bought an Intel 1000/PRO Gigabit ethernet adapter. Folks online, in response to reports of rge not working correctly, seemed to say that the Intel adapter "just worked", but I should have dug deeper... The driver Solaris uses for this is called e1000g. The version of this driver shipping in OpenSolaris 2008.11 doesn't work. It drops packets. Simply pinging the Solaris machine from the outside shows packet loss exceeding 5%. With a ssh session open to Solaris, simply holding down '.' on the keyboard and watching the character getting echoed in the terminal is sufficient to demonstrate the hiccups: the pauses are visible and disturbing. It actually affects typing in ssh, almost as if one was at the end of a dodgy Internet connection, perhaps some half-baked cafe wifi link in Marrakesh. A recursive diff of about 100MB of data across about 1000 files using cifs took a whole 45 minutes. A trivial test using dd to copy a 640KB blob of data using a 64KB buffer size took about 15 seconds and dd reported transfer rate of 45KB/sec.

It was this network connectivity that prompted me to start trying out other distributions. NexentaCore 2 was first in line. It defaults to the gani driver for the Realtek adapter, but an earlier version, 2.6.2. This works, but only at 100Mbps; try as I might, I can't convince it to go higher. (Windows running on the same hardware, same patch cable and same switch, never, ever dropped down to 100Mbps - something I can't say about the other Windows machines I have here.) Alos with significant importance, NexentaCore doesn't ship with a GUI (so no need to disable it), and does ship with apt-get and a reasonable selection of packages, including my preferred Linux/Unix editor, joe (very similar to old Borland DOS IDE text editors). I detest vi - I've never used a program that gleefully punished more heavily every transgression of the user.

Anyhow, that's how I got one reasonably-working 100Mbps connection out of three 1Gbps adapters, four driver versions and two operating system installs.

Next, the network protocol. Solaris defaults to using something that is colloquially called cifs, although the usual identifiers used when e.g. enabling and disabling with svcadm are smb/server and smb/client. I can't say much about smb/client - I'm sure it works well enough, but I don't have much use for it - but smb/server barely works at all. It's implemented as a kernel module, which unfortunately means it has odd and painful limitations compared to how user-level programs view the file system. In particular, only one ZFS filesystem can be navigated from a cifs share, significantly harming the usefulness of creating lots of ZFS filesystems. A ZFS file system mounted inside an existing filesystem, where the mountpoint is visible from a cifs share, will show up as an empty directory to clients, whereas it appears correctly mounted locally on the Solaris box.

Another limitation is symbolic links. By apparent design, cifs prohibits what Samba calls "wide links": symbolic links that resolve to locations that are outside the share's hard-linked subtree. Such symbolic links look much like dead links from Windows, i.e. the text little files they are implemented as. Samba defaults to "wide links" on for performance reasons if nothing else.

A final limitation is hard links created from Windows using CreateHardLink. Cygwin 'ln' ultimately uses this API. For whatever reason (and I didn't dig too deeply, like investigating Wireshark traces), Cygwin determines that 'ln' isn't supported on cifs shares and falls back to copying instead. Cygwin 'ln' works correctly on Windows shares, however, and it also works correctly over Samba.

The verdict: cifs isn't worth it. Needs more time to bake. Use Samba instead.

Finally, ZFS itself. ZFS is the big draw, and the reason I chose to use Solaris. Userspace implementations are available in Linux via FUSE, and also in FreeBSD, but I had bad performance experiences of NTFS-3G/FUSE in Linux, and FreeBSD's implementation sounded dangerously non-production ready. ZFS largely works as advertised. The primary limitation is the inflexibility in removing drives from pools. Raidz vdevs can't have drives removed at all. A word of warning, though: ZFS has what I can only consider a bad bug for files in the 128KB..2MB range. If you have a file of about 129KB, depending on how it was created, it ought to be using up about 100% more space than it should be. In some largeish directories of files I was seeing wastage of about 35% (as measured by 'du' versus 'du --apparent-size'), whereas NTFS on Windows had wastage in the region of 5%. Paring back the ZFS default block size (dynamically settable on ZFS - yay! - but only affects subsequent file operations) to 8KB or 16KB improves things immensely, but still not quite as space-efficient as NTFS.

In conclusion, ZFS/Solaris as a NAS can work well with very carefully chosen hardware and select software configuration. If you get it working better than I describe herein, be very careful what you touch, and whatever you do, don't upgrade your zpools until you're sure that whatever step forward you take is better than what you had previously.