The first computer I fell in love with was the Apple ][ (2? II?). I learned BASIC and a smidgen of assembly to write little games. I grew up with a bunch of other computer-nerd friends and we all swapped disks with games we copied (remember Copy ][ Plus?) or stuff we'd written. Only I didn't own an Apple. Instead the computer my parents chose to buy was a TRS-80. It was a good lesson in hiding your disappointment for the sake of loved ones' feelings.

Fast-forward to high-school and my parents upgraded us to a Mac Classic with 30MB external drive.  I had walked away from programming for a few years and turned my attention to music. I didn't program on that Mac, but I did just about everything else. That computer carried me through my college years.

Once I entered professional life I was in the world of Windows, DOS and PCs. I was working for a financial services company and was about as far away from programming as a person could get. When I returned to the Pacific Northwest fifteen years ago, I found myself back in the world of computers.

My professional career in computers really started with UNIX and Linux. I was a pretty hard-core Linux guy for a long time, until the day I finally ran out of patience with all the fiddly configuration. I bought my first modern Mac five years ago and started learning to develop on the platform three years ago.

The timing of my "conversion" to the Apple platform couldn't have come at a better time. Not only was the marketplace exploding with demand for those skills but, more importantly, Apple rescued me at a time when I was pretty disillusioned with technology.

While I started playing with computers at an early age, it's not my biggest passion. Computers and programming are really just a means to an end to me. It's not that I don't care about my craft, it's more that I care more about the end-product than the process to get there. Honestly I don't know how much longer I could have gone on feigning interest in algorithms and scalability. Don't get me wrong—they're important, but I'm not passionate about them.

When I came to the Apple platform I thought, finally somebody gave a damn about the person on the other side of the glass and I was happy to be a part of it. So all of this is a roundabout way of saying "thanks" to Mr. Jobs and the company and culture he helped build.

Monday Keynote

Officially, the content in the Monday keynote is all anyone at WWDC is really supposed to talk about publicly. Frankly, the details within the rest of the conference are (mostly) an elaboration of what is announced publicly in the Monday-morning keynote address. There are few little geeky odds-and-ends that would be fun to write about, but I'll wait until they're publicly available. But those aren't the big things, these are:

Lion

I don't do enough Mac development (yet) to warrant playing with betas of the OS. I need the OS to be stable so that I can get my iOS work done. That said, I was starting to develop "Lion-envy" whenever I saw other geeks trying it out. This morning I finally sacrificed my laptop to a Lion install just so I could get the new bits. In just the first few minutes I realized this is a really nice upgrade from Snow Leopard. I can't wait for the official release so I can upgrade my main work machine.

When Lion was first announced I wondered if it was going to continue the "flattening-out" in the curve of feature-growth that we saw from Leopard to Snow Leopard. Honestly, I had a hard time imagining what they could do to evolve such a mature OS. But once again, the folks in Cupertino wowed me with some innovative thinking about how to make the OS easier to use. Goodbye Spaces, goodbye Exposé, hello Mission Control and Launch Pad. These seem to be a much better implementation of the ideas behind those original technologies. But I don't know that they would have been the obvious conclusion pre-iOS. It's hard to overstate the impact iOS has had on Apple's thinking about UI design.

I like the muted look of Lion. The Finder is just a wee-bit more toned-down. Standard Aqua buttons have dropped much of their original garish look. Overall it's just a much more mature look. There's just a lot of polish in this release. Features like Air Drop, Resume, Auto Save, Versions do a nice job of just rounding the sharp-edges off. With these features, it's clear that Apple has declared war on the filesystem as a core concept for personal computing.

iOS 5

If there was a sub-theme below iCloud, it was (dare I say?) a bit of humility from Apple for acknowledging the things that needed to be addressed in all the platforms. The thunder of applause that went up when the new notifications UI was announced from iOS showed just how frustrated we had all become with a system that clearly wasn't working. It's one of the few times I've seen Apple acknowledge the superiority of someone else's design and just unabashedly copied it. The new notifications system was clearly inspired by Android and Android deserves a lot of credit for getting that right the first time. Kudos to the Google-Bot.

PC Free was probably received the second warmest welcome from the audience. It was great to hear Apple openly acknowledge the thud-of-a-user-experience we all had when we got our shiny new iPad or iPhone and were faced with the "please plug me into iTunes" screen. Cue the sad trumpet song. The original promise of the iPad as a replacement for a traditional computer is now actually possible.

The two things that surprised me the most were the addition of the Reminders app and the deep Twitter integration. As someone who has worked on a MobileMe-based to-do list app, I can't say that I didn't feel my heart sink a bit when this was announced. But this was one of those apps that had many wondering why it took this long for Apple to build it. I haven't played with it yet, but I'm really curious to see how the location-awareness features work.

Even more intriguing is the first-class Twitter integration in the iOS. Without tipping too much about what's in the API, I can say that it's more than just a cursory implementation. I can only hope that it means that other big services will join up soon and all of us poor developers can stop dealing with the home-grown contraptions we're using now to talk to these services.

iCloud

iCloud was clearly the center-piece of the show. If Apple pulls this off, it has the potential to radically change the personal computing experience. We all have plenty of reason to be suspicious of this offering. MobileMe is Exhibit A in the case against Apple doing online services. But I think it's also fair to remember the services that Apple has gotten right, specifically iTunes and iOS Push Notifications. I remain cautiously optimistic about this. More than anything else announced last week, I think this is the feature that could have the biggest lasting effect.

What makes iCloud so compelling is the chance to take advantage of server-side storage without the penalty of explicit synchronization. Frankly, it's what makes Dropbox so great too. As the tag-line goes, "it just works". But something like Dropbox doesn't quite make sense in iOS where the holy-war on filesystems was declared from day one. Instead, iCloud is a re-thinking of the atomic units of documents in such a way that its constraints promise a tighter integration between devices. The slide in the keynote in which Jobs declared that they were "demoting" the Mac says it all. Out of all the technologies introduced this year, iCloud captures my imagination the most. I can't wait to see what people build on top of it.

The Rest

I've done my fair share of conferences and WWDC has become the one I've enjoyed the most. RailsConf is probably my second favorite, but I have to admit that I've always felt a little miffed that they moved it from Portland.  WWDC is a sensory-overload the likes of which I've never experienced for a geek conference. This year, especially, just felt huge and somewhat overwhelming. Maybe it was the announcements on Monday, maybe it was the crush of humanity trying to move through Moscone, maybe it was just the buzz in the air. This year's conference was my favorite of the three WWDCs I've been to.

There was a lot of great stuff in the sessions that I'll eventually write about once it's all publicly released. There's so much goodness just in the developer-side of iOS that I'm seriously considering just writing for that and waiting for the fall release of the OS. It was a truly inspiring week. Once I shake off the sleep-deprivation I, like many others I talked to, will be back to work with a vengeance.

A large part of what made the 2011 edition so fantastic was getting a lot out of the extra-curricular hours outside of the conference. Lots of late nights talking with other Apple-nerds is a great way to really cement a sense of place in the community. My only regret is that there were still several folks I didn't get a chance to introduce myself to and chat with.

Well, it's time to get back to "real life". I have a lawn to mow and family to spend time with. But starting Monday, I'll be hitting Xcode in a caffeine-fueled rage, ready to kick some tater.

I count myself as extremely blessed to be doing what I do for a living. Not only do I make ends meet doing work I love, but I also have tremendous time flexibility. This makes it so much easier to do the little things like pack a lunch for my daughter and walk her to school each day. That is something I can't put a price on.

The other thing that "going indie" has allowed me to do is take up the drums. I've played guitar for about twenty-five years, but always harbored a secret desire to play the drums. When I was a kid I knew there was no way my parents were going to buy me a drumset (though why they put up with electric guitar is beyond me). As someone in their late thirties whose brain is starting to calcify, taking up something like drumming is a major mind-stretch. It has been, at times, one of the most frustrating things I've ever done. But it's also been one of the most rewarding.

When I get a new exercise or groove from my drum teacher, at the beginning of the week it seems impossible. How am I possibly going to get all these uncoordinated limbs to cooperate? Like any daunting problem, I break it down. I set the metronome to a laughably slow tempo and slowly work my way up. At the end of the week I (usually) get it. At an age when most people spend their days executing what they're experts at, it's easy to forget what the joy of mastery still feels like. Every day I thank my lucky stars that I have the opportunity to remember what that's like.

The best we could do were some ham-fisted tricks to try to slow-down turn-tables, but you could never really get it down to a reasonable pitch to match your guitar. So the secrets to our favorite songs were passed around amongst aspiring musicians like epic poems among ancient Greeks. This fits in my pocket. If I had seen that when I was fifteen it would have melted my brain.

Or take a look at MLB.tv. I've been a huge baseball fan since I was thirteen. If you told me then that I would one day have a device I could sit in bed with, watch any ballgame I wanted to and look at realtime, in-game statistics, my unhinged jaw would never return to its rightful closed position.

Or the Kindle (device and app). I could read a book on my phone while waiting for the bus, then pickup right where I left off on my iPad for night-time reading before bed. That…is…simply…amazing.

So with all of our cool toys and services, are we really getting anywhere? By offloading the tedious and mundane tasks of modern living to technology, are we really taking advantage of our spare mental capacity? Are we really evolving as a culture or species to do more and learn more than our ancestors before us? I don't know. Sometimes I look at something like Farmville and I have to just shake my head at the frivolity of it all (money-making frivolity, but c'mon, are you kidding me?) On the other hand the role of social networks  in the Jasmine Revolution in the Middle East is heartening.

We live in a time breathtaking technical accomplishment. Each of us has the capability to achieve amazing things. We also have the equal opportunity to fritter our time and attention away on the trivial. So the question is, now that we can do so much, what are we going to do that's worthwhile?

Some things, like taxes, are pretty easy to make decisions about regarding my time. But others, especially tech/software projects get a little murkier. I'm certainly less-inclined to noodle on go-nowhere personal projects than I used to be. I still enjoy software, but only in a really focused way. These days, I'd much rather spend those extra bits of time and attention on becoming a better drummer.

So, why is it that I went off for a month to fiddle with my blog setup? Good grief, I have written anything in nearly three months. I don't think that "tweaking my setup" is the reason that the posts have dried-up recently. Even though I know that, there's a little voice in the back of my head that says, "Yeah, but what if it were just a little easier to write a post. You'd write more then, wouldn't you?" The evolved, brain-cortex side of me quickly dismissed that notion as rubbish. But my deep-down, brain-stem part of me took the bait. "Of course! What I need is a smoother blogging setup!"

So, after a month of noodling on this project, here I am writing my first post with The New Setup™. It is a Pyrrhic victory if there ever was one. If I'm honest with myself, I'll acknowledge that if posting more is important to me, tweaking my "flow" isn't going to hinder me from writing. This is the lie of productivity tools—they're useless without real passion and attention from the humans using them. But sometimes you just get an itch that you have to scratch, and this is my itch.

The Itch

About a year ago I went through another major effort with my blog by converting it from a Dreamhost-ed Wordpress blog to a static-site generated using nanoc. How I came to use nanoc is a long and uninteresting story.  Nanoc certainly has a lot of nice features, but suffers a bit from complexity-overkill due to its configurability. In retrospect I wish I'd just gone through the pain of switching to Jekyll. But the main reason I went with nanoc was that I was trying to mimic the structure of the Wordpress site as closely as possible and found it difficult to do that with Jekyll.

In retrospect, this was mistake number one. I went through a lot of effort to build a new system that looked a lot like the old system. Sometimes you can't avoid this, but I don't think I asked myself forcefully enough whether I really needed to do that. That had consequences further down the road.

So here I am going along with my nanoc site. The final output is certainly better than I would have achieved with Wordpress. But the sausage-making behind the scenes to get a post into place was not happening. As much I really wanted to get hip to The Markdown Way, I just didn't do it enough to make it stick. I simply couldn't get hyperlink syntax to stay put in my brain. So I had to constantly interrupt whatever I was writing to go look up Markdown syntax again and again. Ugh.

Oh, and images. I had a glorious Automator folder-action that was supposed to help with images. I could drag an image to the top-level directory of the blog and the action would prompt me for a name and put the URL in the clipboard which I could paste write in the post. Oh my goodness, it was so clever and awesome—except that it didn't work half the time. So I'd end up having to copy files around manually and type out image URLs that were often incorrect the first few times. Round and round and round and round. It took a lot of iteration to crank a post out.

Over time I realized that I really missed using MarsEdit. It's just so easy to sprinkle a little formatting here and there or drop an image or two in a post. Links are a snap. It's nice because my meager brain can stay at one level of abstraction while I write. So I started thinking about how I could graft MarsEdit onto my existing setup.

The Scratch

So I ended up writing a little Sinatra app that pretends to be an Wordpress-like XML-RPC server on top of my existing nanoc setup. It sounds simple enough, right? Well, the end result turned out to be somewhat elegant (if I do say so myself), but the path to get there was a festival of Moses-in-the-desert-like meandering.

In the end I've got something pretty slick. This Sinatra app runs through the built-in Apache instance on my Mac using Phusion Passenger, running on a private "alexvollmer.local" address. Since the stock Apache installation is always running on OS X (unless you go out of your way to disable it), I have an always-on local server backing MarsEdit. The great thing about this is that I don't have to remember that part of writing a blog post is starting a server process. I fire up MarsEdit, I write, I post.

This Sinatra app handles drafts and publishing, deleting posts and image upload. When I hit "Send to Blog" it commits the post to the git repository, compiles the nanoc site and rsync's it out to my server. It feels like a "real" blog setup again. I can't say that I'm not a little pleased with how it turned out.

The Rash

But here I am, a month later, and wondering to myself if I've really accomplished anything. I've essentially migrated from Wordpress to a third-party static-site generator (which needed heavy customization along the way). Then I put in another round of customization to make it look like Wordpress again so I can use the same front-end tool. I've essentially poured a boat-load of time adhering to an arguably lousy interface. Had I gone with Jekyll in the first place, I could have used an existing solution to make it work with MarsEdit and gone on my merry way.

Hell, it's not even like I've built a nice re-usable XML-RPC interface for other nanoc users to use. It's so wired into my nanoc setup that it's value as publicly-available source is little more than demonstrative. Sigh. So that's why I'm calling this the dumbest thing I've ever done. I knew exactly what a waste of time this was going to be at every step of the way. I managed to tell myself to be patient long enough to solve the next little hitch in the process of creating this Frankenstein. After clearing several hurdles that I thought would capsize this project, I ended up at the finish line. Success, of a sort, I suppose.

Now one thing I'd like to point out that I started on this Quixotic cause before Brent Simmons started the call-to-arms for static site generators. However, I did mention my windmill-charging exercise to him at a recent Seattle Xcoders meeting and, as he put it, sometimes ya just gotta do things like that. Yes, indeed, sometimes you just need to scratch that itch. So, scratched it I have, and now it's time to get back to the real work.

P.S. Despite all my self-flagellation, this new blogging setup is pretty sweet…

But that's not what I wanted to write about here. What I wanted to share with you was a little bit of how I avoided going crazy while doing this. And while I'm not completely finished with the process, the end is in sight and I can confidently say that the tools and techniques I'm going to describe below were a huge benefit to me.

One of the reasons I struggled so much with this post was that I had a hard time making it not sound like some long-winded commercial for my favorite software tools. Yes, these tools are great. But really this is about so much more than that. It's really about the whole change in mindset and approach that was necessary for me to get stuff taken care of.

So rather than just give you a lame, amateur impression of a MacWorld review, let me put these tools in context. Let me start by laying out a few rules and constraints. I have, on occasion, been prone to being a bit obsessed with the world of so-called "productivity porn". This is the contant fiddling about with your tools and ways of doing things that quickly eclipses your ability to actually get any meaningful work done. It was clear to me, right away, that if I was going to add anything to my toolbox I had to benefit from it immediately. Similarly, any tool that required me to internalize a whole new methodology was just not going to work.

This isn't to say that my game couldn't use some improvement. Rather, it was a recognition of the fact that I simply don't have time to change my ways too much. I need to get stuff done. Lawyers are calling, taxes are due and I cannot be the roadblock that holds things up.

The last principle (if we can call it that) in all of this was that both space and time were limited resources. I had limited amounts of both and in many cases I was willing to trade money for either. Usually this meant paying a professional to take care of something that didn't make sense for me to do (e.g. estate taxes, carrying out an estate sale, etc.). However, in some cases, I had to make choices between space and time.

Scarcity of time probably makes sense, but scarcity of space may not. No matter how frugal or simple a person may have lived, if they reached adulthood they acquired a lot of stuff over the course of their life. Since this was my mother, some of those things I had some emotional attachment to. But the vast majority was "stuff" I simply didn't want or need. That didn't mean that it wasn't my problem though. More on this later.

So, given that I already have job, how did I keep myself from going crazy while trying to get this stuff settled?¹ In short, I used a handful of tools (mostly software) and a lot of discipline.

OmniFocus

I've been a big OmniFocus user since it debuted. I also have the iPhone and iPad versions. Though all three add up to quite a premium, they all solved the biggest problem I had: having one place to dump things I need to do and sort them out later.

OmniFocus let me dump every little thing that came up that needed attention into a single place that I could then orchestrate into a sequence of actions to take. Take for example, the process of filing tax returns. Depending on where you live and when your relative passed away, you may have multiple tax returns to file (the personal tax returns of your loved one, Estate Taxes and fiduciary taxes for the Estate). Given that these weren't tax returns I had planned to file I suddenly had a metric s**t-ton of paperwork to locate.

So let's say that I had a task like "Get date-of-death valuations for all investments". I would create a task in OmniFocus and give it some rough due date. If it was something like a personal tax return that had a known due date, I'd put that due date in. Now clearly a task like that is too big (or it least it was too big to me). So then I would add several sub-tasks to that, one for each financial institution that I needed to contact.

If I had more than, say, three or four sub-tasks and they were particularly onerous, I might spread them out before the parent task's deadline and space-out the due-dates for each child task. This turned out to be extremely helpful because it allowed me to do a little bit each day, without getting too discouraged. And let me tell you, there were a whole lot of tasks on that list that I did not want to do. If I had stuffed a bunch under one task and waited to do them until the last minute (either because I didn't see them or because I procrastinated) it would have made completing them so much worse.

So part of this was a sort of psychological head-game I'd play with myself to keep going. It's like what any endurance athlete does to complete a race. You don't consider the entire course, instead you run to the next block or bike over the next hill. That's all you care about right now. You can deal with the next hill after that.

Because I was often in lots of different places with a varying "kit" of equipment, I needed to access that list from anywhere. Here, OmniFocus' ability to sync with various "cloud services" (I use MobileMe) was a huge help. This brings me to what I might call Golden Rule #1:

Make sure you only have one place for things.

Once you start scattering stuff around in-duplicate, the genie is out of the bottle, and you will have a hell of a time getting it back in. When it's absolutely crucial that you stay on top of things do not use multiple places/buckets/services to store a disjoint set of the same things. You might find a better way to do it, but the cost of migrating while you're in fire-fighting mode is a heavy price to pay.

So I'm not going to sit here and tell you that OmniFocus is the One True Way. There are certainly enough to-do apps out there to choke a whale. All I can tell you is that it was a life-saver for me.

OmniOutliner

So, where OmniFocus was my tool keeping myself on-track, I needed something else to capture more permanent data. OmniFocus would have been completely inappropriate for historical records. Once I complete a task, it's gone. So, instead, I used OmniOutliner.

Now there's nothing particularly special about OmniOutliner for taking notes. In some ways, it's not really a note-taking tool at all. But if you have a proclivity for thinking hierarchically and in outlines (which I do), OmniOutliner is must-have.

Being a keyboard-shortcut junkie, OmniOutliner is the most effortless tool I've come across for capturing information and ideas in outline form. The ability to quickly move up and down in a hierarchy and move items throughout the hierarchy hasn't been done better in any app that I've seen.

So, I have one giant OmniOutliner file called "mom's estate" that has everything in it. It has account numbers, records of every phone conversation I had, when I sent paperwork, when I got responses…seriously…I mean everything.

I used it most often for phone calls (which have been numerous) with attorneys, tax accountants and financial institutions. This was one of the areas where I need to be the most disciplined. If I ever let myself slip and failed to write down the salient points from a conversation I had, I would have eventually stopped doing it and lost a lot of information along the way. Sometimes I would get a call from an attorney while I was out an about. If I wasn't with my laptop, I'd reschedule the call until I could be where I could put my earphones in and have my hands free to type while conversing.

This was a god-send. It helped me stay on top of things in a way I wouldn't have been able to do with my brain alone. Also, because it was all in one place, it was easy to search through my notes when I was talking with someone.

In-Ear Headphones

It sounds dumb, but I used these things all the time. In fact, I went through two pairs during this whole ordeal. Anytime I had a call related to matters of the Estate, I always politely asked the other party to wait while I put my earphones in so I could take notes.

What this was really about was reducing friction as much as possible. There was simply no part of settling the Estate that was enjoyable, but I could at least try to reduce the pain. Just the simple ability to type while talking meant that I took better notes, more consistently. That meant that I had more information I could easily capture and file that I didn't have to hold in my head. And that meant more confidence in my ability to figure this stuff out and get it done and less stress about forgetting something.

DEVONThink Pro Office and the Fujitsu ScanSnap Scanner

I was introduced to DPTO (as the cool-kids call it) by Ryan Davis during one of the Seattle.rb hack nights. DPTO is an amazing, powerful, abstruse piece of software. It's exactly the sort of thing I generally shy away from these days, but it does one thing so extraordinarily well that I plunked down a good chunk of change and spent some time learning its idiosyncracies.

As I began to clear out my mom's house, I eventually whittled down a lifetime of collected paperwork to six banker's boxes. I don't have space to hold six banker's boxes of paper in my house. Hell, I don't have space to hold the paperwork I already have in my house. But I couldn't just get rid of the paperwork, I needed the information.

Enter the Fujitsu ScanSnap document scanner. Combined with DPTO, these two form a rapid-input document scanning tool that allowed me to practically reduce about 80 cubic feet of documentation down to a pile of bytes on a hard-drive.

What's great about DPTO is that it has remarkable OCR technology. It can scan damn near any document, convert it to a PDF, and overlay searchable, selectable text. All of those statements went from pounds of paper to searchable bits on my harddrive. Not only did this help me save space, it also made all of this documentation searchable which was a huge help trying to piece-together my mother's financial picture (a necessary thing for probate and various taxes).

Now this wasn't easy. In this case, I had to face the Sophie's Choice of trading time for space. It took a lot of time to scan all of those documents. So my solution was to do a little bit consistently and to do it during "down-time". I spent a lot of football games this season with a beer, a giant stack of documents, a paper-shredder and my document scanner.

So it sucks that I had to spend time doing it, but now I've recovered a bunch of physical space and I've made all of these documents easier to get to. In truth, I won't need to access the vast majority of documents, but on the off chance I need one, it would sure suck hard not to have access to it. Keeping all that paper around would be giving in to the fear that "I might need that someday". I hate fear as a motivator. It's very powerful, but things often turn out very poorly as a result. So DPTO and the ScanSnap allow me to make it very cheap to keep all of this for the remote possibility that I'm going to need it someday. All of this boils down to what I'll call Golden Rule #2:

Never pay freight on something you "might" need.

DPTO can do a lot of things. I'm not sure I would really use it as my "all-in-one" bucket. In a lot of ways it's a pretty fussy tool that, in my opinion, takes a bit too much to maintain. However, just using it as a document scanner and archiver makes it, to me, totally worth having.

Backups

Now I'm all for going digital, but doing so means that I've taken on the risk of the inherent fragility of today's storage technology. Paper, in general, doesn't fail for mysterious reasons (perhaps this is why MBTF isn't published on the side of a ream of paper), but disks certainly do.

The only way I could move more things to digital and not put myself in a potential crisis situation with a hardware failure was backups. Lots and lots of backups. And not just one backup, but multiple backups. And not just on one medium, but several. And not just in one location, but several.

So my current backup strategy now uses a combination of Time Machine, Super Duper and Carbonite. I use Time Machine because it's the fastest backup that saves me from the dumb accidental file delete. It's really a first-line of defense and no more.

I use Super Duper for local, full backups. What's great about Super Duper is it's ability to create a bootable volume, which means that if my main drive fails, I can still get to the information without necessarily having to fix the main hard drive first. I use a rotating pair of LaCie drives and backup every other day (I have a repeating OmniFocus task to remind me).

These are great tools, but they use a common medium (FireWire drives) and they are in the same location (my house). I might solve the location problem by coming up with some backup-swapping routine with someone I saw on a regular basis. That is, each person would have two or more Super Duper backup disks, and would routinely give one of them to a trusted source. You would then each rotate your backups and at your next meeting, "trade" backup disks. I use Espionage to encrypt my important data so I'm not worried about it falling into the wrong hands.

I haven't done this yet. But even if I did, I'd want another form of backup, so I bought a Carbonite account to push data out to the "cloud". There are a growing number of services out there like Carbonite and they probably work just as well. The point here is peace of mind. It's like buying insurance. Sure, it's very unlikely that my house is going to catch on fire and burn to the ground with everything in it, but why worry about it at all? And if it did happen, why make the loss worse. Sixty bucks a year seems like a pretty cheap price to pay for the ability to sleep better at night. We'll call this Golden Rule #3:

Computers are great but fail. Make sure that fragility doesn't keep you up at night.

Dropbox

The last tool I'll mention is Dropbox. Dropbox is awesome for so many reasons and I could go on and on about it. But what was really useful for me in this context was the ability to selectively share documents with others. I have some pretty serious, private documents going around that I don't just want to stuff in an email. Yes, there are encrypted zip files and PDFs but there isn't a good standard for the former across platforms and you still have the problem of getting the password to the other party in a secure way.

With Dropbox I can share documents securely with my tax accountant. Hell, it's even easier than email because I don't have to deal with attachments and anti-virus software mistakenly preventing my documents from being received.

Now not all lawyers and accountants are going to be down with such modern technology. In our litigation-happy society people in these professions tend to be pretty conservative in adapting new technologies. All I can say is that I was very fortunate to work with the sort of folks that would go along with something like Dropbox.

FAX Modem

I can't believe I had to buy this thing, but given the spirit of the post, I would be remiss if I didn't mention the humble little Zoom modem. It is extraordinarily depressing and aggravating that in this day and age, the vast majority of financial institutions out there will accept a fax, but won't use email or some web-based tool.

At first I had earnestly hoped that things were going to change and that I would only need to make the occasional trip to the local Kinko's to send a four page fax for five bucks (thank you very much). As time went on though, I realized that any companies willing to do business over the internet were the exception rather than the rule. So, as much as I hated to do it, I finally had to admit that we haven't really evolved as much as I'd hoped and had to trudge down to the local Best Buy to buy a stupid little USB fax modem.

Now, as much as I hate the fact that I have to deal with faxes, the fax software integrated into OS X makes it pretty easy. Most folks don't know that printing a document in OS X is really a gateway to sending documents to all sorts of interesting places (least of which is a printer).

For example, for a number of financial institutions I had to write a cover letter and include court documentation to establish authority over my mother's accounts. I could write the cover letter in Pages, insert the PDF document then "print" it to the fax modem. The built-in fax print location even allows you to optionally compose a cover page right in the dialog without the need for including it in your documents.

Conversely, accepting a fax is pretty simple too. You plug in your modem, set it up to receive calls. When you get the document, you can "print" anywhere. In my case I would often "print" incoming documents straight into the DEVONThink inbox where I could OCR-scan it and file it away for further use. No need to print or scan a damn thing. Bits from one end to the other thank you very much.

Conclusion

So what's this all about then? Why write this post? I'm no productivity guru, but I felt like I found a good set of tools and techniques in this experience that might be useful for others. None of these really help with dealing with the unexpected loss of my mother, but they were a huge help in making the onerous task of settling her Estate palpable and possible.

Notes

In the documentary "Rush: Beyond the Lighted Stage" there's a great scene where Neil Peart is sitting down with the jazz legend Freddie Gruber. In the interview they talk about the first time the two drum masters met. They didn't play a note, but instead focused on the dance-like motion of percussion. This was an epiphany for Mr. Peart, who is considered by many to be the pre-eminent rock drummer of all time. Imagine being at the top of your game for nearly forty years and suddenly having a whole new way to look at your art! Instead of considering drumming to be the placement of notes in time, Gruber thought of drumming as a rhythmic movement of the body that happens to coincide with what we call "music".

I bring this up because I too, am going through a muscle-memory epiphany. I've decided to try vim as my main text editor. I've spent a lot of years with Emacs and the last few with TextMate. There are things I love about both, but neither is perfect. I've worked with lots of smart people who swear by vim, so why not try it out and see what it's all about?

I've always abhorred the silly "holy-wars" between Emacs and vim. It's the worst kind of provincial rah-rah-ism in a world where perspective and decent manners are few and far between¹. I am not "switching sides" because, frankly, I was never on a particular "side" to begin with. These are simply tools—nothing more.

Here's the funny thing: I've noticed that those that trumpet one over the other often have no experience with the alternative. I can think of maybe one or two other developers I know that have truly attempted to become experts with both editors. Everyone else is simply guessing about "the other side". For a profession that espouses hard data and facts, it's amazing how heated the debates can be when each side has so little information about the other.

But I digress…

The Big Question

So it got me to thinking. Why is that so few people have tried to master both? Is it really impossible to learn a whole new set of keyboard motions once you've mastered another?

I often joke with people that the one "super-power" I have as a developer is my ability to remember hot-key combinations. When I code in Xcode I know all the hot-keys to make it sing. When I'm writing in TextMate I know how to make it go. I don't consciously execute keyboard motions, they are just part of the development experience with that tool.

The reason I joke about it is because I don't really think that it's all that extraordinary. It's true that it's not something I see in a lot of other developers, but I don't believe it's because they can't do it. I think it's because they've never learned how to learn motion.

Six Strings and the Truth

Looking back, I've realized that the greatest learning experience I ever had was learning to play the guitar. I started when I was 14 and I was deadly-serious about it from day one. I devoured everything I could find on the subject. I learned music theory, practiced all manner of finger exercises and played and played and played.² As a result, I'm a pretty good rock guitar player.

The funny side-effect of this experience is that I taught myself how to learn. I can still remember learning basic music-theory on a train ride when I was a teenager. It just hit me like ton of bricks. It was all so simple. After that, the mystery of A♭7♯5 was gone. It was obvious what that chord was and what it sounded like.

Not only did I learn how to learn in a strictly academic sense (you should see the binders full of notes I've kept over the years), but I also learned how to teach my fingers to learn. As a result keyboard shortcuts are pretty easy to master. It's like learning another song. Nobody agonizes over having to remember each and every finger-movement to play a new tune. You practice it, play it a few times, see the patterns in it and then it's just there. You don't think about it. Why should mastering another editor be any different?

Brain Stretching

I'm fascinated with the learning process. I've made it a point in my career to make learning a continous process. I simply do not get people who do the same thing for years on end. How do they not go insane?

So I'm always looking for ways to stretch myself. This year I'm extending my learning more broadly and decided to start taking drum lessons. Wow. Wanna feel awkward in less then two seconds? Try playing the simplest syncopation beat requiring independent movement from your limbs. It's a humbling experience. But, man, I can't wait to figure it out!

Circling Back

So what does this have to with editors? Honestly, I'm not quite sure yet. But I think most developers have a pretty narrow view of what they're willing to learn. As often as not, I think folks just stick with whatever they happened to stumble upon the first time. Think about it. How many things are central to your game that are the result of circumstances? Are you an Emacs master just because of that one job you had with that one graybeard who taught you all of the tricks? How would your career have turned out if you hadn't met that guy?

It's not just about editors either. You see it in every dude who lives and dies with a single programming language or platform. I've got my favorites too, but they've never been eternal. I can't help but think that such a narrow view contributes in some way to the mountains of terrible software out there in the world. How could anyone possibly build anything of quality with such a small view of the world?

So I don't think I'll be practicing my vim with a metronome the way I practice stick-control exercises or the way I mastered the harmonic minor scale. Instead we practice these motions in our daily development work as a matter of course. The fact that our livelihoods are tied to this activity I think makes people hesitant to totally re-jigger their game. I get that. But I also think it's not as scary as it sounds (I seem to do this in some way every two years or so) and it gives you a much broader view of the world.

So Emacs-dudes? Listen up. C-x C-c outta that sucker and try something else. You snotty vim dudes? Hit :q and try life on the other side. You may still hate it, but don't ever pretend that you couldn't get better by learning something radically different.

Footnotes

1. Seriously. There are times when I'm just embarassed to call myself a software developer. How did we all become so incredibly socially inept?
2. At one point in college, I wrote a program (in Scheme!) that generated random chord combinations (e.g. Am7, C♯maj7, etc. etc.) I taught myself to to go from one voicing to another with the least amount of fretboard movement. Eventually I extended this to practicing arpeggios too. It was like a little game and it was fun to get better and better at it.

As I've started writing more UIAutomation tests, I've found that there is a common cycle to each test. Generally there is some bit of navigation to get yourself somewhere in the app, then you assert a bunch of things about the view, and usually navigate somewhere else. I don't think this is far off from how you might verbally describe how the app might work. For example, you might say: "Go to the main screen, the left button should say 'Cancel' and the right button should say 'Done'. Fill in the username and password fields then tap the 'Done' button. On the next view you should see the user name in a table cell and another cell to add another account".

It's fairly straightforward process translating that description into test code:

test("account setup", function(app, target) {
      // initial view
      window = target.mainWindow();
      navBar = window.navigationBar();
      assertEquals("Cancel", navBar.leftButton().name());
      assertEquals("Done", navBar.rightButton().name());
    
      table = window.tableViews()[0];
      assertEquals("Enter your credentials:", table.groups()[0].name());
      assertEquals(2, table.cells().length);
    
      userName = table.cells.firstWithName("username");
      userName.textFields()[0].setValue("user boy");
      password = table.cells.firstWithName("password");
      password.textFields()[0].setValue("sekret");
    
      navBar.rightButton().tap();
    
      // verify new account on settings view
      navBar = window.navigationBar();
      assertEquals("New", navBar.rightButton().name());
      table = window.tableViews()[0];
    
      assertEquals(2, table.groups().length);
      assertEquals("Your accounts:", table.groups()[0].name());
      assertEquals("", table.groups()[1].name());
    
      assertEquals(2, table.cells().length);
      assertEquals("user boy", table.cells()[0].name());
      assertEquals("Add Account", table.cells()[1].name());
    });
    

One window's worth of activity isn't so bad, but once you get beyond that, it's pretty easy to lose track of what you're trying to do. Comments help a bit, but if our code had the same structure that matched the way we think about our tests, it would be even better. Now you can do exactly that with a new assertion method in tuneup: assertWindow(). Here's the previous example, re-stated with assertWindow():

test("account setup", function(app, target) {
      assertWindow({
        navigationBar: {
          leftButton:  { name: "Cancel" },
          rightButton: { name: "Done" }
        },
        tableViews: [
          {
            cells: function(cells) {
              cells.firstWithName("username").textFields()[0].setValue("user boy");
              cells.firstWithName("password").textFields()[0].setValue("sekret");
            }
          }
        ],
        onPass: function(window) {
          window.navigationBar().rightButton().tap();
        }
      });
    
      assertWindow({
        navigationBar: {
          rightButton: { name: "New" }
        },
        tableViews: [
          {
            groups: [
              { name: "Your accounts:" },
              { name: "" }
            ],
            cells: [
              { name: "user boy" },
              { name: "Add Account" }
            ]
          }
        ]
      });
    });
    

So what's going on here? Using assertWindow(), we've replaced several lines of procedural code with a javascript object-literal that declares how to match the current window. The property-nesting in the object given to assertWindow() (the "expectation") is matched against the property-nesting of the UIAWindow object representing the current view. For example, when we declare a navigationBar property in our expectation object, it is matched against the object returned by the navigationBar() function invoked on the main window (an instance of UIANavigationBar). Then the rightButton property is matched against the object returned by the rightButton() method (an instance of UIAButton). Finally the name property is matched against the object returned by invoking the name() function on the button, which in this case is a String.

You can do more than just match Strings though. You can also match numeric literals or regular expressions. These are done by applying the assertEquals() and assertMatch() functions, respectively. You can also completely customize your assertions by providing a function for the property. The corresponding object will be given to the function as its single argument. If any exceptions are thrown the assertion fails, otherwise the assertion passes.

assertWindow() also knows how to handle properties that return arrays (or UIAElementArrays). In the example above we matched the tableViews property by declaring an array with one element. The number of elements you put in the property declaration implicitly makes an assertion about the number of elements that should be in the property. If you are only interested in matching some of the elements in an array, you can set null for the elements in the array you don't care to make assertions about. This will allow you to still pass the array-length check without forcing you to write assertions you don't care to.

The onPass property is an optional function you that will be invoked if all of the assertions pass. This is a handy way to group assertions and actions together in one package. In the example above, we use the onPass function as post-assertion navigation to the next screen.

You can find all the gnarly details in the assertions.js file in tuneup. As always comments, feedback, forking and patches are all welcome and appreciated. Now go out and assert yourself!

Kicking it Caveman Style

Before diving into operations and queues, let's imagine doing this in the "primitive" style. Let's say we have three operations: A, B, and C. Each does some asynchronous work and we have no idea when it will complete. The primitive way of handling this is to have each callback methods invoke a single method that checks for the results of all three units of work. Only when all three have returned do we continue processing. It might look something like this:

// our final unit-of-work when all other operations have completed
    - (void)checkAllOperationsAndProceed {
      if (self.resultA && self.resultB && self.resultC) {
        // do that magic thing you do
      }
    }
    
    // our mythical call-back methods
    
    - (void)operationADidComplete:(id)result {
      self.resultA = result;
      [self checkAllOperationsAndProceed];
    }
    
    - (void)operationBDidComplete:(id)result {
      self.resultB = result;
      [self checkAllOperationsAndProceed];
    }
    
    - (void)operationCDidComplete:(id)result {
      self.resultC = result;
      [self checkAllOperationsAndProceed];
    }
    

You can think of this as a "last-one-in-wins" pattern. It's kind of ugly and doesn't scale well with more complicated dependencies. It gets worse when you have specific threading concerns like working with Core Data, which is notoriously fussy about threads.

I had this exact problem using Core Location and Game Kit. I use Core Location to first retrieve the current location, while simultaneously communicating with other devices via Game Kit. When the location is resolved, we then attempt to reverse-geocode it which results in another asynchronous operation. All of these bits of work are intertwined and update the underlying persistent model in different ways. It didn't take too long before it spiraled completely out of control. What's more, it was easy to hit odd edge-cases with synchronizing various threads to keep Core Data happy. Since I wasn't getting consistent failures, it was clear that there were race-conditions in the code. Then a little voice in the back of my head reminded me that NSOperation instances can have dependencies between each other. A ha!

A More Elegant Weapon, for a More Civilized Age

Maybe the caveman approach doesn't offend your sensibilities for such a trivial example. But imagine a much more complicated set of dependencies, perhaps something that looked like this ¹:

Now imagine all the if checks and the spaghetti code that would be required to make this work. I don't know about you, but this makes my Spidey-sense tingle…and not in a good way.

So let's turn these into NSOperation instances with dependencies. We would start with method prototypes and properties declarations like this:

@property (nonatomic, retain) NSOperationQueue  *operationQueue;
    @property (nonatomic, retain) NSOperation       *operationA;
    @property (nonatomic, retain) NSOperation       *operationB;
    @property (nonatomic, retain) NSOperation       *operationC;
    @property (nonatomic, retain) NSOperation       *operationD;
    @property (nonatomic, retain) NSOperation       *operationE;
    @property (nonatomic, retain) NSOperation       *operationF;
    
    - (void)performOperationA;
    - (void)performOperationB;
    - (void)performOperationC;
    - (void)performOperationD;
    - (void)performOperationE;
    - (void)performOperationF;
    

We'll do the actual work in instance methods of our class and wrap each one with an instance of NSInvocationOperation. Here's our setup method:

- (void)initializeOperations {
      self.operationQueue = [[[NSOperationQueue alloc] init] autorelease];
      [self.operationQueue setMaxConcurrentOperationCount:1];
      
      self.operationA = [[[NSInvocationOperation alloc] initWithTarget:self
                                                              selector:@selector(performOperationA),
                                                               object:nil] autorelease];
                                                               
      self.operationB = [[[NSInvocationOperation alloc] initWithTarget:self
                                                              selector:@selector(performOperationB),
                                                               object:nil] autorelease];
                                                               
      [self.operationB addDependency:operationA];
                                                               
      self.operationC = [[[NSInvocationOperation alloc] initWithTarget:self
                                                              selector:@selector(performOperationC),
                                                               object:nil] autorelease];
                                                               
      self.operationD = [[[NSInvocationOperation alloc] initWithTarget:self
                                                              selector:@selector(performOperationD),
                                                               object:nil] autorelease];
                                                               
      [self.operationD addDependency:operationA];
      [self.operationD addDependency:operationC];
                                                               
      self.operationE = [[[NSInvocationOperation alloc] initWithTarget:self
                                                              selector:@selector(performOperationE),
                                                               object:nil] autorelease];
                                                               
      self.operationF = [[[NSInvocationOperation alloc] initWithTarget:self
                                                              selector:@selector(performOperationF),
                                                               object:nil] autorelease];
    
      [self.operationE addDependency:operationD];
      [self.operationE addDependency:operationF];
    }
    

One interesting thing to note here is that the operations that have dependencies could be queued right away. They won't execute until their dependent operations execute first. Another cool feature of NSOperations is that dependent operations don't have to execute in the same operation queue. Believe it or not, I've actually needed this very feature.

To execute these operations in the correct order, we simply need to add them to the queue. If the dependencies are correctly declared, the order in which we add them is irrelevant. Instead the final execution order is determined by the dependency graph of the queued operations. So one of our asynchronous callback methods might look like this:

- (void)didGetSpecialCallback:(NSDictionary *)userInfo {
      self.firstName = [userInfo objectForKey:@"firstName"];
      self.lastName = [userInfo objectForKey:@"lastName"];
      [self.operationQueue addOperation:operationD];
    }
    

The call to add operationD is effectively like invoking the performOperationD method except that it won't get executed until it's two dependent methods, operationA and operationC have completed first. But the beauty of all of this, is that my callback code doesn't have to care about that ordering.

Once your code has enough concurrent asynchronous operations that are dependent on each other, this can be a great way of straightening out what work is going to get done in a thread-safe way.

Rules of Thumb

Queues Make Life Easy

One common principle of concurrency is being able to replace mutexes (muteces?) with queues. Any time you have shared data operated on by multiple threads, you can synchronize them with serial operations in a single queue instead of using explicit locks. The operations in my app are lightning-quick so I'm not worried about slowing things down with a single-threaded queue. A great side-effect in using operations with dependencies is that I know precisely what order those units of work are going to execute. This takes away a slew of edge-cases that are difficult to reproduce and debug.

Be Assertive

Because you know the order of operations, you can put a nice safety-net underneath yourself as you develop with a liberal application of assertions. For example, in the operation 'D', which requires that 'A' and 'C' have completed, the code can expressly assert that effects of 'A' and 'C' have been applied. If they aren't, it means there's a programming error:

- (void)performOperationD {
      @try {
        NSAssert(self.firstName != nil, "First name is nil!");
        NSAssert(self.lastName != nil, "Last name is nil!");
    
        // rest of method elided for readability
      }
      @catch (NSException * e) {
        NSLog(@"ERROR associating place with encounter: %@", e);
      }
    }
    

Try, Try Again

The last piece of advice is to make sure you put @try/@catch blocks in the methods that are executed by your operations. Since these are on a separate thread, you may not see a standard backtrace when an exception is thrown. You need to handle this yourself, otherwise things may silently fail. This is especially true of Core Data which will throw exceptions for programming errors like accessing objects with the wrong thread.

Footnotes

1. This is a simplified version of the dependency graph of an application I'm currently working on, so it's not just some trite example I invented to make a point. Rather than saddle you with the mundane details of each operation, I've chosen to go with the rather abstract, if boring, A-B-C nomenclature.
2. Okay, that's not strictly true. Operations can also have different priorities which also affect the execution order. However assuming that all operations have the same priority, the original statement still holds true.

So what's with the name? Kinda weird, isn't it? I went around and around on various names. Eventually I settled on something related to space. I really love the look and design of Stanley Kubrick's 2001: A Space Odyssey. Not so much for how it looks retro from our modern perspective, but how futuristic it looked from 1969. I settled on having the word "capsule" in the name somewhere because I liked the idea of a self-contained package. Right now it's just me, but even if I ever bring on partners, I don't expect it to be too big. So "space station" would have been inappropriate given the philosophy behind the company.

Why "radiant"?. Rhythmically, I knew I wanted a name in the form of "insert-adjective-here Capsule". I sat down one afternoon, in complete silence, and just wrote adjective after adjective on a sheet of paper. Just trying out words I liked, and listening to how they sounded together. Finally I hit upon "Radiant Capsule" and I felt like I had a winner. I just really like the way it sounds.

The design was bootstrapped by the logo which was created by Ryan Schroeder. After that I was off and running. I don't know whether you can call the site design and business cards "good", but it is certainly some of the best visual design I've ever done. Like all of this, it's a learning experience.

It's About Dependencies

I think it's worth backing up and explaining why I've spent so much time noodling on test fixtures. Any system of moderate complexity has features that depend on the existence of other features. For example it doesn't make sense to have a "recover your forgotten password" feature without having an authentication feature that requires a password in the first place. If you drew each of your features in a box then drew a line between each feature and its prerequisite features, you would have a graph of feature-dependencies. ¹

When we design a system, we look for patterns in these dependencies. If we're smart enough and have enough experience, we'll do our best to reduce those dependencies, and figure out how to minimize the coupling between them. When we get this wrong, we something that gets hard to maintain and change over time.

A big part of using dependency-analysis in our design is figuring out exactly at what level our features are dependent upon each other. In our forgotten-password example, the question to ask is how much of the user-authentication feature does recovering a forgotten password depend on?. To answer this question, you have to start imagining different ways the authentication feature could be implemented. Which ones would break the forgotten-password feature? Which ones would make it irrelevant?

From the perspective of system-design it makes no sense to tie the implementation of the forgotten-password feature to the form-submission process of authentication. Instead, these features should be coupled at wherever you store authentication information (likely a database). You should be able to modify the authentication process without directly breaking the forgotten-password feature.

Returning to automated testing, the same idea applies. But I think a lot of people don't factor their tests this way. Instead, a lot of test-writers get so wrapped-up in code reuse that they often recycle tests of dependent features as a way of getting the SUT into a starting state. This is a mistake. Just like we wouldn't over-couple the implementations of dependent features, we should't over-couple our tests for them. If we do, we end up with a very brittle suite of tests.

We also end up losing a lot of feedback from the testing process. If the authentication test is failing, does that necessarily mean we should invalidate all the other tests that use authentication information? Logically, the answer is no. But with test-reuse that's exactly what we get. So when one feature breaks and a bunch of tests fail, we have no idea just how broken the system really is.

A New Perspective

Coming back to UIAutomation, I made the decision early on to build test fixtures that focused on the current persistent state of the system. Much of the logic of the application revolves around the state of Core Data database. I made a conscious decision to couple the features at the persistent-store layer so, by extension, I've done the same for my test-fixture code.

In my first attempt, I built a rather complicated Rube Goldberg contraption that simply wasn't going to scale as I added more fixtures. The first problem I needed to solve was to cut down on the amount of build variation. My original solution involved a separate build-configuration for each test-fixture. The problem was that my tests had an implicit dependency on a particular test build, but I had no way to express that with UIAutomation. The best I could do was simply add comments to the top of my tests. Blech. Too easy to get that wrong and waste a bunch of time figuring out why things weren't working. This is not what I want out of my integration tests.

So the first step was to get rid of the separate build configurations. I replaced them with a single "Integration Test" build configuration that defines a build-time variable that is used in my app delegate like so ²:

- (void)applicationDidFinishLaunching:(UIApplication *)application {    
    #ifdef MEECHU_RESET_FOR_TESTING
      self.testFixturesController = [[[TestFixtureViewController alloc] initWithDelegate:self] autorelease];
      [window addSubview:[self.testFixturesController view]];
    #else
        [window addSubview:[navigationController view]];
      [self checkFirstTimeUser];
    #endif  
      [window makeKeyAndVisible];
    }
    

If the MEECHU_RESET_FOR_TESTING compiler flag is set, we insert a special test-fixture controller into the UI. That view controller is simply a control-board for running any number of pre-defined test-fixture operations. Here's an example of one such operation that completely nukes my Core Data database:

@implementation ClearAllDataOperation
    
    - (void)main {
      MeechuAppDelegate *meechu = (MeechuAppDelegate *)[[UIApplication sharedApplication] delegate];
                                                        
      NSPersistentStoreCoordinator *coordinator = [meechu persistentStoreCoordinator];
      NSError *error = nil;
      for (NSPersistentStore *store in [coordinator persistentStores]) {
        [coordinator removePersistentStore:store error:&error];
        if (error) {
          NSLog(@"Unable to remove persistent store %@: %@", store, error);
          return;
        }
        
        [[NSFileManager defaultManager] removeItemAtURL:[store URL] error:&error];
        if (error) {
          NSLog(@"Unable to remove persistent store file %@: %@", [store URL], error);
        }
        
        if (! [coordinator addPersistentStoreWithType:[store type]
                                  configuration:nil
                                            URL:[store URL]
                                        options:[store options]
                                                error:&error]) {
          NSLog(@"Unable to create new persistent store at %@: %@", [store URL], error);
          return;
        }
      }
    }
    
    @end
    

I have a couple of other test-fixture operations like this. All of them fiddle with the underlying Core Data database.

The next step was to create some kind of first-class representation of this "control-board" in my JavaScript layer. So I wrote a small wrapper for the test-fixture view controller:

TestFixtures = {
      ensureFixturesMenu : function() {
        this.target = UIATarget.localTarget();
        this.application = target.frontMostApp();
        this.mainWindow = this.application.mainWindow();
        assertEquals("Test Fixtures", application.navigationTitle());
      },
      
      addTwitterAccount : function() {
        this.ensureFixturesMenu();
        cell = this.mainWindow.tableViews()[0].cells().firstWithName("addtwitteraccountoperation");
        cell.tap();
      },
      
      addEncounters : function() {
        this.ensureFixturesMenu();
        cell = this.mainWindow.tableViews()[0].cells().firstWithName("addencounterhistoryoperation");
        cell.tap();
      },
      
      clearAllData : function() {
        this.ensureFixturesMenu();
        cell = this.mainWindow.tableViews()[0].cells().firstWithName("clearalldataoperation");
        cell.tap();
      },
      
      close : function() {
        this.mainWindow.navigationBar().rightButton().tap();
      }
    };
    

With these in place, the final task was explicitly defining the test-fixtures for each automation test. Now the pre-requisites for the test are specified right in the test, which greatly clarifies the pre-requisites for the test:

#import "tuneup/tuneup.js"
    #import "fixtures.js"
    
    test("Additional Account", function(target, app) {
      TestFixtures.clearAllData();
      TestFixtures.addTwitterAccount();
      TestFixtures.close();
    
      // test continues here
    });
    

Now when the tests run, the test-fixture view controller appears, the test pokes a button for each fixture operation it requires, dismisses the test-fixture setup and continues on its merry way.

Conclusion

This is definitely a step forward, but I think there is some more refinement to be had. First, there's still too much duplication for my tastes. For each new test-fixture operation I have to update the view controller as well as the JavaScript wrapper file. I don't think it would take too much effort to automatically generate the view controller and wrapper JavaScript code simply by snooping the directory that contains the test-fixture operation classes in the same way Xmo'd does for Core Data models and generated classes.

Footnotes

1. It's amazing how much a simple drawing exercise can reveal about your project in ways that simply reading the code doesn't. I've found it really helpful to sketch stuff out with paper and pencil or use OmniGraffle for quick box-sketches.
2. You could argue that having separate builds isn't strictly necessary. All I'm really trying to do is avoid having test-related code in my final application bundle. That could simply be expressed by the differences between the stock "Debug" and "Release" build configurations.

One of the biggest "a-ha!" moments was triggered by creating my favorite slide in the whole presentation, which came near the end after we'd covered all the ins and outs of OCMock.

This phrase came to me as I thought about the "design pressures" that TDD often brings to your code. To me, these pressures are one of most important (yet subtle) forces that TDD brings to bear. TDD practitioners often refer to TDD as a feedback mechanism. To me, this notion of where to draw the lines of responsibility became really clear when I thought about the first place I often do it—in my tests.

Let me explain what I mean by that phrase. First, I should say that I tend to bring in mock objects in a limited set of circumstances. It's not the first tool I reach for in my unit-tests. I don't like tests that simply re-assert the implementation as I think it gives you a false sense of security. The times when I will use mocks are:

• When I'm replacing third-party collaborators that I don't control
• When creating a stub would just obscure the test
• Writing single use-case test stubs. I'll wait until I get three of them before I factor out a common stub.
• When I need to test for side-effects (e.g. logging or caching)

The first reason is the primary motivation I'd have to consider a layer of abstraction. First, because (obviously) the third-party is under someone else's control, but also because I usually want to simplify a more fully-featured and general API to something more specific to my domain. By encapsulating the details of something like GameKit into a little façade class, I've built a nice little box around it that doesn't leak into other classes.

So maybe it's not an earth-shattering realization. Maybe it's just that I finally have a clearer articulation of the idea in my head. Like I said, it's amazing what you learn when you teach it to someone else.

Why We Do What We Do

Before moving forward, it's probably worth reviewing what my goal is with UI Automation (or any integration testing setup for that matter). To me, testing, whether it's unit-testing or integration-testing, is all a matter of economics. There is a balance to be struck between the up-front investment in the test harness and the long-term risk mitigation that it provides. One of the things that has frustrated me about test automation in Cocoa is how expensive it can get to build and maintain this stuff compared to other environments I've worked in. At a certain point, if the cost is too high, I'm perfectly willing to abandon the effort.

I like to think of it like the last scene of the film THX-1138. There's a wonderful shot in the command and control center that shows the running cost of pursuing the escaped fugitives. At a certain point the cost exceeds the budget of the pursuit and the hounds are called off. That's somewhat like my attitude toward test automation. It's not an absolute, totalitarian, philosophical must-have. I only insist on it when it makes sense.

Setting the Stage

The UI Automation toolkit provides a portion of what I think we need for an effective integration-test suite, but not all of it. A good suite would have the following properties:

1. runs on the device or simulator with equal ease
2. has a rich API to dig deep into the details of the running system
3. is extensible so that we can build abstractions over the aforementioned rich API
4. is completely automatable from the command-line for continuous integration builds
5. provides a way to set your application in particular states as a pre-condition for tests

UI Automation covers the first three items, but completely misses on the fourth item. It doesn't help with the last item, but doesn't necessarily preclude a solution either.

What I did was create a special block of code in my application delegate that looks for some extra configuration in the application plist file. Since I don't plan on ever needing any of this stuff in a release, it's all guarded by an #ifdef statement.

- (void)applicationDidFinishLaunching:(UIApplication *)application {    
    #ifdef MEECHU_RESET_FOR_TESTING
      NSArray *operations = [[NSBundle mainBundle] objectForInfoDictionaryKey:@"MUTestHarnessOperations"];
      if (operations) {
        NSOperationQueue *queue = [[NSOperationQueue alloc] init];
        [queue setMaxConcurrentOperationCount:1];
        for (NSString *opName in operations) {
          Class clazz = NSClassFromString(opName);
          NSOperation *op = (NSOperation *)[[[clazz alloc] init] autorelease];
          [queue addOperation:op];
          NSLog(@"Enqueued operation: %@", NSStringFromClass(clazz));
        }
        
        [queue waitUntilAllOperationsAreFinished];
        [queue release];
      }
    #endif
    
      [window addSubview:[navigationController view]];
      [window makeKeyAndVisible];
    }
    

In the special block, if I find the string array associated with the key MUTestHarnessOperations in the application plist, I treat each string as a class name which is instantiated and added to an operation queue. Now I don't have any concurrency requirements which led me to use an NSOperationQueue. Instead, it seemed like a nice way to package up discrete bits of work.

One of these setup classes creates an account (an instance of MUAccount) that some of my tests rely on. It looks something like this:

@interface AddTwitterAccountOperation : NSOperation {
    }
    @end
    

#import <CoreData/CoreData.h>
    
    #import "AddTwitterAccountOperation.h"
    #import "MUAccount.h"
    #import "AccountType.h"
    #import "MeechuAppDelegate.h"
    
    
    @implementation AddTwitterAccountOperation
    
    - (void)main {
      MeechuAppDelegate *meechu = (MeechuAppDelegate *)[[UIApplication sharedApplication] delegate];
      
      NSPersistentStoreCoordinator *coordinator = [meechu persistentStoreCoordinator];
      NSManagedObjectContext *moc = [[NSManagedObjectContext alloc] init];
      [moc setPersistentStoreCoordinator:coordinator];
      
      MUAccount *newAccount = [MUAccount insertInManagedObjectContext:moc];
      newAccount.identifier = @"alexvollmer";
      newAccount.type = @"twitter";
      newAccount.password = @"supersekret";
      newAccount.enabled = [NSNumber numberWithBool:YES];
      
      NSError *error = nil;
      if ([moc save:&error]) {
        NSLog(@"Added twitter account for 'alexvollmer'");
      }
      else {
        NSLog(@"ERROR: Unable to create test twitter account: %@", error);
      }
       
      [moc release];
    }
    
    @end
    

Once I've built up a library of these setup actions, I create a different build configuration for each combination of setup actions. Each of those configurations uses a specific application plist file with the MUTestHarnessOperations key set to an array with the appropriate setup class names.

To create a new build configuration, I just duplicate the existing debug and modify two settings: the name of the application plist file, and defining the MEECHU_RESET_FOR_TESTING symbol for the preprocessor.

If I build and run a configuration of the application with a particular collection of setup actions, that application will always run with those setup actions. This is exactly what you need for repeatable testing.

Now it's simply a matter of choosing the correct application build and test script in Instruments:

You can save your Instruments files off to capture the combination of the test script and executable. One thing to keep in mind is that Instruments won't build your code. If you have a failing test and you're making fixes, you need to re-build the executable in Xcode then re-run the test in Instruments. You don't need to reconfigure Instruments, it will just use whatever version of the executable you have at the specified path.

Conclusion

OK, this works, but I think it's pretty hacky. There are still way too many steps to go through and interface points that need to be configured correctly. I'm sure there's a more elegant, less error-prone way to do this. If you think of it, please tell me. I'm only 50% satisfied with this.

There's a lot I really like about UI Automation, but it is missing a few key ingredients that make me think long and hard about using it long-term. This test setup approach is somewhat workable. Now we just need to address the remaining issues of scriptability and complete automation.

UI Automation is both a probe for Instruments as well as a JavaScript library provided by Apple to exercise and validate a running application. In this case, "running application" isn't restricted to the simulator—you can also automate the application on a real device. To my knowledge, this is the first time I've heard of anyone being able to do this.

This is huge. Having the ability to automate workflows in your application yields two benefits: you cut down on manual testing which saves you time, and you can rely less on your memory to execute all your tests. Instead, you just push a button (okay, two or three buttons) and run your full regression suite. Have I piqued your interest yet?

Preparing Your Application

First, you need to do a little groundwork to prepare your application to work with the automation tool. The UI Automation library relies on accessibility information in your UI, so adding a little accessibility info to your UI will make testing it easier¹. More specifically, the UI Automation framework looks for the accessibilityLabel property of your UIViews. If you've built any web applications, this is somewhat akin to sprinkling id attributes in your HTML markup so that you can find particular DOM nodes easily with JavaScript.

For views constructed in Interface Builder, you can set this property in the Inspector in the "View Identity" tab (the fourth one). Note that not every kind of view provides accessibility configuration in Interface Builder ². You need to enable Accessibility and you need to provide a Label value. You'll use these later in your tests to identify particular views.

If you aren't using IB or your view doesn't expose accessibility information graphically, you can still set it manually in your code. You can set the isAccessibilityElement and accessibilityLabel properties to get the same effect:

- (UITableViewCell *)tableView:(UITableView *)tableView cellForRowAtIndexPath:(NSIndexPath *)indexPath {
      static NSString *CellIdentifier = @"Cell";
      
      UITableViewCell *cell = [tableView dequeueReusableCellWithIdentifier:CellIdentifier];
      if (cell == nil) {
        cell = [[[UITableViewCell alloc] initWithStyle:UITableViewCellStyleDefault reuseIdentifier:CellIdentifier] autorelease];
      }
      
      // cell configuration elided
      
      cell.isAccessibilityElement = YES;
      cell.accessibilityLabel = @"user name"
      return cell
    }
    

Writing Your Tests

The next step is to write your first test in JavaScript³ in your editor of choice. This step is a bit like being dropped off in a field somewhere with no map, tools or supplies and being told you need to build a house. There's no built-in structure for your tests. When you execute a test script, Instruments will run it from beginning to end in linear fashion.

To get started you need to get a reference to the running application from which you can access all the other parts of the app. Put these two lines of boilerplate at the top of your test:

target = UIATarget.localTarget();
    application = target.frontMostApp();
    

The UIATarget is your primary portal into the application running on the device or simulator. It acts as a sort of proxy for the user of the application and is the object you interact with when you want to perform operations on the device such as fiddling with volume controls, shaking, or performing user gestures. The application object (a UIAApplication instance), gives you access to the top-level structure of your application for things like the navigation bar, tab bars, and the main window.

The UIAApplication class has a mainWindow() method that returns a UIAWindow instance. This class is an extension of UIAElement which nearly all of the other UI-related classes derive from. This class provides access to things like the parent view, child views, links, pickers, sliders, table views and nearly anything else you can imagine.

While Apple didn't provide much high-level information about UI Automation in the form of a Programming Guide, the UI Automation Reference Guide is worth bookmarking or keeping handy in PDF format. This is an essential reference that describes the JavaScript API in great detail. Check the Xcode documentation.

So let's look at a real example ⁴. Let's say we have an application connects to a variety of popular web sites. When the user launches the app for the first time, we want to detect that they don't have any accounts configured and prompt them to create one. Our screen might look something like this:

We want to verify that when we launch the application we see this screen. Once we've verified this, we want to create a twitter account, so we'll select the "twitter" table cell. Here's the test script so far:

test("Initial screen", function(target, app) {
      // check navigation bar
      navBar = mainWindow.navigationBar();
      assertEquals("Add Account", navBar.name());
      button = navBar.leftButton();
      assertEquals("Back", button.name());
    
      // check tables
      table = mainWindow.tableViews()[0];
      tableGroup = table.groups()[0];
      assertEquals("What type of account?", tableGroup.name());
    
      assertEquals(4, table.cells().length);
      ["facebook", "flickr", "github", "twitter"].each(function(i,e) {
        assertNotNull(table.cells().firstWithName(e));
      });
      
      // more to come...
    });
    

The test, assertNotNull and assertEquals functions are ones I wrote to add some structure and high-level validation to the testing process. I found the way the UI Automation suite does test case declaration to be really verbose and crude so I boiled test declaration into something simpler. First, the test() method takes a string title and a function to execute as the body:

function test(title, f, options) {
      if (options == null) {
        options = {
          logTree: true
        };
      }
      target = UIATarget.localTarget();
      application = target.frontMostApp();
      UIALogger.logStart(title);
      try {
        f(target, application);
        UIALogger.logPass(title);
      }
      catch (e) {
        UIALogger.logError(e);
        if (options.logTree) target.logElementTree();
        UIALogger.logFail(title);
      }
    };
    

This handles the boilerplate of getting the target and application references. It also provides some structure around UIALogger.logStart(), UIALogger.logPass() and UIALogger.logFail(). These three methods are how the UI Automation framework demarcates tests. However I found that using if checks and calling logFail() really muddied the tests, so I wrote some assertion methods that throw exceptions and the test structure automatically catches them and logs the test as a failure.

Here are the assertion methods:

function assertEquals(expected, received, message) {
      if (received != expected) {
        if (! message) message = "Expected " + expected + " but received " + received;
        throw message;
      }
    }
    
    function assertTrue(expression, message) {
      if (! expression) {
        if (! message) message = "Assertion failed";
        throw message;
      }
    }
    
    function assertFalse(expression, message) {
      assertTrue(! expression, message);
    }
    
    function assertNotNull(thingie, message) {
      if (thingie == null || thingie.toString() == "[object UIAElementNil]") {
        if (message == null) message = "Expected not null object";
        throw message;
      }
    }
    

Whew. Still with me? OK, back to our test. Next we want to fill out the text fields in the next view with our twitter credentials:

We extend the test code above with the next set of stimuli and assertions:

  // create a new account
      table.cells().firstWithName("twitter").tap();
      mainWindow = app.mainWindow();
      table = mainWindow.tableViews()[0];
    
      userName = table.cells().firstWithName("user name");
      assertNotNull(userName, "No username field found");
      userName.textFields()[0].setValue("mrfoobar");
    
      password = table.cells().firstWithName("password");
      assertNotNull(password, "No password field found");
      assertNotNull(password.secureTextFields()[0], "No text field found for password");
      password.secureTextFields()[0].setValue("sekret");
      
      finish = table.cells().firstWithName("finish");
      assertNotNull(finish, "No finish button found");
      finish.tap();
      
      // more to come...
    

The first line above finds the "twitter" cell by calling the firstWithName() method on the collection of cells in the table view. This table view was generated programmatically, so that "twitter" label came from setting the cell's isAccessibilityElement property to YES and its accessibilityLabel property to @"twitter". Next we touch that table cell via the tap() method to navigate forward in our application.

The next two stanzas validate that we have user name and password fields, and also fills them out using the setValue() method. Finally we look for the cell that contains the "Finish" button and tap it to finish creating our account.

The last bit of testing to do here is to validate that we did indeed land on our settings screen. So we add a few more assertions to our test:

  // validate settings screen
      finish.waitForInvalid();
      mainWindow = app.mainWindow();
      assertEquals("Settings", mainWindow.navigationBar().name());
    

Here I call the waitForInvalid() method on the finish button object. Without this, I found that the other code executed too quickly before the window transition completed. The waitForInvalid() will wait (up to a configurable timeout value) for that object to be invalidated. Our last assertion is that the title in the navigation bar is "Settings".

Running Your Tests

Now that we have a test, we need to run it in Instruments. When you launch Instruments, choose "Automation" from the iPhone templates.

In the details for the Automation instrument, choose your script via the "Choose Script…" drop-down. Next, you need to choose your target from the toolbar. Once you have these setup, you run the flow by (somewhat counter-intuitively) hitting the record button (or using ⌘+R). This will launch your application, wait a few seconds and then run your test. Note that even if your test completes, Instruments will keep running your application. To formally end the test toggle the red record button, or hit ⌘+R again.

Tests are listed in the detail view, with the test name in the "Log Messages" column. If your test passes, the "Log Type" value will be "Pass" in comforting green. If your test fails, the value is "Fail" in scary red. You can expand the test to see the details of what happened. Because of the way I wrote the test structure above, any test failures automatically log the view hierarchy for inspection.

Debugging Your Tests

When things go wrong with your tests you don't have a lot to look at. However there are a few things you can inspect to try to figure out what's broken. First, log the element tree (via UIATarget.logElementTree()) liberally and often. Although view hierarchy is a tree and it's presented in a tree control, it's "flattened-out" out for display purposes. However, the numeric prefix listed in each row indicates the level that node is at, so you can infer parent-child relationships:

In this case, the tree looks something like this:

• UIATarget
  • UIAApplication
    • UIAWindow
      • UIATableView
        • UIATableGroup
          • UIAStaticText

You don't always have to traverse the entire tree, node-by-node, to find what you're looking for. Take a look at the various methods on the UIAElement class and experiment a bit.

You can also log any message you choose using any of these log methods on the UIALogger class:

• logDebug()
• logMessage()
• logWarning()
• logError()

All of these just log messages of varying severity, but don't affect whether or not the test is marked as passing or a failure.

Building on UI Automation

While Apple may not have provided much support for automation testing beyond a JavaScript API and a reference doc, they (wisely) left the whole mechanism open to easy extension. You can import other JavaScript into your tests with the #import statement. I'm not 100% sure how smart that statement is with regard to paths, but I found that if you put supporting scripts in the same directory as the test script you're executing, Instruments will automatically pick it up.

So I took the extensions I mentioned earlier, plus a few more, and put them into a small, but growing, JavaScript library I call tuneup_js.

You can simply copy all of the JavaScript files into the same directory as your automation tests. For my project, I created a separate directory called "Automation Tests" and dumped everything in there. This is pretty crude and I'd like to improve it, but I need to spend some more time figuring out the nuances of how Instruments handles paths in #import statements.

Please check it out, file bugs, send patches and/or your feedback.

Stepping Back

These kind of tests are high-level, flow-oriented tests that exercise entire scenarios. As much as possible, these run against a "real" system though, at times, you may swap out external dependencies with mocked ones in order to control the test environment and execution.

There have been other third-party open-source projects that have addressed the lack of good automation testing. In particular iCuke and UISpec both have a lot going for them. If you like Cucumber, iCuke is nice for writing integration tests. It's pretty impressive what they've been able to accomplish on their own. Personally I prefer the Cucumber approach to integration tests. However, because iCuke uses a small embedded web server and XML to exercise and validate the UI, I found that I often ran into time-out issues and got frustrated wrestling with XPath expressions to validate my UI.

What iCuke has going for it, and what Instruments can't do, is the fact that you can automate the entire process. As far as I can tell, there isn't a good way to automate getting Instruments to run your automation tests. I hope this is something that either Apple addresses, or somebody figures out a clever hack for.

I like the way the Apple library is connected to the running application. It definitely feels "first-class" compared to iCuke's implementation ⁵. Let's hope that we can bring the two approaches together to have one fantastic automation tool.

One thing that none of these tools UI Automation can't really help you with is building reliable, repeatable test harnesses. One of the hardest parts of making good integration tests is setting up application state to test various modes of your application. This is one of the reasons why xUnit-style tests don't work well for automation. Too often people write fine-grained tests that have an implicit order, each test relying on the side-effects of the previous one. This makes your tests highly-coupled and extremely brittle.

In my application I have a couple of initial application states to test. None of these tools provides much help for that. For now, I've chosen to sprinkle a few #ifdefs around in my app delegate to setup the application in a known state. Then I use different build configurations to build separate binaries, each with a different known initial state. There's no doubt that it's a bit Rube Goldberg-esque, but it's a start.

There is a growing list of options available out there for writing automated integration tests for your iPhone apps. Apple's UI Automation tool is just one of them. It has it's pros and cons, but it's a big step forward in the right direction. Now it's up to us to figure out how to build upon them, develop extensions and idioms around them and, ultimately, build better apps.

Resources

As I stated earlier, there isn't much out there in the way of documentation, but here are a couple things to look at:

• UI Automation Reference Collection
• O'Reilly Answers Post on UI Automation

Footnotes

1. I don't know if this was a ploy to get developers to be more mindful of disabled users or not, but the net-result is a win for everyone. You get better test automation and disabled users get better apps.
2. I haven't spent the time to figure out which ones do and don't so perhaps there's some kind of rhyme and reason to this.
3. I have to admit that I was surprised to see JavaScript as the language of choice given Apple's support of the MacRuby project. JavaScript has had to weather a lot of bad press over the years. It's a language that has a lot of quirks but, once you separate it from the vagaries of various browser implementations, you'll find that it's a pretty flexible and expressive language.
4. This comes from a real app that I've been working on. You can't beat learning new things from a real project.
5. No disrespect meant for the iCuke team. Honestly, they've done an awesome job, I can't wait to see where they go next. Definitely keep your eye on this project.

This isn't the only tutorial out there on OCMock, but hopefully it's the clearest. I found the documentation on the site to be a bit, ahem, lacking. As a result I ended up figuring out how to use it by going through the source code¹.

Installation

First, you need to download the OCMock project. Before Xcode 3.2.3, you used to be able to simply add the OCMock.framework to the "Link Binary With Libraries" build phase of your unit tests. This appears to have stopped working, so you have two options: you can include the OCMock source Xcode project and declare a build dependency on its static library target, or you can build the static library once and put the following header files somewhere accessible to your Xcode build:

• NSNotificationCenter+OCMAdditions.h
• OCMArg.h
• OCMConstraint.h
• OCMock.h
• OCMockObject.h
• OCMockRecorder.h

I didn't want to rebuild the library over and over so I built the static library once and put the libOCMock.a and OCMock.h files in a directory in my project and added that directory to the LIBRARY_SEARCH_PATHS in the unit-test target configuration ².

Underlying Concepts

I'm assuming that you already know what mock objects are and how to use them. I'm not going to explain their value here, or the design changes you end up making to accommodate them. Mock objects are simply one of many tools you can employ in your testing. They have their pros and cons so let experience and intuition be your guide. Instead, I want to focus on how to use OCMock.

The general recipe for using mocks in unit-tests is:

1. Create the mock object
2. Specify the expected invocations and return values
3. Associate the mock object with the code under test
4. Execute the code under test
5. Validate that your assertions are correct

It's worth spending some time looking at the test cases that come with the OCMock source code. The coverage is quite good so it's a great demonstration of OCMock's capabilities.

Making Mocks

To create a mock object, use one of the factory methods available on the OCMockObject class: +mockForClass:, +mockForProtocol: or +partialMockForObject:. There are also two variants of this called "nice" mocks, which you can get by calling either +niceMockForClass: or +niceMockForProtocol:. The difference between a "nice" mock and regular (mean? stand-offish?) mock, is how they behave when they receive an unexpected method invocation. A "nice" mock will simply ignore the unexpected invocation, whereas a regular mock will raise a NSException.

If you're like me and you like to keep your builds free of compiler warnings, declare the returned object to be of type id to shut the compiler up.

id myMock = [OCMockObject mockForClass:[MyClass class]];
    

The +partialMockForObject: method allows you to turn an existing object into a mock. This can be useful in cases where a collaborating object has several of its methods invoked, but you only want to override one or two. The obvious dangers here are that, 1) your collaborating class probably violates the Single-Responsibility Principle and 2) your tests now indirectly rely on the subtleties of which methods you mocked and which you didn't. Use at your own peril.

Setting Expectations

The object returned by any of these factory methods allows you to both setup expectations, invoke methods and verify its configuration. OCMock does a clever little trick to distinguish between invocation configuration and handling method invocations. Calling either the -expect or -stub method will return an object that you can use to setup your expectations. The cool thing about this object is that you setup your expectations by (more or less) invoking the methods on it that you want your class under test to invoke.

Let's imagine that we have a simple stock-portfolio management application. Our portfolio model is encapsulated by the AVStockPortfolio class. An instance of that class is given an object that implements the AVQuoteService protocol, which is our gateway to a real-live stock quote service. When we test the AVStockPortfolio class, we don't want to be encumbered with a "production" implementation of AVQuoteService. Instead we want to craft a stand-in object to exercise and validate the portfolio class.

Our first test is to see that the portfolio class initiates a connection with the quote service in its init method:

- (void)testInit {
      id mockService = [OCMockObject mockForProtocol:@protocol(AVQuoteService)];
      [[mockService expect] initiateConnection];
      
      AVStockPortfolio *portfolio = [[AVStockPortfolio alloc] initWithService:mockService];
      
      [mockService verify];
    }
    

When the -expect method is called, it returns a "trampoline" object that captures additional method calls dynamically. Any methods invoked on the service that I haven't explicitly configured will raise an exception. I can also check that the methods I've configured were all invoked by calling the mock object's -verify method.

You can also configure your mock object to act as a more forgiving "stub" for particular methods by using -stub instead of -expect ³. Admittedly, the ability to turn a so-called "mock" object into a "stub" for particular method invocations muddies the nomenclature a bit, but it can be handy to have this ability at times.

Return Values

If methods on your mocks need to return values, you have a variety of methods to call on the object returned by -expect or -stub. Any method we call on this object that isn't one of the built-in OCMock methods, is captured dynamically as an expected method invocation.

Since each of these methods returns the same expectation object, you can chain these method calls, where it makes sense. For example to post a notification and return a specified value we could do this:

NSNotification *notfication = [NSNotification notificationWithName:@"foo" object:nil];
    [[[mock expect] andPost:notfication] andReturn:@"FOOBAR"] doSomethingMagical];
    

The -andDo: option is one of my favorites since it uses blocks and blocks are the most awesome addition to Objective-C in a long time. You can do a lot of validation locally within a block which keeps the code very succinct and clean:

- (void)testSellSharesInStock {
      id quoteService = [[OCMockObject] mockForProtocol:@protocol(AVQuoteService)];
      [[[quoteService expect] andDo:^(NSInvocation *invocation) {
        // validate arguments, set return value on the invocation object
      }] priceForStock:@"AAPL"];
      
      AVStockPortfolio *portfolio = [[AVStockPortfolio alloc] initWithService:quoteService];
      [portfolio sellShares:100 inStock:@"AAPL"];
      
      // other validations and assertions
      
      [quoteService verify];
    }
    

Arguments & Return Values

Basic method expectations are useful, but we also want to validate the arguments given to methods. Back to our stock portfolio, imagine that when we buy a stock, we know that part of that process is to ask for the current price of a stock from a stock service, so we want to assert that when we sell a particular stock, our portfolio object queries the quote service for the latest price for the same stock symbol:

- (void)testBuyShares {
      id mockService = [OCMockObject mockForProtocol:@protocol(AVQuoteService)];
      [[mockService expect] andReturn:[NSNumber numberWithFloat:214.57]] priceQuoteForSymbol:@"AAPL"];
      
      AVPortfolio *portfolio = [[AVPortfolio alloc] initWithQuoteService:mockService];
      [portfolio buyShares:100 inStock:@"AAPL"];
      
      // other validation and assertions
    
      [mockService verify];
    }
    

In this case, we're declaring the expected argument to be the string literal @"AAPL". We've also configured the method with a return value, in this case a NSNumber instance with the value 214.57. However, OCMock can do more. You can use any of the following OCMArg class methods in place of a real argument when setting up your method expectations:

OCMock also provides a few handy macros for argument matching:

But Wait…There's More!

One nice little feature of OCMock is its handy support for testing notifications. To create a mock for notifications, call the +observerMock factory method on the OCMockObject class. This will return a mock that works differently from the mocks/stubs described earlier. Instead of setting up method invocation expectations, you can set up notification expectations which are useful when you want to assert that your class under test posts notifications. Let's say that we want our aforementioned portfolio class to post notifications whenever stocks are bought or sold:

- (void)testSellSharesInStock {
      id mock = [OCMockObject observerMock];
      // OCMock adds a custom methods to NSNotificationCenter via a category
      [[NSNotificationCenter defaultCenter] addMockObserver:mock
                                                       name:AVStockSoldNotification
                                                     object:nil];
                                                   
      [[mock expect] notificationWithName:AVStockSoldNotification object:[OCMArg any]];
    
      AVPortfolio *portfolio = [self createPortfolio]; // made-up factory method
      [portfolio sellShares:100 inStock:@"AAPL"];
    
      [mock verify];
    }
    

If the AVStockSoldNotification isn't posted by the time we call verify, the mock will raise an exception. Note that you need to add the mock as an observer, which you can do by the handy -addMockObserver:name:object: category method on NSNotificationCenter provided by OCMock.

Execution & Validation

Any method invoked on your mock object will be checked against the configured invocations. Any unexpected invocations will result in an exception being thrown (unless it's a stub or the mock is in "nice" mode). Also, any configured invocations that did not happen will trigger an exception when the mock's -verify method is called. As a matter of habit, you want to call this at the end of your test.

Conclusion

Using mocks, stubs and test-doubles in your testing is a handy way to keep your tests lean and focused on one class at a time. This is especially helpful when you have dependent classes that require a lot of setup that you would rather avoid in your test-harness (e.g. Core Data).

The downside to testing with mocks and stubs is that you lose some test coverage for the objects you're replacing with test-doubles. This approach probably works best when your code is interacting with objects from other libraries that you don't want to deal with in your tests.

Footnotes

1. I know it seems like a drag, but it's hard to overestimate the value of reading the original source code for any tool that you use. You'll be amazed at how much more of an intimate understanding it gives you of the tool and the rationale behind its design.
2. I forked the OCMock codebase and have started to make a few tweaks to it. Feel free to take a look at it on GitHub.
3. If you aren't familiar with mocks vs. stubs, Martin Fowler has written the canonical text on the differences.

I just returned from the 2010 edition of Apple's WWDC. The week before I took a week off from my iPhone and lived with an Android Nexus One. Both my personal experience with Android and the following week's announcements left a strong impression that, in the so-called "smartphone" space, we are really seeing the success of two very different mind-sets, cultures and philosophies. I'm not interested in a winner-takes-all kind of war. Instead I wanted to look at the differences between these two companies in the context of them co-existing and each having a happy, satisfied base of customers.

I was going to write about how capable the Nexus One seems to me, but how amateurish the UI appears. At times, the look and feel of the Android UI looks like its designers simply couldn't figure out how to fill the space with something interesting.

I was going to write about how the ubiquity of the four hardware buttons are both a blessing and curse. I understand that the designers were trying to provide a set of near-universal options in a consistent place. Unfortunately it was too easy to accidentally hit the "home" key while typing on the soft keyboard.

I was going to write about the moment I realized just how different the design philosophies between Android and iOS (nee iPhoneOS) are when I wanted Instapaper integration with a good Twitter client. I needed to find an Instapaper app, not a Twitter app with Instapaper integration. I simply didn't grok the universal plugin architecture of Android at first. In hindsight it makes perfect sense—if you're a software developer or tech-geek. It seems insane to require the user to have a mental model that's so intimate with Android's implementation details.

I was going to write about how delighted I was with Android's GMail integration and how much I wish something similar existed on the iPhone. I love GMail. I couldn't care less about other mail configurations (POP or IMAP). Maybe the built-in regular mail app is as clunky as Apple's, but I didn't use it. Kudos to Google for such a seamless integration with not only Gmail, but the calendar and contacts too.

I was going to write about all these little bits of evidence that showed just how different the philosophies are between these two companies and these two platforms. I was going to write about how it's a bit odd to put them in competition with each other because, in a sense, they are really trying to build two different things: Google wants to build the most bad-ass feature-rich, piece of mobile technology there ever was. Apple wants to build a mobile experience.

I was going to write about all of these things, but after attending the keynote at WWDC last week, I realized that there was only one thing that I needed to point to that most effectively highlights the differences between the two:

At the end of this video, you had a room full of five-thousand geeks drying their eyes. You can call the video cheesy or manipulative. You can call it a piece of marketing fluff. But that video was not about protocols, compatibility, specifications or any of the myriad technical details we debate on a daily basis. That video was about telling human stories that we can all relate to.

I haven't yet been to a Google I/O event. I'd really like to attend one. But right, wrong or indifferent, you would never see something like this from Google. I can't think of anything that does a better job of summing up the completely different views that Google and Apple have of the world.

Given the current state of affairs, it's not hard to see why there's a bias against unit-testing in the Cocoa community. ¹ Unit-testing tools, support and idioms within Cocoa are nowhere close to where they are in, say, the Java, .Net, Ruby or Python communities. Compared to those environments, unit-testing in Cocoa is just flat out difficult. I can easily understand why a developer would conclude that the cost and effort of TDD outweighs the benefits.

The Dismal Science

Think of every "best practice" you've ever been exposed to in software. They all come down to cost-benefit tradeoffs in the end. Up-front requirements gathering? It's an attempt to manage change and indemnify certain parties when things go awry. Iterative development? It's merely a short-term review process to evaluate the cost of future development against opportunities in the future. Automated testing? Computers (generally) work more cheaply than humans, so invest in automating repeated tasks so that, over the long-haul, more work is accomplished by cheaper workers. Hell, even the oft-cited goals of code-reuse in object-oriented programming are economic ones. The list goes on and on and on.

But, even as a long-time practitioner of test-driven development, I don't view it as axiomatic. In this big, bad world of ours there have to be cases where the economics simply don't add up, and it simply isn't cost effective to build stuff using TDD.² Returning to Cocoa, I can see how developers come to the conclusion that TDD in Cocoa simply isn't worth the price of admission.

So what makes TDD in Cocoa so expensive? Would things look different if we could change the balance sheet?

Bear-Skins & Stone Knives

Back in the day, OCUnit was the xUnit toolkit of choice for Cocoa programming. In 2006 Apple put OCUnit right into Xcode and TDD received its first official blessing. Now I don't have a beef in particular with OCUnit—it's essentially a faithful implementation of xUnit patterns for Objective-C. What is surprising is how weakly it's integrated with Xcode.

To unit-test, you have to create an entirely new target to execute your unit-tests within. However, you don't run it like a normal executable. It's baked into the build process for that target so that testing errors show up just like build errors. I'll admit that's kind of cute and at least makes an attempt to integrate TDD into the development process. But keeping unit-tests as a separate target means more drop-down flipping in Xcode to go between unit-testing and running my application. Oh and the bloody drop-downs in Xcode…don't even get me started…

The other issue is debugging. If your unit-tests are failing, god help you if you want to find out why. It takes a lot of environment-variable hijinks to be able to actually debug your unit-tests —something that is significantly easier on every other platform I've ever worked on. All of this leaves me with the distinctly uneasy feeling that Apple and the Cocoa community at-large are merely paying lip-service to TDD.

Compare this to how JUnit is integrated into your standard Java IDE or awesome Ruby testing tools like autotest. These tools are so much more immediate and easier to reach for. Xcode looks like it came from the era of the horse and buggy. These kind of tools and this kind of support needs to be a part of the development environment in a more natural way. Right now it's just a primitive, bolted-on afterthought. It's a wonder anyone has the patience to use it.

A New Mentality

Another challenge in Cocoa is figuring out where TDD fits in such a framework-driven environment. To Apple's credit, the frameworks that Cocoa provides do a pretty good job of "making the simple easy and the difficult possible". However because Cocoa is so prescriptive, it can be difficult for developers to stand back and figure out what to test. It's just so easy to just let the idioms fly off the fingers, that testing them seems like a silly exercise.

I think a common conclusion for the the would-be TDD'er is that they often find their tests essentially repeating the implementation. These are most expensive tests to write and maintain. Not only do you end up duplicating the code (thus adding coupling and brittleness), but they also take up a lot of time to write and can be pretty error-prone.

There has been a lot of thought about similar problems in other communities, so why not steal these ideas and apply them to Cocoa development? Surely strategies like inversion of control and mocks would help make TDD in Cocoa an economic possibility. ³

The Cocoa community simply hasn't evolved a good set of testing practices the way others have. Given the benefits I've seen in other environments, it's hard for me to believe that Cocoa and Objective-C are exceptional in this regard. I think that there are ways to do it, we just haven't discovered them yet. I recall from my Java and Ruby days that testing idioms and practices evolved a lot before we got somewhere reasonable. Simply put, the Cocoa world's collective testing skills and knowledge lag severely behind a lot of other languages.

Facing this requires the Cocoa community to face its own isolationist and exceptionalist attitudes. There's nothing so special about Cocoa and Objective-C that conceptually invalidates the effectiveness of TDD. ⁴ Cocoa folks need to look outside of their walled garden to see what others have done. This is not something I've seen much, if any of, in the Cocoa community. Frankly, the dominant attitude seems to be one of snobbery and elitism. It's an unfortunate attitude that holds us all back.

Footnotes

1. Normally it wouldn't be fair to link to a five-year old post and call it "representative" of a community's attitude, but I think for an insular group like Cocoa-nerds, this is totally reasonable.

2. I'm not saying that I, personally, have encountered such a thing, but I don't think it's unreasonable to assume that the possibility exists.
3. There is, of course, a pathological extreme to this line of thinking. That extreme is called Spring.
4. OK, I'll admit that iPhone development is a little different because of having to deal with the device vs. the simulator. Getting tests running on the device is a non-trivial exercise. But I'm not convinced that unit-tests have to run on the device. Yes, there are differences between the real and simulated environments, but those differences should be accounted for in integration tests—a topic I'm not going to address here.

But what if you're integrating with a web service? What's the best way to convey the user's current settings to an external party? I've run into this exact situation a couple of times and wanted to share the approach I like to take.

The Server-Side

The first thing you need to determine is how you'll express the user's current locale settings. If you own the servers-side, you can implement this in whatever way you see fit. Personally, I like to take advantage of as much of the HTTP specification as I can. Rather than overloading every request with a request parameter, I like to use the Accept-Language header. The specification says that the format of the Accept-Language header allows multiple locales with "quality values" optionally given to each one to express order of preference. For example, a valid Accept-Language header might look like this:

Accept-Language: da, en-gb;q=0.8, en;q=0.7
    

In this example, Danish (with no specific region) is preferred first, followed by English (in the Great Britain region), followed by English in any region. The language and region are delicately intertwined in a bunch of different ways. Depending on the combination, the region may affect how words are spelled (for example "armor" in American English vs. "armour" in British English). Not all locales and regions work this way so there are plenty of corner-cases.

Ruby on Rails is my server-side stack of choice these days. Like Cocoa, Rails is pretty good at making it easy to localize and internationalize your application. However, despite Rails claim to being "opinionated", it's surprisingly indecisive about the best way to express language and region preferences. The Rails i18n Guide suggests that you can get the locale as a query parameter (blech), via the domain name (if you've set your application and client up this way), via URL parameters (i.e. as a path segment within a larger URI structure), via built-in user settings, via GeoIP or using the Accept-Language header. As I stated before, I like the latter solution the most. However parsing the header, dealing with the quality scores and then figuring out the best one is something that has already been solved. For that, I like the http_accept_language gem.

In the ApplicationController, I setup a before_filter that uses the gem to parse the Accept-Language header. Since I may not have localized the web server to match a given request, I use the compatible_language_from method to figure out what, if any, language match there is.

before_filter :check_language
    
    def check_language
      params[:locale] = if params[:locale].nil?
        request.compatible_language_from(MyApp::AVAILABLE_LANGUAGES)
      else
        params[:locale]
      end
      I18n.locale = params[:locale]
    end
    

MyApp::AVAILABLE_LANGUAGES is constant array defined in a Rails initializer (in config/intializers) that figures out all of the possible locales I have configured:

module MyApp
      AVAILABLE_LANGUAGES = Dir[RAILS_ROOT + "/config/locales/*"].map do |f|
        File.basename(f).split('.').first
      end
    end
    

Despite my aversion to using request parameters, it can be handy for ad-hoc testing to be able to tack a query parameter on to change the locale. Also, once you've determined the locale, you need to notify the I18n framework as I've done in the last line of the check_language method.

The Client-Side

OK, great. Now we've figure out how to vary the locale on the web-side. How do we extract this information from the device and ship it off in our HTTP headers?

Here you have two tasks: the first is to determine what the value of the Accept-Language header should be and second, to actually make sure you send those values. Since there are several ways of making HTTP requests in iPhoneOS I'm not going to go into the details. Whether you're using NSURLRequest, ASIHTTPRequest, HTTPRiot or your own homegrown thing, presumably you have some reasonable way of setting the Accept-Language header in a consistent way.

To figure out what locale your user is in, you need to consult the NSLocale class. This provides several class methods that yield a NSLocale instance that reflects the user's current settings. If you look at the docs, there are three class-methods that return a NSLocale instance: +systemLocale, +currentLocale and +autoupdatingCurrentLocale. The differences between the three are not apparently obvious and, I think, reflect NSLocale's original desktop heritage.

The +systemLocale method will return a default NSLocale instance when one cannot otherwise be determined. In my experience, I've never seen this return anything but a blank and useless NSLocale instance. Don't bother using it.

The +currentLocale method returns a NSLocale instance that reflects the user's current setting. However there is a bit of language in the docs that I initially found confusing:

Settings you get from this locale do not change as System Preferences are changed so that your operations are consistent. Typically you perform some operations on the returned object and then allow it to be disposed of. Moreover, since the returned object may be cached, you do not need to hold on to it indefinitely.

Huh? To understand what's going on here, we need to take a look at the docs for the +autoupdatingCurrentLocale method:

Settings you get from this locale do change as the user’s settings change (contrast with currentLocale).

Remember, a lot of CocoaTouch came from the desktop Cocoa environment. On the desktop you can open the System Preferences and change your region and language settings anytime while applications are running. However, until iPhone OS 4.0 is released, this is not something you can do on any iPhoneOS device. So the net effect, and this matches my own observations, is that on iPhone OS 3.x +currentLocale and +autoupdatingCurrentLocale are essentially the same. But, if you want to be prepared for the future, go ahead and use the +autoupdatingCurrentLocale method. Once you have a NSLocale instance, you call the -localeIdentifier method on it to get a NSString like en_US or fr_FR.

So if you're like me, you want to test that this actually works. So, naturally, you would figure out how to modify the settings on your phone to yield different responses. What you'll soon discover is that the current locale settings are derived from two separate and independent settings and that various combinations of the two can produce unexpected results.

Start by launching the "Settings" application. Select General ⇢ International. You'll get a screen where you can pick your language, your keyboards and your region format. What may surprise you is that changing the language does not affect the current locale. I'm in the United States so my region format is set to "United States". If I change my language to French, but leave the region format setting alone, all localization within the system and applications will use the French language, but my locale remains en_US.

OK, so NSLocale doesn't quite work as expected, but where does the system store the current language? It turns out the NSLocale provides yet another class method, named +preferredLanguages, that returns an array of language codes as strings. With the settings described above, the first entry in that array is "fr".

If you change your settings where you leave the language set to English, but change the region format to French (in the "France" region), the current locale will now be set to fr_FR and the first entry in the preferredLanguages array will be "en" for English.

What It All Means

Out of the box, Rails doesn't provide full-blown region-specific localization. You can either hack something or use one of many i18n plugins or Globalize2. So if you try to send a language-region combination to Rails (such as en_US), chances are good that you won't have the right localization setup and you'll fall-back to your default locale.

You have two choices: you can make the client send something that the server understands, or teach the server to respect more fine-grained localization values. If you want to go the first route, I would suggest simply querying the +preferredLanguages array and using the first entry as the value for your Accept-Language header. However if your server-side localization can handle it, use the +currentLocale instead so that you can handle regional differences and spelling much better.

The iPad has generated a lot of criticism, scrutiny and analysis in recent months. The debate about the ills and assets of the platform have, and will continue to be, debated endlessly. I think one thing is pretty clear though— whether you like Apple or not, whether you have an iPad or not, whether you think it's the most awesome-est thing ever or not, it is revolutionary device. I don't mean in terms of technical execution (yes the A6 chip is a marvel, the power-consumption is amazing, blah blah blah). What I mean is that somebody with Apple's clout finally stood up and threw away the last three decades of user-interface paradigm and they're laughing all the way to the bank doing it.

So, if you jettison The Way We've Always Done Things ™, what do you replace it with? More importantly, which things from The Old Way are no longer applicable to The New Way? This is a big topic that will take some time to settle itself. We've had this CPU/mouse/keyboard setup for nearly thirty years and we're still learning how to make it work for humans. I don't expect us to settle the micro-debates of touch-interface design for quite a while, but I do want drill down to something pretty specific.

Take a moment and ponder what the mouse really is in a graphic user interface. It's a proxy. It's a loosely-connected device where movement and gestures in one space are translated to movement and gestures in another virtual space. To operate a mouse well, you have to perform constant mental transform operations from the physical world to the world you are trying to manipulate on the screen. The mouse is, effectively, a rather thick layer between you and what you're trying to do.

Exploiting the Gap

As we got better and better with the mouse ¹ software started to exploit this gap between you and your machine. The first place this showed up was in context menus. Once we already had to make a mental stop on the train ride from our brain to the machine, why not make the station more useful to folks? Hell, let's put in some vending machines, and perhaps some couches. We could even install Wi-Fi and people could get even more productive. Pile it on, pile it on.

Taken to an extreme, the mouse becomes the primary means by which we interact with our machines. If you don't believe me, take a look at any professional CAD workstation or that god-awful OpenOffice mouse.

However these are extremes. There isn't necessarily anything wrong with things like extra mice buttons or context menus. But, in a world of touch, where the mental and physical distance between our intentions and execution is much smaller, that thick layer/opportunity is gone. Context menus? How the hell do I "right-touch" the screen? Mouse-overs? Current touch interfaces don't have any proximity sensors ²—either you touch something or you don't. Yet these two very simple interaction models have become a crucial part of the UI vocabulary we have all acquired over the years.

Think about a mouse-over. What is it there for? Mouse-overs offer two things: a preview mechanism allowing you to learn more about something without having to commit to it, and as a backup when a pictorial icon's meaning isn't clear enough. We don't have these on the iPad ³. So as an application designer and builder, how do we give people some notion of what this thing can do? What kind of affordance can we offer?

Spelunking

As you start using and learning an application, you generally have two questions:

• What can this thing do?
• How can I get this to do ______?

These are asked from two opposing angles, but both are about discovering the capabilities of the software. When a user doesn't know what your application can do, how can you build it in a way that they can discover it? An equally important question is how can they explore your application without having to commit to any action, particularly a destructive one?

Let's look at one of my favorite iPad apps in particular, Net News Wire. A nice feature of the application is the ability to progress through all of your unread items with a single tap. When I first got the app, I knew that there had to be a way to do this, but I couldn't find it. There was a mysterious-looking button in the corner of the screen, but I wasn't sure what it was for. I mean, it might do anything. How was I to know what would happen? It wasn't until I saw a presentation by Brent Simmons and Brad Ellis on the design of NNW that I actually found out where that button was. You can't really figure out what it does unless you poke it. If you poke that button, there isn't a corresponding "undo" button for that action.

That's not a dig against Brent and Brad, I think they did a wonderful job with NNW. But it does highlight how difficult it can be to convey such a feature to the user. I think it may be one of the biggest challenges on the platform. How do we let the user know what our application can do?

Let's be honest here, if you're relying on documentation to teach the user the basics, you've already lost the battle. Documentation is fine for really deep, detailed information (see OmniGraffle on the iPad for an example). But requiring a user to read the Owner's Manual before they can even use your software went out about the same time as floppy disks. On the iPad an up-front documentation requirement is just laughable.

Little Gets Big

One final thought is that although the iPhone and iPad obviously share the same DNA, the difference in size makes this acutely problematic on the iPad. On the iPhone you simply can't cram that much stuff into an application. Read the iPhone Human Interface Guidelines and you'll quickly get the message that on the iPhone, less is more. By and large, the best iPhone apps simply don't have that many features integrated into them. This gives application designers more freedom to make clear the intent of the features that are implemented.

On the iPad, it's a different story. There's so much more real-estate. Running an iPhone application on the iPad is so laughably awkward that it starkly highlights just how different the platforms really are. So now, as iPad application builders, we have more space than ever. While we're freed from the space constraints of the iPhone, we now have a challenge (and responsibility) to use it effectively. It's not hard to imagine some developers embracing these new green fields to produce some truly awful interfaces. Please, don't be one of them.

Tangents

1. Make no mistake here, we've had to train ourselves to use the mouse. Just watch your grandparents struggle with a mouse and you'll realize how un-intuitive the device really is.

2. I'm not even sure if they did that it would be such a good idea. Requiring that level of fine-finger dexterity immediately makes such a device exclusive to the young and facile.

3. OK, there is a common trick of touching and holding a hyperlink long enough to get a menu of options.

Size and Geometry

There is a axiom in military science that says no plan survives contact with the enemy. In a similar vein, no slide layout survives contact with an actual display screen. As we craft presentations it's easy to get so locked into looking at our screens that we often forget about the context of the room in which it will be given. There are two important things to consider about the room: how far back the audience stretches from the screen and the angle of the audience. In a room with a "deep" audience and shallow angle, anywhere from the bottom 20% - 30% of the screen will only be visible to the front row. Some professional presenters make it a matter of habit to simply avoid the bottom quarter of the screen. Think of this as a creative constraint to embrace and help you focus your message.

The dimensions of the audience also affect the size of the content in your slides —specifically the font size you choose. Somehow font sizes that look like billboards on your laptop are diminished to footnotes once they are projected. In practice, it's pretty hard to make text too big. If you have a problem fitting what you want to say on the slide, perhaps you need to reconsider the phrasing.

Aside from direct quotations, I find that it's rarely useful to have full paragraphs of text in your slides. You want your audience to be focused on you, not your slides. Think of the slides as simply the supporting cast to what you're telling the audience verbally. How can you craft your slides to complement what you're trying to convey?

Live Demos

Slides can only take us so far, especially for technical presentations. Sometimes we need to turn away from the "what", and explain the "how" of something. We want to demonstrate how something works. For coders, this often means a live coding or tool demonstration. If you're trying to convince your audience that technology XYZ is the greatest thing since sliced bread and will save them all sorts of time and headaches, you want to be able to show it, not just tell it.

Fantastic. Good for you. Here's the problem: live coding demos are a recipe for disaster. I can't think of a better scenario that demonstrates Murphy's law than trying to execute a live coding demo. You may not have an internet connection (oops, there goes the ubiquitous twitter client demo). You may have installed some beta software on the plane ride out and, unknowingly, destroyed your development environment. The list of things that could go wrong is unbounded.

The other problem with live demos is the switching in and out of the presentation software you're using. Not only is it visually jarring for the audience, but there is invariably some down-time as you switch displays and try to figure out which windows are where. Don't give your audience the opportunity to tune you out during these noisy transitions.

So, just like you wouldn't create your slides on-the-fly as you present, you shouldn't demonstrate live without a net. Instead, record a video of it and embed it in your presentation. With a video you get a chance to edit out all of the hiccups and noise of the demonstration process. Mistyped something? Got a compilation error? Tools are running really slow for some reason? Great, just edit those frames out.

Leaving these blemishes in only gives the third-grader part of your audience's brain time to sneak in and distract them. You want to keep your audience engaged with you every step of the way. Introducing these interruptions into the continuity of your presentation breaks the audience's concentration and your flow.

Once you get your video captured and edited, practice speaking over it. Really make sure that what you say matches well with the video. Get the timing down and stay focused on what you're trying to show. Is there something you want to say that you isn't in the video? Get back in there and record the right video. Don't half-ass it. When you put the time in to make a good supporting video, you keep your audience longer.

Keynote Specifics

If you're using Apple's Keynote, here are a couple of tips. Once you've captured and edited your video, simply drag the video object onto a new slide. Open the inspector palette and select the last tab with the Quicktime logo on it. Select the checkbox labeled "Start movie on click". This way your video won't start running before you're ready to talk about it.

Also, with a stock Apple remote, you can pause and resume your video as needed. This is especially helpful when you have a multi-step process to explain that you want to cover in bite-sized chunks. This means that you should consider pause-points in your capture and editing process. Practicing your speech along with the video will help you figure out where the natural breakpoints are. Alternatively, you could split the process up into multiple videos on different slides.

Eschew Style

Customization and personal style are well and good. On your machine you should feel free to tweak every little setting to your heart's content. But, please, don't force these styles on your audience. Ditch the alpha blend on your terminal. Turn off your crazy four-line shell prompt. Pick a color theme in your editor that is easy to read in a large font. Pick a display font that is appropriate for large projection, not just what you prefer.

Make those fonts BIG. Just like your regular speaking voice needs to be amplified to carry to the back of the room, so does the visual volume of your content. When recording your coding demos, pick unusually large font sizes and take up all of the screen. Also, when capturing your video, try to keep the capture frame focused on the essential parts of the demo. For example, if you're only editing text, don't clutter the video capture up with a toolbar full of things you won't use. Remember, keep it focused! Don't give your audience a chance to get distracted by clutter.

Keep It Focused

When presenting a technical topic, it can be frustrating having to jam a large amount of content into a small space. There's just so much to share! How can I possibly discuss Core Animation in forty-five minutes? The answer is, you don't. What you can do is give your audience a taste of the topic you're trying to present. Give them enough information for them to decide if they want to pursue more on their own. Give them enough terminology and background concepts to help them continue the journey.

Presentations are not a good vehicle for in-depth training. Books or workshops are much more appropriate for real hands-on learning. Presentations are good at giving people a mental roadmap to get started. Get them excited by your topic, but don't exhaust them with it.

Just look at that. Do you think that fits on anything but a large display? I can't even see most of these icons on my built-in MacBook display. Only humble little Finder, with its spartan set of menus, yields enough real estate to display all of these items.

Sadly, I had gotten into the habit of knowing which applications hid status bar items from me and would automatically switch to Finder just to reveal them. It was the classic mistake we all make of giving in to bad design instead of addressing the problem.

So I waged a small holy-war this morning to kick out all but the essential applications from my status bar. I feel a little better that I've reclaimed my territory, but can't help but feel frustrated that I'll have to do it again soon. Why? Because, somehow, application developers got it in their heads that they all have to have menu bar items.

I want this to stop. Right now. Please. Do it for the children.

I can't help but think that this reflects a dangerous attitude underlying a lot of applications. Stop for a moment and think of the reasons why an application provides a menu bar item. Menu bar items exist for applications running in the background. These come in two flavors: applications with no other user interface (like a window or a preference pane), and ones that do.I don't have much of a beef for the ones that can only provide a menu bar item as their sole interaction mechanism. It's the applications that do have a main UI that I want to give a stern talking-to.

In a windowing system users get to choose what they focus on (save for a few cases with alerts and modal dialogs and such). Applications are supposed to do the bidding of the user. When I switch away from application XYZ, it means I have something more important I need to do. Having that application pop back in my face drives me nuts. It drives everyone nuts. I think most folks would agree that this is a Bad Experience.

But some applications want to keep your attention. So they compromise by creating a little persistent bit of goo and sticking it in the menu bar. Because these apps are so fabulous, how could you not want them around all the time?

The best menu bar items are ones that provide the essential capabilities of the application and give you a quick way to expand to the full application. I run iStat menus because it gives me a lot of information in the small space it takes up. Why is my machine so slow? Ah, the CPUs are pegged. Why are they pegged? Well, just click the menu item and see that application XYZ is taking up 90% of my cycles. It's a great example progressive disclosure.

Some applications like iShowU or Little Snapper provide a menu bar item because they need to get out of your way. You don't want to screen capture the screen capture tool itself, right? For these kinds of apps that provide functionality that spans multiple applications, I think this is okay.

What I can't stand are applications that provide a menu bar item, then keep it there after I've quit the application. Evernote is a good example of this. You have to really go out of your way to make Evernote not do this. Worse yet, its default behavior is to install itself as a startup item so that it's sitting there in your menu bar until you take time out of your busy day to turn it off. Boo. Hiss.

I like Evernote quite a bit, but I get the impression its creators think it's so special that it deserves to be in your face, all the time. It's like that annoying guy you got stuck talking to at the company party that you can't rid yourself of. Go away.

The worst case of this is VMWare. I can't, for the life of me, figure out how to disable VMWare's menu bar item. For folks that need to run a lot of non-Mac apps I can see the advantage. The Unity stuff in VMWare is pretty freaking cool. But if you're not one of those people, it's pretty rude to have this whole other application-launcher sitting there, taking up space, that you only occasionally use. Shame on you VMWare. I'm happy to pay for a license because I think it's a great product, but that doesn't mean I want to see your goddamned menu item in my status bar all the time.

So if you're building a Mac application and considering a menu bar item, please consider the following advice. If you're building a background utility with no other UI, think hard about how much the user needs to interact with your application. Is it something they need to fiddle with more than once a session? If not, try to make a Preference Pane and leave it at that. If it's something the user is going to use a lot, or displays some kind of realtime information (à la iStat menus) then adding to the menu bar is okay.

If you're building an application with a main UI component, think long and hard about the necessity of menu bar item. If it can integrate with other applications (like Little Snapper) there's a reasonable case to be made for providing a menu bar item. Even then, you should consider providing a Services menu item and/or a global keyboard shortcut. This is an especially good idea if your application only provides one or two actions that users can take. It's hard to justify a two-item menu bar item, so sometimes you see applications put other non-essential items in there like Preferences or an About dialog. Don't do this.

For these sorts of applications, the menu bar item should always be optional. The app should still be able to do what it needs to without having that menu bar item enabled. Also, leave it off by default and make it easy for the user to turn it on. Make it an opt-in feature rather than an opt-out feature. As much as I love Tweetie, the opt-out nature of its menu bar item is the kind of stuff that drives me nuts. The option is enabled (or not disabled?) and the language is confusing. Worse yet, it doesn't take effect until you restart the application.

Finally, unless you have some kind of special papal dispensation, never, ever, ever keep your menu bar item running when the application is gone. OK, technically if there's a menu bar item the application is still running, but that's a distinction that user's don't care about (nor should they have to).

There's been a long, ongoing debate about the necessity of optimizing the performance of applications. As horsepower has increased, programmers have been afforded some laziness in optimizing their applications. I only have a problem with that in extreme cases because hardware power has increased at such an impressive rate. However, our displays aren't growing at the same rate. We need to treat that screen real estate as precious. The menu bar is a great place to put global, cross-application functionality, but there's only so much space for all of these applications. Think long and hard about the necessity of taking up that space and always put the power in the users' hands to choose.

Since her death, I've been tossed into the deep-end of estate-management as I try to clean up the remnants of her financial and legal life. It's amazing how much a person leaves behind. There are all sorts of things to clean up like calling her dentist and canceling a pending appointment, or telling the local pharmacy that she won't ever be coming in again to pick up her prescription.

In the last month, I've realized I'm not the same person I used to be. It sounds trite, but this event has given me a serious perspective adjustment. Things that used to seem scary to me now seem insignificant. Things I always wanted to do but put off for some day in the future, get addressed now. Life is short. You never know what can happen.

By nature, my wife and I inveterate planners. My god, I started an IRA when I was 21. Who the hell does that? But now it's time to think more about the present. We haven't completely abandoned our careful planning, but we've stopped putting off things we want to do.

One of those things has been a growing desire to be my own boss. For the longest time I could never imagine running my own show. There was too much work, stress and responsibility. I was happy trading independence for the comfort and security of a steady paycheck. But over the last few years I've been feeling a deep and acute dissatisfaction working for someone else. I came to the conclusion that I wasn't going to be happy until I stopped working on other people's ideas and took a shot (or two) at working on my own.

So now it's time to stop being so unhappy about it. As of May, I'm on my own. My plan is to build a small product-development company. I don't need to make scads of money, I just want to support my family. I want to work on things that I deeply care about. I want to spend my time doing things that matter to me, and as little time as possible on "shoulds", "musts", and "have-tos". I'll pick up contracting work as needed (tell all your friends), but the long-term plan is to live off of product revenue. I guess some people might call it a "lifestyle business". I call it dropping out of the rat-race and living the life more fully.

How am I going to do this? What's my plan? Beats the hell out of me. I'll figure it out as I go. All I know is that everything I've gone through in the last month has made me realize that it's going to work out fine. There's nothing that could happen that would equal what I've recently gone through.

So what about the title of this post? What exactly am I so thankful for? Well, certainly not about the loss of my mother. Nothing could replace her. Nothing will bring her back. She was my moral compass. She was the greatest teacher I ever had. I didn't really appreciate the tools she left me with until she was gone. Now I realize that she left me with everything I needed to cope with this tragedy as well as being able to move beyond it. I don't know if I would ever have been shaken out of my day-to-day routine without her loss. I wouldn't trade this epiphany for her, but I don't have that option. What I can do is move forward and live a life that honors her memory.

My mom was a survivor. She survived the breakup of my family. She survived cancer. She survived my brother's death. After her divorce she rebuilt her life from scratch, eventually becoming an assistant Dean at the School of Education at the University of Oregon. Her crowning achievement was completion of school's new building on the U of O campus. Over the span of nearly a decade she raised funds from private donors—mostly teachers. But the final dollar amount isn't what's impressive about her work. What's impressive is how she took a group of people, turned them into a community and got them to believe in themselves. Mom, I wish I could have told you just how proud I am of you.

So, mom, this life is for you. You always had a strong independent streak. You didn't get a chance to enjoy the future that you worked so hard for. You gave me so much. The best way I can think to pay it back, is to live the life you would have wanted for me, and to pass that on to my daughter.

Thanks, mom. Thank you, so very much.

First, is the script that selects the SDK for the device:

tell application "Xcode"
      set myProject to active project document
      set active SDK of myProject to "iphoneos3.1.3"
    end tell
    

Next up, the script that selects the simulator SDK:

tell application "Xcode"
      set myProject to active project document
      set active SDK of myProject to "iphonesimulator3.1.3"
    end tell
    

Now, open up Xcode and and select "Edit User Scripts..." in the scripts menu:

This opens the script organizer. In the left-hand pane is a list of all the script groups. Which group you put a script in effects what submenu they are available from in the main scripts menu. But we like shortcut keys so who cares where it goes? I put these in the "Build" group, but feel free to put them wherever makes the most sense to you. Select the group you want to add these scripts to and click the "+" button and choose "Add Script File…":

Use the standard Finder dialog to find the scripts you just downloaded and select one of them. To set the shortcut key for the script, double-click the right-hand column of the newly-added script. For the simulator script I use ⌃⌘⎇S and ⌃⌘⎇D for the device script. You can try out different combinations. If you come up with a key-chord that clobbers an existing key-binding Xcode will give you a warning at the bottom of the window.

One thing to be aware of is that while you're setting the shortcut key for the script all of your keyboard input is directed to that field. So you can't use ESC to cancel or Enter to set the value. You need to grab that infernal mouse and click somewhere else.

Before you close the dialog, be sure to set the "Output" drop-down menu to "Discard Output" otherwise Xcode will paste some odd gibberish into your current file. And, just in case there's some kind of unforseeable error, set "Errors" drop down to "Display in Alert". Once you have both scripts configured you can close the dialog and try your new hot-keys out.

I don't claim to have any proficiency with AppleScript, so there may very well be a better way to do this. Certainly one thing that is brittle about these scripts is how they are hard-coded to a particular version of the SDK. Whenever a new SDK is released I'll have to update the strings in the script. Oh well, it beats having to reach for the mouse several more times a day.

So I've setup a tiny bit of zsh-fu to automatically switch which ruby implementation to use based on the directory that I'm in. To do this, one needs to take advantage of zsh's hooks, particular the one invoked when you change directories. By default, you can simply declare a function named chdir() and it will automatically be invoked whenever you change directories. Mine looks like this:

chpwd() {
        [[ -t 1 ]] || return
        case $TERM in
          sun-cmd) print -Pn "\e]l%~\e\\"
            ;;
          *xterm*|rxvt|(dt|k|E)term) print -Pn "\e]2;%3~\a"
            ;;
        esac
    }
    

This function just prints out the name of the directory I've change into and it's found in a lot of zsh distributions. To keep things clean I want to have a separate function that just manages the RVM updating. For the moment, let's ignore the fact that I already have an existing chdir() function, and look at the hook function I created to invoke RVM:

chpwd_check_rvm() {
        current_version=$(rvm info | grep " version:" | cut -d '"' -f2)
        dir=$(pwd)
        while [ "${dir}" != "" ]; do
            cfg="${dir}/.rvminfo"
            if [ -f ${cfg} ]; then
                want_version=$(cat ${cfg})
                if [ "${want_version}" != "${current_version}" ]; then
                    rvm use ${want_version}
                fi
                break
            else
                dir=${dir%/*}
            fi
        done
    }
    

This function looks for a file named .rvminfo in the current directory (this function is invoked after we've changed directories). If the file isn't found in the new directory, it searches upward through each parent directory to see if it can find one. If it's found, the function invokes rvm use with the version string found in that file.

$ cat .rvminfo
    1.8.6
    

There's also some additional checking to avoid extra rvm invocations if we're already set to the desired version. This is done more to reduce the chatter in the shell than for performance reasons.

Now let's get back to invoking multiple hook functions. If you have more than one function you'd like to have called when you change directories, you can declare these in an array named chpwd_functions. After declaring the chpwd_check_rvm() function above, having two functions for the cd hook is trivial. I just add this declaration to my ~/.zshrc file:

#--- chpwd_functions
    chpwd_functions=( chpwd_check_rvm chpwd )
    

Now, whenever I change into a directory with a .rvminfo file, my hooks are automatically executed and I get the right ruby version:

[island] ~/Development: cd alexvollmer.com 
    <i> Now using ruby 1.8.6 p383 </i>
    

One final note is that I put all of my specific functions in the ~/.zsh/functions directory, and automatically load all of them in my ~/.zshrc file with the snippet below. I do all of this before I declare the chpwd_functions array.

#--- Shell Functions ---
    #
    fpath+=(
      ${HOME}/.zsh/functions
    )
    
    autoload -U ~/.zsh/functions/*(:t)
    

You may want to organize your functions differently. Just be aware that zsh needs to know about your functions before you can put them in the chpwd_functions array.

UPDATE (3/1/2010): Per comments below from RVM's creator, Wayne Seguin, RVM actually comes with it's own built-in way of handling this. Somehow I completely missed the rvmrc documentation for this feature (how embarassing!). I'm usually inclined to avoid re-inventing the wheel, but there are two small things I like about the way I implemented this. First, the solution proposed here doesn't redefine the cd command, which feels a bit like duck-punching the shell to me. I don't know if bash has the same kind of hooks that zsh does, so the implementation is understandable. As for me, I like the hooks better. Second, the solution I show here works for any sub-directory within a project that has a specific RVM setting. That said, I'll probably just use what's already been implemented in RVM.

I chose to use Nanoc as the site-generator. Originally I was going to go with Jekyll. I really liked its simplicity and the fact that it had prescribed locations for everything. However, I still wanted to maintain some backward-compatability with the old site. This meant a little bending of Jekyll's rules and, frankly, Jekyll just isn't as configurable and flexible as Nanoc. That's not a dig against Jekyll, nor is it an endorsement of Nanoc. Nanoc is a general-purpose solution for generating static sites, but it's not necessarily optimized for blogging. But with Nanoc I can use HAML, SASS and other Rails-isms with which I'm already familiar. If you want to see the gory details, I've put the whole thing up on GitHub.

Sadly, this ended up taking much longer than I anticipated. First, I had to import the old posts from Wordpress. The database schema for Wordpress is pretty wacky looking when you're used to working with Rails. Thankfully, Jekyll already gone down this path and I used their Wordpress importer as inspiration for writing my own. Along the way I discovered what great gem Sequel is. A large part of the importing process was converting from all of the jacked-up formats I've amassed over the years to Markdown. It took a lot of iterations to catch every weird character-encoding and formatting issue.

Another large effort was creating the RSS feeds. Now Nanoc has some support for Atom, but it didn't quite do what I wanted. I was trying to match what Wordpress was generating for me so I basically re-implemented feed-generation from scratch.

Moving old comments into Disqus took about a week to get right. As a user, the Disqus service is pretty cool. As a web API, it's a little…ahem… lacking. If you're used to well-constructed REST APIs, you'd best drink something stiff before settling in with the API docs. The disqus gem borders on useless. So, I ended having to write my own importer with httparty. It took several iterations to really get this nailed and the process was pretty frustrating.

The sad part about this, is that originally I was on the fence about adding comments at all. Initially I thought I'd try it out and see if I liked it. But I'm a geek, right? I couldn't get it working right away and it drove me nuts! I had to figure it out. So eventually I figured out the magic recipe, and now I'm wondering if I even really wanted comments integrated at all. Oh well.

One trick that I found very helpful throughout this process was creating some scripts that dropped me into an IRB session with the Nanoc blog or Disqus API objects already initialized. This was inspired by the Rails console which I find invaluable for tinkering with things in real-time. For example, while I was dickering with the Nanoc internals, I found this console script to be quite helpful:

#!/usr/bin/env ruby
    
    require "irb"
    require "rubygems"
    require "nanoc3"
    load "lib/helpers.rb"
    
    @site = Nanoc3::Site.new(".")
    @site.load_data
    
    IRB.start
    

Now, Wordpress and I differ on what constitutes a sensible URL scheme. So that last bit I needed was to concoct some redirect rules to port the old URL scheme to the new one. Currently, this site is hosted on http://dreamhost.com which, provides support for .htaccess files. With a little Apache-fu I was able to get most of the old WP URLs to redirect correctly:

RewriteEngine ON
    RewriteRule ^index.php/feed/ /feeds/rss_2_0.xml [R]
    RewriteRule ^index.php/feed/rss/ /feeds/rss_0_9_2.xml [R]
    RewriteRule ^index.php/feed/atom/ /feeds/atom.xml [R]
    RewriteRule ^index.php/(.*) /posts/$1 [R]
    
    ErrorDocument 404 /missing.html
    

Well, that took longer than I thought. I can't say that it wasn't educational. Now, what was I supposed be doing?

Those that damned the iPad seemed to come in two flavors: those that saw the iPad as the embodiment of corporate greed and walled gardens, and those that thought the device didn't measure up to their personal expectations.

There is a sizable (and growing) population of folks that simply will not purchase an Apple product or participate in its eco-system for various reasons. I think that's fine. While I personally enjoy their products, I don't believe that everyone has to agree with me. I get no less enjoyment out of my MacBook or iPhone by someone else having a Lenovo ThinkPad or an Android Phone.

Unfortunately, that view isn't shared by many. Take the flame-baiting in the aftermath of the iPad announcement and replace the term "iPad" with "gay marriage" and the arguments look nearly identical. Gay marriage is so upsetting to conservatives that they can't imagine being in a world where it exists. It's not that gays are coming into their homes and performing unspeakable acts, it's that they can't imagine even being on the same planet with them. So it goes with the iPad—there are the FSF types who can't imagine living in a world where Apple controls every single thing that goes on a device.

I'm not aware of anything in the President's address that said every man, woman and child in the U.S. was going to be forced to use an iPad. As far as I know, people are still free to choose whether or not they wish to purchase and iPad and wander about in Apple's walled garden. Don't want an iPad? Fabulous. Don't buy one.

Folks, it's a big world out there. There doesn't have to be a single winner. Even as Apple trumpets having over 50 million touch devices, that pales in comparison to the worldwide population of those who could afford such devices. That's even without considering the fact that if you own one iPod/iPhone, you probably own another. In terms of real numbers, I don't see a growing "collectivization" movement to dress everyone in Apple products. If Apple were interested in market share, they would charge a lot less for their products.

Besides calling the iPad a freedom-killer, the other category of critiques were that the device was simply disappointing. This reaction particularly surprises me. Yes, there are certainly quibbles to make about various features of the device (no camera, no multitasking, etc.) but the dismissive observations that the iPad is simply "a big iPhone" I think misses the point. Yeah, it is a big iPhone, which has been a huge success by any measurement. If you love the iPhone, then the things you most wish it had were more of are screen real-estate and speed. So making one that is bigger and faster is totally logical. What did people expect? Roomba integration? A toaster? 3D goggles?

ReadWriteWeb attempted to break down the anger with this graph:

Looking at this chart, I would categorize 38% of the complaints as just pure snark. The next 27% that dismiss it as a "big iPhone" are missing the point. The remaining complaints are all valid and worth talking about a bit:

No Flash

Hooray, I say. I don't run Flash on my desktop or laptop if I can help it. I simply do not like Flash. It rarely enhances my experience on a site and often pegs my CPUs for no good reason. Yes, I'm aware that Adobe has to do a lot of video codec work without the aid of hardware, but that doesn't address the jarring user experience that is Flash. I have not missed Flash on the iPhone either. If you want Flash, don't buy a iPod, iPhone or iPad. It's as simple as that.

Oh, and if you're running a Mac and feel the same way, do yourself a favor and install the Click to Flash plugin, which lets you ignore Flash on the web completely.

No Multitasking

I'm on the fence about this one. I'm not convinced that multi-tasking is essential for a device like the iPhone or the iPad. The truth is, I have not had many instances where I thought, "Damn! If only I could run two apps at once!" I think a lot of people think they want it because their computer has this capability.

I think that's the point that is being missed. The iPad is not, in any way, being put forth as a general computing device. In fact, to categorize it as a "tablet PC" is to completely misunderstand the iPad's underlying design philosophy. When I watched the presentation and demonstration video what I saw was easy execution of tasks. Nobody was demonstrating bit-rates, frames-per-second, or tail-call optimization. The underlying technology and "platform" are simply the means to an end.

This is a different enough device that people will have to change their thinking about their relationship with their computers. I don't think it means a comprimising relationship, but one that is different from today. Fraser Speirs eloquently called the inability to comprehend this shift as Future Shock.

Lock-In

Another major complaint of the iPad and the entire Apple/iTunes ecosystem is that it is a closed system. Users don't get to mix and match devices freely from other manufacturers. The party-line from Apple is that they want to control every aspect to provide the best user experience. Detractors argue that it is part of Apple's plan to own all of your content like Big Brother (as Microsoft has attempted to in the past).

I get why people feel uncomfortable with handing Apple control of their stuff. Personally, I have a lot of Apple products in my house and I'm happy to give them that control because, for me, it is a better experience. I will happily trade some vague notion of "freedom" for stuff that just works. Endless fiddling and integration of disparate technologies isn't freedom at all—it's an enslavement of the most precious resource I have, which is my time.

Not everyone is happy with that and they prefer an alternative. Hey, no problem. Live and let live. You want to run Boxee on Linux connected to an iRiver. More power to you. I won't sneer at you or call you a fool. If it makes you happy, your choices don't affect me in the slightest.

As an aside, if anyone complaining about the lack of Flash is also complaining about lock-in, that person needs to be hit repeatedly with a rake.

No Camera

Sigh. Yeah, this one is a bit disappointing. I don't really care so much about having for video chatting (which I do maybe once a year). But I really like using a camera to capture things I would otherwise have to take a lot of effort to write down. How cool would it be to have your iPad in a brainstorming session where you can still jot a few notes down then associate a snapshot of the whiteboard?

Now there is an interesting design conundrum here. Do you put the camera on the front or the back? Depending on the tasks you expect to perform, this makes a huge difference. If you want people to see you, you put it on the front and figure out how to give the user the user visual feedback so they can position the camera.

If you want to take pictures of other things, you put the camera on the back and use the video display as the viewfinder. However you can't reasonably reverse the tasks for these two positions. OK, so do you put two cameras on? I don't know. How do you control them? Does each bit of software implicitly decide which camera to enable? Does the user have to decide?

I don't know the answer, but I would be very surprised if a camera didn't show up in the next version of the device. They put cameras in the iPod Nanos for cryin' out loud. Surely they can figure out how to stick one in an iPad.

More AT & T

The amount of bad press AT & T has received via the iPhone has to make them wonder what sort of devil they've made a deal with. Are there serious issues with AT & T? You betcha. Do they effect everyone? I'm not convinced. Coverage in San Francisco and Manhattan seem acutely poor, but in my corner of the Northwest coverage is pretty damn good.

The non-iPhone types love to make fun of AT&T, fueled largely (I think) by schadenfreude. I'd be very curious to see how other carriers handle a similar load. I used to work in that industry and the mentality among carriers is all the same. They all want to wring every penny they can out of you by charging for every bit you use. Let's not kid ourselves.

Kudos to Apple for making a WiFi-only version. For me, this is a pretty compelling configuration. I'm an urban kid. There are few places I go where I can't get some kind of access to WiFi. In the cases where I can't get WiFi, I have an iPhone. Plus, I imagine using an iPad primarily at home, or on vacation, but not for commuting.

I'm no AT & T apologist—there are some serious issues they need address— but I'm not convinced that this makes the iPad a "failure".

Conclusion

Am I an Apple apologist? I hope not, but people are notoriously bad judges of their own character. What I can say is that there are certainly things about the company that frustrate both as a user and developer. However, I am still more delighted by their products than I am frustrated with them. When that balance changes, I'll move on to something else.

I'm pretty jazzed about this device, both as a consumer and as a developer. What sucked me into the iPhone was how it just oozed with utility. Not in terms of potential energy, but in that it fulfills the notion of computers as and extension of our own minds. The iPad just takes the next step forward. I can't wait to see how it all unfolds.

P.S. Just I'm glad to see the iPhone finally get some competition, I'm also looking forward to somebody putting up a good fight against the iPad.

After returning from WWDC I was inspired to try this technique out and after playing with the idea for a bit, it was clear that there was a lot of repetition in the process. To really make prototyping cheap, we need a simple framework that makes the process fast and easy. Enter Briefs, a lightweight iPhone application that allows you to embed and run simple prototypes built with nothing more than images and a property list.

With Briefs, you don't write a separate application for the device (imagine the provisioning headaches), but instead embed separate prototypes (called "briefs") into a single Briefs application. Briefs is still pretty new and I expect to see (and contribute) many enhancements, but if you like this style of application development, it's certainly worth spending some time with Briefs.

Let's look at how it works. We're going to start with the goal of prototyping the built-in Photos application. Now obviously, nobody is going to actually re-write the Photos app, but since we all have familiarity with it, it's a good example to see how a working application can be expressed as a Briefs prototype.

Getting Briefs

Before we do anything, we need to get Briefs on our local machine. Briefs is distributed in source-only form via GitHub. You'll need to find a nice place to check out the two main Briefs project, then do the following:

$ git clone git://github.com/capttaco/Briefs.git
    $ git clone git://github.com/capttaco/Briefs-util.git
    $ cd Briefs
    $ git submodule update -i
    

In order to put our first Brief together, we need to build a command-line tool that is part of the Briefs project. Open the Xcode project in the Briefs folder (Briefs.xcodeproj) and select the "Briefs Compactor" target and set the architecture to "Base SDK".

Now execute the target to build the compact-briefs executable. When it's finished, Xcode will put the final binary in build/Debug/compact-briefs in the Briefs project directory. Copy this file to a location in your $PATH. Personally I like to keep such stuff in ~/bin. We'll need this command in a little bit.

How it Works

Before we dive into our project, we should get a little terminology under our belt. Each application prototype you create is called a "Brief". Each Brief is composed of one or more "Scenes". Scenes usually have an associated image, so you can think of a scene and a screen in the application as being more or less equivalent.

Scenes can also have any number of "Actors", which are regions on the screen that perform some action when touched. For example, to mimic navigating to a new view via a view controller, a region on a table view could be rigged to an action that slides another scene into place.

Briefs are expressed as Property List (Plist) files in the Briefs application. If you like writing XML or are proficient with the Property List Editor application, then you're golden. Otherwise there is another format in which you can write your brief that compiles down to a final Plist. This is the route we'll explore in our prototype.

Protyping Photos.app

Generate Image Files

Now let's generate the image files for our prototype. In the case of the existing Photos app, I just screen-captured the running application on the simulator. Obviously, you won't be in the same position with a new application, so you'll need to generate some images by hand.

If you use OmniGraffle, there are several good iPhone stencil sets on Graffletopia. If you prefer Photoshop, checkout the exhaustive iPhone GUI PSD. Either one of them will generate the appropriate files. There's no requirement that the image files look like the iPhone. You could use a tool like Balsamiq or scan hand-drawn sketches to generate very simple wireframe sketches. Regardless, all you need to do is produce PNG, GIF or JPEG files that fit within the normal iPhone dimensions (320 x 480).

Let's create a folder called "photos app" and put our image files in it. I've created a tar-ball of this entire project, which you can download, explore and run.

Writing the Brief

Let's think a bit about the flow of our application. We're only going to implement a sub-set of Photo.app's full navigation—we'll just implement the flow for the Hawaii photo album. We have a top-level screen that shows all of the photo albums. From there the user can select "Hawaii" and drill into a display of a few images in the album. Touching any of the images will take the user into the image browser where users can go back and forth between the various images. Each screen has a common "back" button in the top-left. The overall screen layout and flow looks like this:

The pink boxes highlight the touchable areas in each scene. The arrows between them show which scene a particular area navigates to. In Briefs-parlance, each screen is a "Scene" and each pink box is an "Actor".

Let's step through the Brief (the photos.bs file), a bit at a time. If you get lost, refer back to the application flow diagram above.

First we declare that our starting scene is the "Home" scene. Next, we define that scene with the image home.png as the background image and one actor that will navigate to the next scene. The position and size of this actor corresponds to the table cell labeled "Hawaii". If we wanted to support the table rows, we'd need to add an actor for each one.

When the user touches the "Hawaii" row, we'll transition the "Hawaii" scene (defined shortly) by executing a "push left" animation. This will mimic the standard animation of pushing a new view controller onto the navigation controller stack:

start:Home
    scene:Home
      image:home.png
      actor:HawaiiPictures 
        position:0,123 
        wh:320,53
        action:goto(Hawaii, push left)
    

Next, we'll define a scene that shows the thumbnail images for each photo in the Hawaii album. This scene will display the hawaii.png background image and has an actor for the return navigation button in the upper-left as well as for each thumbnail image. Note the differences in the direction of the push animation in the actors. The "back" button pushes right (like the normal return navigation animation), while the thumbnails push left.

scene:Hawaii
      image:hawaii.png
      actor:Home 
        position:17,30 
        wh:87,24 
        action:goto(Home, push right)
      actor:image1 
        position:5,69 
        wh:72,72 
        action:goto(Image1, push left)
      actor:image2
        position:84,70
        wh:72,72
        action:goto(Image2, push left)
      actor:image3
        position:162,70
        wh:72,72
        action:goto(Image3, push left)
    

Now we'll define the scenes for each of the three images. Each image scene has an actor for the return navigation button in the top-left. The first image also has two actors to navigate to the next image. The NextImage actor is for the navigation button in the toolbar, whereas the NextImageSwipe actor covers the right side of the image itself. This is done to mimic the swiping motion normally used to navigate photos. Right now, Briefs only supports single-touch interactions so it won't behave exactly like the real application in supporting swipes.

scene:Image1
      image:image1.png
      actor:NavigateBack 
        position:14,33 
        wh:44,23 
        action:goto(Hawaii, push right)
      actor:NextImage 
        position:202,450 
        wh:23,17 
        action:goto(Image2, push left)
      actor:NextImageSwipe 
        position:156,60 
        wh:160,380 
        action:goto(Image2, push left)
    

The "Image2" scene is a lot like the "Image1" scene, except that it has button and swipe actors for both the previous and next images:

scene:Image2
      image:image2.png
      actor:NavigateBack 
        position:14,33 
        wh:44,23 
        action:goto(Hawaii, push right)
      actor:PreviousImage 
        position:94,452 
        wh:23,17 
        action:goto(Image1, push right)
      actor:PreviousImageSwipe 
        position:0,60 
        wh:160,380 
        action:goto(Image1, push right)
      actor:NextImage 
        position:202,450 
        wh:23,17 
        action:goto(Image3, push left)
      actor:NextImageSwipe 
        position:156,60
        wh:160,380 
        action:goto(Image3, push left)
    

Finally, the last image scene. It only has "backward" navigation in the photo set since we've only put three photos in our album:

scene:Image3
      image:image3.png
      actor:NavigateBack 
        position:14,33 
        wh:44,23 
        action:goto(Hawaii, push right)
      actor:PreviousImage 
        position:94,452 
        wh:23,17 
        action:goto(Image2, push right)
      actor:PreviousImageSwipe 
        position:0,60 
        wh:160,380 
        action:goto(Image2, push right)
    

We've just done simple scene-to-scene transitions in this Brief, but there are also several transitions you can perform directly on actors such as moving them, fading them in and out and others. I've created a syntax cheatsheet which you can use as a reference for your own Briefs.

Compiling the Brief

Once we have our photos, it's time to start writing our Brief. As mentioned earlier, the Briefs application consumes Plist files. However you can't just create one out of the box because it will only have image file references in it, but not the actual data. The reason we built the compact-briefs command earlier, was so that we could take our intermediate Briefs file (usually called a source brieflist) and embed the image data directly in the final product.

However I find editing XML tiresome, whether it's by hand or with a tool. Briefs provides an alternate syntax in which to write briefs which is transformed by a Ruby script into the intermediate Plist format. You can then compile the source brieflist to the final format using the compact-briefs command. The flow between formats and tools looks like this:

In the Briefs-util project (which you checked out earlier), you can find the Ruby script in BS/bs. I run these tools all at once in a script named compile:

#!/bin/bash
    ~/Development/Briefs-util/BS/bs photos.bs > photos-source.brieflist && \
        ~/bin/compact-briefs photos-source.brieflist ~/Development/Briefs/sample/photos.brieflist
    

Installing the Brief

The next step is to build the Briefs application with our newly-compiled Brief. Switch back to the Briefs Xcode project and choose the "Briefs" application target. If you haven't yet added the new brief to the Xcode project, right-click on the "My Briefs" and choose Add > Existing Files. Select the photos.brieflist file in the samples directory and in the following dialog make sure the checkbox at the top is unselected.

Now you can build the Briefs application for either the simulator or the device. If you're building for the device, make sure you have a proper provisioning profile in place that matches the bundle identifier of the project. Now we can Build & Run the project, and our photos.brieflist should appear in the list:

Select our new Brief and play with the application. To exit a Brief, touch and hold your finger down until a new screen appears asking you if you want to exit the current Brief.

Distributing the Brief

The idea behind these prototypes is putting a tangible example in front of users to get their feedback. So what do you do if you want to distribute Briefs to a wider audience? Ad-hoc builds are a pain and require access to the physical device. Fortunately, Briefs provides a tool called "Briefcasts", which is simply one or more Briefs embedded in an RSS feed. The Briefs application can retrieve Briefcasts via HTTP, so distributing Briefs is a snap if you have a web server to serve new Briefcasts from. This means you can install the Briefs application once on a device and provide updates anytime without having direct access to the hardware.

Currently, there isn't a tool to easily create a Briefcast directly out of a list of Briefs. For now, the project web site recommends copying and modifying the RSS template below:

<?xml version="1.0"?>
    <rss version="2.0">
      <channel>
        <title>*Briefcast Demo*</title>
        <link>*http://island.local/~alex/briefcasts/*</link>
        <description>*Demonstrate how awesome it is to use a briefcast to get briefs on the iPhone.*</description>
        <language>en-us</language>
        *Thu, 12 Nov 2009 03:05:00 GMT*</pubDate>
        <lastBuildDate>*Thu, 12 Nov 2009 03:05:00 GMT*</lastBuildDate>
        <item>
          <title>*Photos.app Sketch*</title>
          <enclosure url="*http://island.local/~alex/briefcasts/photos.brieflist*" length="*954708*" type="application/brief" />
          <description>*An example brief of the built-in Photos.app.*</description>
          *Sun, 13 Sep 2009 03:05:00 GMT*</pubDate>
          <guid>*http://island.local/~alex/cast/1#item1*</guid>
        </item>
      </channel>
    </rss>
    

The bits marked in bold need to replaced with your particular details. In the example above I only have one Brief, but if you had more than one, you would just include additional elements. You need to have this RSS file on disk as well as the brieflist file referred to in the tag.

Assuming that you have your web server all setup with the RSS file as well as the Briefs files, you can launch the application and add the Briefcast by selecting the "+" button in the top-right of the home screen. Enter the details on the next dialog and touch the "Save" button:

On the next screen you'll see the details of the Briefcast. Each Brief in the Briefcast will be listed under the section titled "Enclosed Briefs". Touch the "Download this Brief" button to retrieve the associated brief.

Once the brief is downloaded, you end up on a rather unhelpful confirmation screen:

You have to touch the screen, then navigate back to the top. Then you'll find the new Brief in your list of briefs. To update the Brief, you navigate back into the Briefcast section from the home page and download the correct Brief again.

One nice enhancement I'd love to see is a custom application URL for Briefs so that you could just email the link to someone. They could read the email from their device, select the URL and Briefs would launch and download the newest Briefs.

Conclusion

Briefs is a great tool for building live prototypes. Since it's a relatively young project, it still has a lot of rough edges, but I expect it to mature over time.

Don't hesitate to dive into the code behind Briefs. There are occasions when Briefs (the application or the command-line tool) will crash without much detail. I found that this is usually the result of bad syntax in the Briefs so a little sleuthing may be required to get things working.

Resources

Here's a list of related resources:

• Briefs Homepage
• Briefs on GitHub
• Sample Brief of Photos.app
• Briefs Syntax Cheatsheet
• iPhone Templates for OmniGraffle
• iPhone PSD for PhotoShop

LaunchBar

When I first switched to the Mac, I discovered Quicksilver and it was like discovering something essential that you didn't know you needed. QS was a revelation to me because it drastically reduced the mental distance between the what I wanted to accomplish and executing it.

Over time though, I found some things in QS that just didn't work consistently. It would crash and some of configuration was pretty clumsy. It became pretty clear that there was little, if any, forward progress being made on it. So, on the advice of several local Ruby/Mac nerds, I tried out LaunchBar. At first I was disappointed that it didn't have the breadth of QS, but after investing a little time and effort I found that it handled 99% of the things I wanted. More importantly, there was somebody actually working on it which meant it was going to evolve and improve.

Now I can hardly use a machine that doesn't have it installed. I can't believe people use the dock to launch apps. Yuck.

Mailplane

By and large, I'm biased to using the built-in Apple-given tools where possible. Apple usually gets it right and most other tools are designed to work with these apps. I've used Mail.app since I switched to the Mac, but always felt the interface to be inferior to GMail's tag-oriented structure. But since I had some non-GMail accounts, and I wanted to have a single app, I stuck with Mail.app.

However, once I dropped my last IMAP-only account I was ready to switch back. The only parts I really missed were the tight integration with the desktop that Mail.app brought. Lo and behold, along comes Mailplane to fill the gap. It is, essentially, a WebKit view of GMail with some extra bits to glue the web application to your desktop. I think it's brilliant and was happy to part with some hard-earned money for the privilege to use it.

zsh completion

My introduction to zsh came many years ago from a co-worker who boot-strapped my entrée into zsh-land with his configuration files. For the longest time I treated them like mystic, sacred tablets to be worshipped, but never to be understood. But over time I found little gaps I wanted to fill in with my zsh setup, especially with command auto-completion.

I'm a keyboard kind of guy. I like the command prompt. I also like going fast, so anything that makes me faster in the command prompt is like gold to me. I love shell auto-completion and found lots of things I wanted completion for, like RubyGems. So this year I finally pounded through enough documentation and samples to put together a few home-grown completion bits. I still don't understand most of the incantations that make it go (guilty as charged of cargo-culting) but it has made my command-line work-flow that much better.

Dropbox

Since switching to the Mac, I've always had at least two machines that I needed to share files between. For the longest time I used to store files in S3 then sync them back and forth. This worked, but it sucked because I had to do it manually and I had to get the order right so I didn't clobber any files.

When Dropbox came along, the clouds parted and the golden light of heaven shown down from above. Dropbox makes sharing files super-duper, idiotically simple. Combined with tools like 1Password, having consistent data across multiple machines is like magic on a daily basis—it just works and I don't even have to think about it.

One Bus Away

The thing I love about the iPhone is how it's an Swiss Army Knife for information in the modern age. You can find apps that do very specific things, very well and put them in a single box that you keep in your pocket. The One Bus Away app is one of my favorite iPhone tools.

I ride the bus a lot and to have real-time transit data in the palm of my hand is fantastic. No guessing about when the next bus is coming. No extra waiting. I know just when to leave to catch my next ride.

Mint.com

I was a long-time Quicken user but finally had it last January. It's a bloated app with serious usability issues. I finally reached a point where using Quicken had a net-negative impact on my life. It simply drove me crazy to use it.

Enter mint.com. At first I was very hesitant to give my information to these guys. But eventually I was convinced to try it because: * if I use Quicken another minute I'll go postal * sometimes the future doesn't start until you acknowledge it, and this looks like the future to me * I'm probably already more open to financial security risk anyway, so what the hell?

I get exactly what I want out of mint: a high-level dashboard that just updates itself. I use a few of the budgeting features which are nice, but really it's all about automated data acquisition and display. Oh yeah, did I mention that they have an iPhone app too?

Instapaper

Several years ago I built my own news-feed reader and one of the features I really wanted to implement was a temporary "holding pen" where I could put URLs to visit later. These were things that didn't quite warrant a Delicious bookmark, but I wanted to get back to.

By itself, Instapaper satisfied my original need and worked great as a bookmarklet in my browser. However it went from a convenient curiosity to an essential tool once other apps started integrating with it (e.g. Net News Wire, Tweetie 2) and Kindle integration (see below).

Oh yeah, did I mention that there's an iPhone app for that too?

Kindle (device and app)

The final tool of 2009 that made itself indispensable was the Kindle. It's a great example of "good enough". The Kindle is fraught with polish and usability issues, but they don't overwhelm the utility of the device.

I've been on a long dispense-with-all-physical-media kick, with books being my primary target. I have lots of books, but I don't have lots of space. Since I'm not emotionally wedded to the form-factor of paper books, the Kindle solves my reading-appetite vs. space problem quite well. I can stuff that thing full of books and not take up an inch of extra space in my house.

Like so many of the other tools I've listed, it's the other things that integrate with the Kindle that multiply its utility. First, the Kindle has gained a lot of adoption by other publishers. The Prags and O'Reilly, in particular, have been eager to jump into the e-book biz with first-class Kindle support. I probably go through about fifteen to twenty tech books a year and having them on the Kindle is great.

Another great bit of integration is how Instapaper delivers me a weekly digest of links I've captured right to my Kindle. I can read the full content of every page I bookmarked right there on my Kindle without the need for a network connection. This is a great for travel.

So?

Looking over this list, if there's one theme that emerges, it's all about ubiquity and convergence. (Did I really just say that?) Despite all of the queasy feelings those "market-tecture" terms give me, I don't quite know how else to define it. For me, 2009 was the year that I could get to my stuff whenever I liked, however I liked, through a multitude of channels.

I wonder what the theme of 2010 will be?

With the exception of Gehrig, I've purposely ignored anyone before the 50's. In my opinion, the game is just too different. I don't feel like I can trot Cy Young out as my starting pitcher just because of 511 wins. Those wins don't compare at all to wins in the 60's, 70's, 80's or 90's.

I'm also not penalizing anyone for being part of the Steroid Era or for even having used PEDs. There are, no doubt, some effects of these substances on the game, but I don't see the evidence being as conclusive as some would have you believe. Did Barry Bonds cheat? Yeah, I'm pretty sure he did. He was a helluva ball-player before he started and he still would have put together a Hall of Fame career without them. Steroids (a rather inaccurate term for PEDs) don't make you see the ball any better or improve your hand/eye coordination. At best they allow athletes to recover faster from working out. They don't build muscle, they amplify the body's own healing process in order to work out more. You can call steroid users cheaters, but I wouldn't call them lazy.

So this is my list. If you have a different one, I'd love to hear about it. So, without further ado, here's my list.

Catcher: Roy Campanella

Great all-around athlete, hit the crap out of the ball, threw runners out and worked well with pitchers. He was probably one of the most complete catchers to play in the majors.

First Base: Lou Gehrig

OK, I cheated a little bit here and dipped back in time further than I said I would go. But c'mon—it's Lou Gehrig.

Second Base: Jackie Robinson

They retired the number 42 for lots of reasons—those are the same reasons he's my starting second baseman.

Third Base: George Brett

Brett is the last guy to come close to hitting .400. He ended the 1980 season with a .390 average. What's even more impressive is that he had more home runs than strikeouts that year. He's got the longevity, the ring and the career to put him on my list.

Shortstop: Alex Rodriguez

Does A-Rod's admission to using PEDs damage is Hall of Fame credentials? Maybe, but as I contended earlier, I'm not convinced that they can carry a player to greatness. We love to dump on A-Rod's failures, but it's really only because he's been spectacularly successful in his career. Drugs or not, he's my starting shortstop. This, coming from a jilted M's fan too!

Left Field: Ted Williams

The Splendid Splinter. The greatest hitter of all time. Period.

Center Field: Willie Mays

Hard to find a better combination of offensive and defensive skills.

Right Field: Hank Aaron

How could I possibly leave Hank Aaron off the list?

Designated Hitter: Edgar Martinez

Edgar Martinez presents a serious conundrum to Hall of Fame voters. He's really the first "pure" designated hitter that voters will have to face. His career are numbers are impressive for the time he played, but he doesn't have some of the magic totals people like to see. That said, he's my vote for DH. Plus, I named my dog after him.

Starting Pitcher: Sandy Koufax

There's probably no roster spot that would engender more differences of opinion than starting pitcher. There are so many great ones to choose from. Koufax didn't have the career longevity like Satchel Paige or Tom Seaver, but at the height of his powers he was simply jaw-dropping amazing. His years from 1963 through 1966 are some of the sickest consecutive years a pitcher ever put together.

Relief Pitcher: Mariano Rivera

While there may be controversy over starters, the choice for reliever is pretty straight-forward. Seriously, is there any disagreement here?

Manager: Earl Weaver

He managed Hall of Fame players, he won a couple of rings, he wrote the seminal book on baseball strategy and he had a video game named after him. Yup, that fits my criteria.

I always wanted a personal knowledge base like this, but I couldn't do it the way she had. First, I have never had the space to collect that much paper. I cannot have, and do not want, a giant filing cabinet of folders. Second, I don't have the kind of recall she did to find that stuff. I could certainly remember that I once filed something, but wouldn't have a clue where I filed it. Which brings me to the third issue, which is suffering with the restriction of filing information under a single hierarchy.

When I first started using GMail I had a major epiphany about the differences between organizing information in folders versus tagging. These days that seems about as earth-shattering as realizing our galaxy is helio-centric, but it seemed like a pretty radical idea at the time. I flat-out love this style of organization. I may not remember where I filed something, but I'm pretty consistent when it comes to labeling things. With systems like GMail and Delicious, I just slap as many tags as I think are reasonable for a message or bookmark and I have a really good chance of finding it again when I need to.

But email and bookmarks aren't a knowledge base. For me, they are a reference, but usually aren't in a succinct enough format to be a good reference. I want to boil down newly-acquired knowledge into some quick, efficient prose. What I needed was a system that: * Works across multiple machines (e.g. work and personal) * Is fast to add and edit content * Is (relatively) searchable * Has some structure * …but not so much that I fight with it

So over the last few years I've started building a personal knowledge-base in earnest. It's not overly complicated or too involved. I simply keep a set of text files (specifically, files compatible with org-mode) in a special Dropbox folder. Dropbox makes synchronizing stupid-easy. I just treat these files just like local files and they appear on both my work and personal machines.

For a while I used VoodooPad, which is a very cool program. But I like having a little structure in my notes and imparting that structure in VoodooPad took more effort than I wanted. Later, I discovered Emacs' org-mode, which turned out to be a perfect fit for me—it's a relatively lightweight structure on top of simple text-files. The only thing org-mode doesn't do well is handling non-text media. I've considered moving to something like OmniOutliner, but I've found that org-mode works surprisingly well.

The technology isn't particularly interesting, nor is it what makes this work for me. Rather, it's the discipline of writing down little notes on various topics of interest. For example, I can never remember how to enable zombies for Cocoa debugging. I looked it up once, then added a section to my xcode.org file on NSZombieEnabled. I get two benefits out of this: first I've got it somewhere handy that I can easily look up again (via spotlight, grep/ack, whatever). Secondly, the very act of formulating a paragraph or two on the concept somehow reinforces it into my brain. In fact, it's very likely that I won't ever have to look up how to track zombies again.

The first, is a gem I started over a year ago, called word-salad. It uses the built-in dictionary file to randomly pull words out. This is great when you just need to generate a bunch of English-like text, but don't want to fall back on the boring old "lorem ipsum…" routine.

3.words ==> ['draw', 'ameliorate', 'bonanza']
    2.sentences ==> ['Shoot jonesing the make castle.', 'Blue murdered slight bastion.']
    2.paragraphs ==> ["Brachypterous gastropod pheretrer overeager toploftily denaturalization stokesite demented benzalhydrazine archaeologic. Haverer hypophonous lenticularly brickliner urocele paucipinnate pik unprintably perhalogen. Subglenoid bearish gesticulative staircase gallop vesuvianite pneumatically overyear conterminous dreamish. Nonalliterated galliwasp superconfirmation Comandra entoil millionth parcellize rarefaction Cynoidea. Podolian metamorphosable nativeness integriously protonematoid undoctor stochastically dissatisfactory unchastity.", "Increate unloquacious unsatisfiedly flareboard internuncio beguine equivocation snowshoe Rhynchonellacea. Parochially curliewurly vermix consistorial cond consciencelessness Anaxagorize recoct sempiternally Campanulatae. Scorpionida Castalides homoanisic semipenniform Novemberish assessor preterlethal acrotarsial knoller hartin. Procrastination boatwise canonize differentiate faunlike countermarriage obstinance dilatableness drumloid. Gerate squirr Silvanus Physostigma booting thyroarytenoid diminutival legpuller medisance radiobserver."]
    

The second is a gem for automatically retrieving and locally storing sales reports from the iTunes Connect site. If you have an iPhone app in the App Store, you already know how much (anti-)fun it is to get reports from Apple's site. So I put the itunes-connect gem together to help out. It's pretty basic for now, but will probably grow more capabilities as time goes on.

This all works great—until it doesn't. The ability to stage parts of a file is one of my favorite features of git, but there are times when you have changes that git won't let you split apart. Usually I give up at this point and just abort the rebase.

But yesterday I must have had too much coffee because I was determined to figure out how to do this. The changes that I wanted to split out were pretty easy to find. It was easy to remove them by hand and commit those. But I didn't want to have to manually add them back—that was too mistake-prone. What I needed was a way to invert the commit that removed those lines. What I needed was git revert. If I could revert my subtractions (in effect, adding the changes back), then I could squash the commits and rewrite history.

So here's what I did:

1. Started an interactive rebase session (git rebase -i)
2. Marked the commit I wanted to split as an "edit"
3. Manually removed the code I didn't want in the first commit, staged the changes and made a new commit
4. Revert the subtraction with git revert HEAD
5. Complete that rebase with git rebase --continue
6. Start a new rebase and squash the original commit with the manual changes one

Note that I didn't use git reset after step 2. It wouldn't have helped me because interactive staging wasn't working for me. I also marked all the commits in the second rebase as "edits", so I could fix the commit messages.

Put another way, the commits transformed like this, from left to right:

Maybe I'm overly fussy about clean commits. I'll admit that at times that my attention to cleanliness borders on the obsessive, but having the ability to go back and clean things up is one of my favorite things about git. Maybe I'm just not smart enough, but I usually don't know nearly as much at the beginning of a commit as I do at the end. With git, I can make it look like I do.

Changes:

1.0.2 / 2009-10-02

• Merged patches from Adam Salter:

  • Added new "presence" method for options.
  • Updated rspec usage to match the latest & greatest syntax
  • Make descriptions optional for flags and options

• http://clip.rubyforge.org

But now it's time for a change. My professional clock is set to about three years—when that bell rings it's time for me to move on. This was as true at Evri as it has been anywhere else, with one major difference: I made a career-change along the way.

OK, I haven't given up software to become a sous-chef or to take troubled kids out into the woods to teach them discipline. But I think I have reached the end of my professional Java programming career. This is kind of a "Big Deal", because I've been doing it for a decade. I know, I know: never say never. It's entirely possible that I'll eat my words a year or two down the road, but for now, I can't help but feel like I'm closing a major chapter in my professional life.

I've been with that language for a long time. Lately I've made a sport of complaining about it a lot, but honestly that's simply contempt bred from familiarity. There is a lot of good about the language and, despite my railings, the baby needn't be thrown out with the bath-water. What it comes down to is that I simply don't enjoy it anymore. It used to be fun to put those pieces together, but not any longer. So I'm trading in my Java hat for my two new loves: Ruby and Objective-C.

I may be stupid, but I'm not foolish enough to think that these languages will be the only remaining languages of my career. First of all, I'm not in the habit of defining my identity from the particular technologies I work with. Secondly, these are simply the environments I'm in love with now. I'm a dedicated family-man when it comes to people, but when it comes to programming languages, tools and environments I'm as bad as Liz Taylor. But jeez, why not embrace change? I don't do this for the paycheck. I do it because I love it. Why spend my time on things that aren't making me happy anymore?

With that question burning a hole in my brain, it became clear that, for now, the twisting road that is my professional career is going to take a long detour through Ruby and Objective-C. I hope it gets some time with Erlang too—I've only dabbled with it a little bit, but found myself quite attracted right away.

As I realized that my time was at an end at Evri, the question of "what next?" loomed large. Was there someone out there that wanted someone with the skills I wanted to cultivate? If so, were they doing something cool? Was I going to go "indie"? What would that look like? Questions, questions, questions. Fortunately I stumbled across a gig that is going to let me scratch most, if not all, of those itches.

I'm taking October off to continue and complete the series of iPhone screencasts I've been writing for PeepCode. In November I'll pick up my full-time gig and kick off the next phase of my professional life. You can't imagine how humbled and fortunate I feel to be able to do this.

I'm pleased to announce the release of the first in a series of PeepCode screencasts for the iPhone SDK I've been co-writing with Geoffrey Grosenbach. The first one (done in two parts) covers View Controller basics. We decided to start with View Controllers because they are at the core of all "productivity"-style applications. Once you have them down, you can start to put some meat on the bones of your iPhone apps.

Once we've covered those, we'll work our way through the entire MVC Holy Trinity, one bit at a time. Along the way we hope to deep-dive on a few ancillary topics like Core Location, Game Kit as well as provisioning and unit-testing. We also hope to look at visual technologies like Core Graphics and Core Animation, as well model-related APIs like Core Data and web service-related libraries.

The iPhone SDK is huge and there's a ton of material out there to cover. We're very excited to kick off what we hope will be a useful and successful instructional series for perspective iPhone developers out there. Let us know what you want to see.

I just wanted to make a quick announcement that the iPhone app I've been building at work just released a major update today. It's in the App Store and it's free. Check it out.

Working on this release was a ton of fun. We cleaned up a good bit of the mess from the earlier version and added some cool new features. What was even more fun was removing functionality that either didn't work well or wasn't obvious to the user. My god, there is nothing I love as much as deleting code. I don't know why I don't look for more opportunities to do this. Not only is it personally and emotionally liberating, it's often the "right" thing to do.

The features we pulled out really had no business being in there in the first place. Of course, we didn't know that until we released the first version, got feedback from folks and spent some time outside of the development process. Y'know, hindsight, 20/20, blah blah blah.

I think it is exceedingly difficult to get perspective on your work while you're creating it. There's just too much personal investment in the activity, especially if you get any sort of "flow" going. Creative types love "flow" and hate giving it up for any reason, even if the work suffers as a result. I think that's fine, but that's why there's a separate process known as "editing". The trick, of course, is to balance satisfying your possibly-self-indulgent creative urges with creating something of value.

Anyway, that's enough of my philosophical rambling. We also put together a little brochure site at http://evriverse.evri.com, including a screencast with music composed by yours truly. That process was a whitepaper-grade case study in "stuff you can do with iLife". I grabbed all the footage using iShow U, did all the music in Garage Band (on the bus using the computer keyboard no less) and glued it all together in iMovie. Super simple, super fun.

All in all I'm pretty pleased with the result. It's free, so check it out and let me (or Evri) know what you think.

In this post we'll look at three ways to allow your objects to communicate with each other in a highly-decoupled manner using the Cocoa frameworks. Cocoa provides three common ways of connecting objects for messaging: Notifications, Key-Value Observation and delegate pattern.

Notifications

Both Cocoa and Cocoa Touch provide a notifications framework built on two classes: NSNotificationCenter and NSNotification. The former contains a singleton instance, (via the defaultCenter method) with which observers register themselves and observables post notifications. The latter is the "envelope" for the notification which can contain an NSDictionary instance of customized data as its payload.

When an object registers for notifications, it specifies a notification to listen for (as a NSString) and, optionally, a target object. If the target is set to an object, only notifications posted by that object will be received. If the target object is set to nil, any object posting that notification will be sent to the receiver. Senders can optionally include a NSDictionary instance filled with domain-specific objects. You can use this as a way to further parameterize a notification, or eschew it altogether.

Consider the example below:

@implementation Foo
    - (void)init {
      if (self = [super init]) {
        [[NSNotificationCenter defaultCenter] addObserver:self
                                                 selector:@selector(didReceiveNote:)
                                                     name:@"TheBigNotification"
                                                   object:nil];
      }
      return self;
    }
    
    - (void)didReceiveNote:(NSNotification *)notification {
      NSLog(@"Hey, I got a notification with user info: %@", [notification userInfo]);
    }
    @end
    
    @implementation Bar
    - (void)doSomething {
      NSLog(@"I'm up to something!");
      NSDictionary *stuff = [NSDictionary dictionaryWithObjectsAndKeys:@"foo", @"bar", nil];
      [[NSNotificationCenter defaultCenter] postNotificationName:@"TheBigNotification"
                                                          object:nil
                                                        userInfo:stuff];
    }
    @end
    

Here the Foo class registers itself as a notification receiver for the notification named "TheBigNotification" on any object. The Bar class will post the same notification when the doSomething method is invoked.

We could have been much more specific about which object to observe, but I think this violates one of the key features of notifications which is that the sender and receiver are decoupled. To me if the receiver is going to go to the trouble of listening for notifications from a specific object, you'd be better off declaring a custom protocol and using the delegate pattern (explained below).

While at the 2009 WWDC, I had a chance to pick the brains of some Apple engineers and one of them told me that notifications are really intended for one-to-many broadcasts. I think that's a great rule of thumb, because the code can really feel like overkill for single object-to-object messages. However, I'd add one more clause to that rule which is that notifications work for one-to-one messages when you would otherwise have to pass instances around of delegates just to connect objects.

As an example, consider an iPhone application with a stack of view controllers in a UINavigationController instance. If you have a view controller several steps removed from another view controller (i.e. one is a further up the stack in a tab bar controller) and it needs to call some method back to the view controller lower down in the stack, each intermediate object would need a reference to the delegate class just to pass it down the line. This seems like the worst kind of encapsulation breakage since you're forcing a bunch of otherwise-unrelated classes to have knowledge of a class they don't even use.

Maybe it's my long track-record with Java, but I pay attention to the classes I import and I like to keep that list as short as possible. The more classes know about each other, the more difficult it is to modify them.

There's one final thing worth knowing about the NSNotificationCenter, and that is how it works with threads. When one object posts a notification, that call blocks while the notification is delivered to all targets. This means that you want your notification-handling code in the receiving object to be as quick as possible. If you can live with deferred notifications and want to avoid the synchronous nature of notification delivery, you can use the NSNotificationQueue instead.

Key-Value Observation

When I first read about, and used Key-Value Observation (KVO) it seemed like magic. You simply point one object at another and say "I'd like to know when this attribute changes" and Cocoa handles all of the notifications and threading automatically. Genius!

KVO also allows you to use a special syntax when specifying the attributes you want to observe so that you can register with an object and traverse its object graph. Let's look at the classic customer/orders/line items example. If you want to know when changes are made to any line items in an order you would observe changes in the order like so:

Order *order = [self anyOrder];
    [order addObserver:self
            forKeyPath:@"lineItems"
               options:NSKeyValueObservingOptionNew | NSKeyValueObservingOptionOld
               context:nil];
    

Whenever you register for KVO, you must implement the method - observeValueForKeyPath:ofObject:change:context:. Unlike notifications, you don't get to specify a selector. This means that if you're object is observing several other objects, you will have to inspect the object given in the callback method and dispatch appropriately.

Key paths have a ton of flexibility including the ability to traverse deep object graphs and observe aggregate functions on a collection (e.g. observe the max date of all of a customer's orders). Check out the Key-Value Programming Guide for details, which is an indispensable reference.

KVO works like notifications in that the thread that invokes the change will block until all observers have been notified and their callback methods have been invoked. Unlike notifications, there is no built-in asynchronous KVO-triggering mechanism unless you explicitly mutate properties in another thread using something like the detachNewThreadSelector:toTarget:withObject: class method of NSThread.

For classes to be key-value observable they must be, what the documentation calls, "key-value compliant". This generally means that each observable property exposes certain methods for accessing and mutating them. Again, the KVO guide is your go-to reference.

KVO is structurally similar to notifications, but is really intended for fine-grained messaging, generally around your model classes. Notifications are intended for broader application-level events. Both mechanisms give you a way to decouple classes that would otherwise need more intimate knowledge of one another.

Consider the simple case of managing a tabular view of line items in an order. When a user adds a line item, removes a line item or updates the quantity of a line item we want a field showing the total price to be updated. Without KVO we would need to add extra logic in our event-handing for the table view to also update the total field. With KVO our event-handling logic can simply focus on updating the underlying model. The total field will simply observe the total cost of the order and be updated appropriately. Now if we want to remove the total field or perhaps add another view of total information, we don't have to touch the original event-handing code.

Delegates

If you've spent any time with the Cocoa docs you'll run across The Delegate Pattern. This thing is like the quark—it is the fundamental building block of the Cocoa APIs. Conceptually, it's really pretty simple. A delegate is simply an object that provides custom behavior for another object, often by implementing a specific protocol.

Let's return to our running example. A table view of line items in an order has to handle a lot of things such as rendering each cell, handling scrolling, managing user-generated events and so on. These features are common enough that they are simply part of the table view class itself. What is specific to your application is the data that goes in those cells and the actions to be taken for user-generated events. Cocoa solves this by providing protocols for delegating that behavior to custom classes. In the case of providing data, a data-oriented delegate would be queried by the table view for the total number of rows, appropriate object to put in each row, the column headers, etc. In the case of event-handling, the table view will forward events to the event-handling delegate.

Notifications use the NSNotificationCenter (or by proxy, the NSNotificationQueue) as an intermediary between message-senders and message-receivers. KVO uses the observed object as the intermediary. In the delegate model there is no intermediate object, but you still have relatively decoupled code because the object that invokes methods on the delegate has no knowledge of what class of object it's dealing with. Put another way, you could remove all of the classes in your project that implement a particular delegate protocol, and the class that uses the delegate would still compile correctly.

The delegate pattern isn't so much about decoupling which objects are communicating with each other, as much as decoupling the classes of those objects. This doesn't make it any less powerful than the other two methods. In a lot of cases a simple delegate model is much more straightforward and preferable the alternatives.

The delegate pattern is Cocoa's version of dependency injection. The object receiving the injected dependency has no idea of the type (and by extension, the implementation) of the object it is receiving, only what it's capabilities are via a contract specified as a protocol. This is basic polymorphism where the interface and the implementation are separated to reduce coupling.

Delegating is a popular alternative to sub-classing. It is a preferred alternative because it's less likely to break encapsulation. If customization is done through subclassing it can be difficult to know which methods to override and implement, and easy to break the parent class. This is why Java provides all sorts of safety features like marking methods abstract and final. However that is simply more of a headache to deal with and using polymorphism would be a much better design.

So there you have it, three ways to keep your code flexible and loosely-coupled!

Resources

• Apple's Notification Programming Topics
• Apple's Key-Value Coding Programming Guide
• Apple's Cocoa Fundamentals
• Mike Ash's Well-Reasoned Critique of KVO Design

We live in a very interesting time for application development. The distinctions between desktop, browser and mobile applications are blurring more and more every day. "Ubiquitous platform" applications, like Evernote, are where the future is because it reduces the major hurdle of access for users. As time goes on users are going to expect applications to be available in a bunch of different ways that all need to work together.

The common way to do this is base the application on shared-state accessed via an HTTP-based API. This makes it relatively easy to create apps for various devices as well as offer up a public API for others to use. An interesting side-effect of this is that the API, the shared-state and data behind it become the differentiator between products. The end-user client software is simply the packaging, turning the stand-alone application market into something more akin to the "razors and razor-blades model".

Okay, none of this should be earth-shattering news to anyone. You should be prepared to write more apps that are integrated with one (or possibly more) "backend services". These services will, as likely as not, be connected over HTTP. As an iPhone developer you should know how to do this in your sleep.

The Evri API is absolutely crucial to the EvriVerse phone application. Without it, the app does nothing of interest on its own. Our iPhone application is merely one kind of presentation of our web APIs. I don't think that's unique. What good would Tweetie be without Twitter?

Dealing with Web APIs

An unfortunate fact of integrating with web APIs is that they are, relatively speaking, slow. We don't want the user to get frustrated and give up on our app because they have their flow continually disrupted by waiting for data.

There are some things you can do to mitigate the latency costs, if you control the API such as using some sensible caching parameters. However, especially if you don't control the web API, the bulk of what you should focus on is concurrency within your application. The biggest arrow in your quiver is figuring out how to turn unnecessarily serial operations into parallel ones. Now keep in mind that the iPhone doesn't support true multi-core concurrency. What I'm talking about is taking advantage of the I/O wait times that are inherently part of working with a high-latency I/O source (the web).

Coming from a Java background, using threads seemed like a pretty natural approach and after reading the docs on the NSThreadclass I felt pretty confident I knew how to make it work. However, I was surprised by how quickly the code complicated. There is a lot to like about Objective-C, but I sorely miss Ruby's blocks and even Java's inner-classes. In Objective-C you connect methods together dynamically by handing objects selectors. The problem with this is that it's difficult to tell the high-level methods from the low-level callback methods just by looking at your class definition. Yes, you can use special comments and the like, but if you've worked in a language that has structural support for this differentiation, moving to a language without it feels like losing an arm.

So instead of managing your own threads, let Cocoa Touch do the work for you. The NSURLConnection class is inherently asynchronous—you can fire a request and immediately move one with your life. Now you just need a little structure for handling the response and getting it back into your views.

The EvriApi

We tried to push as much of the mechanics of URL-handling into a separate code-base we call EvriApi. A good bit of the design was inspired by Matt Gemmell's MGTwitterEngine, so I can't claim that I came up with the whole idea myself.

We wanted the API to be as simple to work with as possible. Each API call has the same structure: * Each API method takes zero-or-more domain-specific parameters, a selector and a target object. * Each API method returns a unique request identifier * The target object will have the given selector called when the request completes, and will be given an instance of the EvriAPIResponse class. * If the response is successful (check [response success]), the payload of the request is retrieved as a domain-object * If the response is unsuccessful, check the error message on the response * Requests can be cancelled by handing the request identifier back to the API * Attempts to cancel already-completed requests do nothing

The NSURLConnection class makes asynchronous requests by default, and uses the familiar delegate model for handling the response. When a client invokes an API method, it returns immediately with a unique identifier (as a NSString instance) and the request is initiated on another thread which is handled automatically by NSURLConnection. The client is responsible for keeping track of that and using it to cancel any outstanding requests.

When the request completes, the body of the response is parsed using the NSXMLParser class. The parser is given one of our specialized XML handlers as a delegate. These handlers deal with the XML event-stream and produce a model object or collection of model objects. The framework then packages this up into an EvriAPIResponse instance and invokes the selector given in the original request on the target given in the original request.

Wiring it Up

The EvriVerse app is primarily a "productivity" style application. In Interface Guidelines-parlance this means that we use a lot of hierarchical navigation. In general we have no more than one or two API calls associated with a particular view controller.

The common pattern for handling a user-initiated action that requires an API call is:

1. A new view-controller is instantiated and displayed, typically with an activity indicator visible
2. User-interaction is disabled on the new view controller
3. The new view controller is displayed (usually by pushing it on the navigation controller stack).
4. A request is submitted to the EvriApi, setting the new view controller as the target and some special method as the target selector.
5. The request ID from the EvriApi is given to the newly-created view-controller
6. When the API request finishes, the handler method on the new view-controller executes, handling both success and failure cases. It also re-enabled user interaction on the view.
7. If the user navigates back before the request has completed, any outstanding requests are cancelled. This is usually handled in the viewWillDisapper:(BOOL)animated method of the view controller.

Stepping back, the actors and interactions look something like this:

In code (snippets) it might look something like this:

    // our parent view controller
        @implementation ParentViewController
        - (void)didSelectEntity {
          ChildViewController *cvc = [[[ChildViewController alloc] init] autorelease];
        
          [self.navigationController pushController:cvc animated:NO];
        
          NSString *requestId = [EvriApi fetchEntityByURI:entity.uri
                                          performSelector:@(didReceiveEntityResponse:)
                                                 onTarget:cvc];
        }
        @end
    
    
    
        @interface ChildViewController : UITableViewController {
          NSString *requestId;
        }
        
        @property (nonatomic, retain) NSString *requestId;
        @end
        
        @implementation ChildViewController
        
        @synthesize requestId;
        
        - (void)didReceiveEntityResponse:(EvriAPIResponse *)response {
          // TODO: hide the activity indicator
          // TODO: re-enable user interaction
        
          if ([response success]) {
            Entity *e = (Entity *)[success result];
             // TODO: redisplay as necessary
          }
          else {
            // TODO: show an alert
          }
        }
        
        - (void)viewWillDisappear {
          [EvriApi cancelRequest:requestId];
        }
        
        @end
    

So there you have it, one developer's way of integrating the iPhone with web APIs. More to come. Stay tuned.

Ever since I saw Laurent's and Rich Kilmer's presentations about MacRuby and HotCocoa at RubyConf '08, I've been jazzed about what MacRuby brings to the world of Cocoa development. So check out the screencast for the coolest new Ruby on the block.

I wanted to kick this off with a little appetizer. We're going to look at how we implemented incremental find in EvriVerse. Evri has a huge database of structured entity data (people, places and things) and our users want to be able to search for any of them. After putting the initial prototype for EvriVerse on a few folks' phones, a lot of them wanted a search feature that worked more like our website.

In the original implementation a user would type in the search field, then hit the "Search" button and then get their results back in the table view. This takes too many steps, and we wanted something that felt faster and more responsive.

So our incremental find solution needed the following properties:

• It should only send a search request when there is a measurable pause in input
• It should enable some kind of activity indicator while a search request is running
• A user should be able to modify the contents of the search field while a search request is in-progress
• It should not preclude the existing two-step search process
• If a search request is in process when another one starts, the first one should be cancelled and the new one should be the only request in-process

At first I tried managing my own NSThread instance while tracking the timestamps of touch events in the searchBar:textDidChange: method (part of the UISearchBarDelegate protocol). If the the gap between time samples was big enough I'd cancel any outstanding request and start a new one. This seemed like a good approach at the whiteboard, but really fell apart in the implementation. I had race-conditions left and right and I had the sneaking suspicion that I was swimming upstream against "the Cocoa way".

As an aside, if there's one thing I've learned in my brief time with Cocoa, it's that mucking about with your own threads is a pretty crummy way to do things asynchronously. Usually there's a much more elegant, built-in solution in the Cocoa APIs that will handle the threading for you. I'll riff on this theme in a later post.

So back to the problem at hand. What to do? When I get stuck like this I often start thinking of how I'd do this in another language. This got me to thinking about how one would go about implementing this sort of thing in Javascript. This is standard Web-2.0 kind of stuff—right after drop-shadows and rounded corners, you learn about type-ahead search at AJAX School.

See, Javascript has this very nifty little function named setTimeout where you hand it a function and tell it how long you want to wait until that function is executed, and it returns you a reference identifier. There is a companion function, clearTimeout that allows you to cancel any previously-created "timeout" function by handing it the reference identifier you got earlier from setTimeout.

In the common AJAX-ified dynamic search that you often see, an event-handler observes the search field in a web form for key presses. Each key press cancels any existing timeout function, and starts a new one that will execute in whatever time-interval was chosen as the "quiescence period". The trick is that because you're scheduling future tasks, and you cancel any outstanding ones when you kick off a new one, you get the "measurable pause" feature in a simple, elegant solution.

Cocoa provides a very similar mechanism, through the NSTimer class. In my view-controller, the code ended up looking something like this:

@implementation FindViewController
    // much implementation code has been elided for demonstration purposes
    
    // observe text-field changes and fire the scheduled task
    - (void)searchBar:(UISearchBar *)searchBar textDidChange:(NSString *)searchText {
      if ([searchText length] > 2) {
        if (timer) {
          [timer invalidate];
          self.timer = nil;
        }
        NSDictionary *info = [NSDictionary dictionaryWithObjectsAndKeys:searchText, 
                              @"Text", 
                              nil];
        self.timer = [NSTimer scheduledTimerWithTimeInterval:1
                                                      target:self
                                                    selector:@selector(submitSearchRequest:)
                                                    userInfo:info
                                                     repeats:NO];
      }
    }
    
    // The method that runs in the NSTimer--equivalent to the function 
    // we'd pass to Javascript's "setTimeout" function
    - (void)submitSearchRequest:(NSTimer *)aTimer {
      NSDictionary *info = [aTimer userInfo];
      NSString *text = [info objectForKey:@"Text"];
      [activityIndicator startAnimating];
      self.requestId = [EvriApi fetchEntitiesMatchingPrefix:text
                                            performSelector:@selector(didReceiveEntitiesDynamically:)
                                                   onTarget:self];
    }
    
    // The callback from our API class, included for completeness
    - (void)didReceiveEntitiesDynamically:(EvriApiResponse *)response {
      [timer invalidate];
      self.timer = nil;
      [activityIndicator stopAnimating];
      if ([response success]) {
        [entities removeAllObjects];
        [entities addObjectsFromArray:(NSArray *)[response responseObject]];
        [tableView reloadData];
      }
      else {
        UIAlertView *error = [[UIAlertView alloc] 
                              initWithTitle:@"Uh oh…"
                              message:@"Sorry, we were unable to execute your request. Please try again.
    "
                              delegate:nil
                              cancelButtonTitle:@"OK"
                              otherButtonTitles:nil];
        [error show];
        [error release];
      }
    }
    
    @end
    

In the searchBar:textDidChange: method we check to see if our instance field, timer, points at an existing NSTimer instance. If it does we cancel it, clear the reference, and start a new timer instance. That timer is configured to run in one second and will execute our submitSearchRequest: method. This makes a call to our API and designates the didReceiveEntitiesDynamically: method as the callback for that asynchronous operation.

Finally, the didReceivedEntitiesDynamically: method handles the UI by disabling the wait indicator, dealing with any error cases and populating our table view if our request was successful. I'll write another post later about how the EvriApi class works and its design.

So that's it. Simple and elegant. One of the things I love about working on several different programming languages is figuring out how to apply the best tricks of one language to another.

That's all for now, but stay tuned! More iPhone posts to come…

After several months of work, I'm pleased to announce the release of Evri's EvriVerse for the iPhone. We've tried to capture the unique things we do at Evri into the awesomeness that is the iPhone. I won't spill all the details here, so check out out the "official" company post.

I've been working on this thing for a few months, mostly on a part-time basis. We have a 20% time program at Evri where devs can pitch ideas they want to spend one day a week on. I was dying to write an iPhone app and jumped at the opportunity to turn it into a 20% project. After some early prototypes I got enough attention that I was given a solid two-week sprint to finish it off.

This was a ton of fun to work on and I have five or six posts on some things I discovered while working on it.

It's free in the App Store now. We'd love to hear any feedback you may have, which you can post on the Evri Support Page.

For most of us at work, our primary Git experience is with GitHub. This is a good way to learn how Git's distributed model works, but GitHub provides some nice infrastructure that you simply don't have with a stock Git install. You have to cobble-together the rest from a variety of sources. So GitHub guys—if you're listening—it would be fantastic if you provided real white-label GitHub solution for organizations to install locally. You guys have really spoiled us. (oops, nevermind. Checkout GitHub:fi). But until then, we have to roll our own…

The Goals

First, let's review what exactly we were trying to accomplish. We wanted a solution that: * easily converted existing Subversion repositories to Git * provided a usable web view over all repos that was equal to, or exceeded, the stock SVN/DAV model * allowed us to designate a set of "authoritative" repositories * provided for "developer" repositories to allow developers to publish their changes * allowed for patch reviews (if teams decide to do that)

The Solution

Right away we knew that we wanted two different types of repositories: authoritative and developer. Authoritative repos are ones that host the "official" source code and from which we create release builds from. When a developer wants to start working on a project, these are the repos they clone to get started.

The other type of repository belongs to an individual developer. Developers have read/write access to their own repos, but others only have read-only access. This allows developers to share changes in a pull-request model similar to how GitHub works.

Permissions & Connectivity

Developers connect to these repositories in two different ways. We use the built-in git:// protocol (with git-daemon) for read-only access and ssh:// for read-write access. Developers clone authoritative repos and other developers' repos using the read-only git:// protocol. Developers have a remote reference to their developer repository using ssh.

For each project, we designated one or more maintainers that have write-access to the authoritative repositories. These folks accept patches from other developers, test and integrate them on their local machines, and push final changes to the authoritative repositories. We manage the authoritative repos using gitosis. Check out this great tutorial for details on setting up gitosis.

gitosis

On our central "git box" we create a "git" user which owns both the gitosis-admin repository as well as all the authoritative repositories. Gitosis is bootstrapped with a single SSH public key that gives the associated user the ability to write to that Git repo over SSH. That user can then add more keys and configuration allowing other users to manage projects. Gitosis works a bit like the CVSROOT project in CVS—it is a special Git repo that allows us to configure other Git repositories. We use gitosis to configure our designated maintainers for each project.

Our authoritative repos are housed in /home/git/repositories. Our developer repositories are located in /opt/repos, where each developer has their own directory to put Git repos in.

git daemon

We run git daemon to allow for anonymous read-only access. We configured git daemon to serve up all Git repositories found in the /opt/repos directory with read-only access. We also symlink the authoritative repositories (/home/git/repositories) into this directory so that we can serve up read-only access to authoritative repositories also.

gitweb

For all the power Git has, there sure are a lot of half-baked web interfaces for it. We did our best to vet the tools listed on the Git site. A good number of the them were either abandoned or simply didn't work. In the end we went with the venerable old gitweb.

Gitweb has a ton of functionality—including all kinds of search capability, different views (by tag, by commit, by tree) and even serves up an Atom feed of changes—but it is definitely a designed-by-developers-for-developers. (NB: In case you missed the markup, that is considered an epithet on my team).

Frankly I was amazed that somebody hadn't built a Rails app to manage this stuff. Yes, we looked at gitorious and even tried to set it up. After three hours I still didn't have a working installation and my patience had run out. While gitorious appears to be a nice turn-key solution, I don't think it's actually that great of a fit for us. So gitweb it is. Sigh.

The Flow

With me so far? Maybe not. Let's look at a picture then…

In this picture the authoritative and developer repos are drawn in separate boxes (since they're logically separate), but they are located on the same machine in reality. Let's cover a couple of common scenarios:

New Developer, Old Project

A developer starts by cloning an authoritative repo using the git:// protocol (remember, read-only). This will create, by default, a remote reference named origin that points back to the authoritative repo.

The next step is for them to create a developer repo in their directory for the same project on the central git machine. They also want to a remote reference to their developer repo so that they can push changes to that. That would looks something like:

git remote add alex ssh://git/opt/repos/alex/circus/clown-car
    

If this user is also a maintainer, they need to add a third remote reference which gives them read/write access to the authoritative repository. The way that gitosis works is by accepting SSH keys on behalf of the git user. So if you're properly configured, you can push changes over SSH by logging in as the git user. A maintainer would add a read/write reference like so:

git remote add main ssh://git@git/home/git/repositories/circus/clown-car
    

Because of the way gitosis configures the SSH keys in /home/git/.ssh/authorized_keys and the fact that the default protocol is ssh, this remote reference could also be added like this:

git remote add main git@git:circus/clown-car
    

Now because there are number of steps involved here, we wrote a little command-line tool (as a RubyGem) that takes care of these steps all in one go. Right now, it's very specific to our setup at Evri, but I can see how we might extract a common tool (oh boy…another side-project…)

Old Developer, New Project

When it's time to create a new project, a developer usually starts out by creating a new Git repo on their local box while they're building out the initial version. Once it's time to share, that developer can create a new developer repo on our central Git machine just by SSH'ing in and making the appropriate directory, adding the remote ref to their local repo and pushing changes. Again, our internal tools sets this up all in one go.

Once that project is ready to have an authoritative repo, a gitosis-enabled user will pull the latest changes for the gitosis-admin project. They'll edit the gitosis.conf file to add the new project (and members) and commit the changes. Then the developer adds a new remote ref (a read/write one as the git user) and pushes changes out to the main repo.

Sharing Patches

The most common flow is when developers share patches with each other. There are a couple of ways to do it. Developers keep their developer repository up-to-date and email requests to their teammates to pull changes from their repo (a la GitHub). Developers may also choose to share patches via email using git format-patch, git send-mail and git am.

Ultimately the maintainers are responsible for gathering patches from developers and integrating them back into the authoritative repositories.

For folks used to version control systems like CVS or Subversion this feels like an awful lot of hoop-jumping. One part that I can't diagram or explain as a series of technical bullet-points is the stewardship the pro-Git folks have to take on. People who are switching to Git get frustrated by the byzantine nomenclature and steep learning curve, so a big part of the change and setup is simply helping people out when they get stuck.

While doing Java development, I've often used Eclipse's file history feature to roll things back. The downside is that it's file-by-file and not easy to tag the current state of your entire project in one go. It appears that Xcode has this feature, but since I'm using Git for my SCM, I figured why not use that instead?

So my new flow has been to stage changes to the index whenever I make any progress and the app is still in a working state. This is different from a commit, which I still like to think of as a succinct, whole change around a particular feature. The incremental staging is more like dribbling micro-changes to the next stage prior to committing. It can take several attempts to get to a real, first-class commit.

I like using magit, so I keep Emacs running in the background. When I get to a good checkpoint, I stage hunks or entire files to the index. When I get enough to make a full commit, I commit them. If you're running git exclusively on the command-line, this would be the equivalent of using git add -i.

When I first started using Git, I couldn't see the value in Git's stage-commit-push workflow. Now I get it. Like most Git tricks, this one is probably obvious to a lot of folks, but for me it's been a real life-saver on this project. I've been able to be much more cavalier with experimentation because I can easily revert changes with a single keystroke. Nifty!

As anyone who has read this blog recently knows, the luster is coming off of Java for me in a big way. These days my goals are simply to co-exist with it in a way that keeps me happy. I'm not going to be able to toss Java overboard so my working-life becomes a question of how to be happy with the situation.

While our home-grown framework has worked well for us, I'd much rather see us using something with wider adoption and use. So I took a look at Jersey and came away pretty impressed. Looking at how the API is built and what you need to do to build REST-ful web services in Java I couldn't help but feel like Java has learned some lessons from the rest of the world.

The original Servlet API is what could be termed a "classic" Java API, wherein consistency and type-safety are the rules. In the Servlet API, you interact with requests and responses as monolithic objects. Each type of HTTP method (i.e. GET, POST, PUT and DELETE) has their own method which takes a ServletRequest and a ServletResponse object.

It's a reasonable first solution, but it breaks down quickly. The primary issue is that by packing all possible request-related data into a single request object, and all possible response-related data into a single response object the classes themselves start to feel bloated. A more sinister second-order effect is that these classes, especially the response object, have some hidden state and "gotchas" because they are so general.

As an example, consider the case where you want to return a response with a non-200 response code, an entity body and some headers. The order in which you set these on the response object is important because of how it's implemented. You need to set the status and headers before you write any output to the stream. This make sense because you want a stream-oriented interface which means not storing the entire response in memory and then flushing. But, due to the strict order of HTTP response messages, you need to flush the header information prior to sending the body. It's not a particularly difficult thing to remember, but it is unnecessary mental overhead that is an artifact of the implementation.

The request object is its own special brand of fun to deal with. Again its broad coverage makes for a rather clunky API to deal with. You want request parameters? You have to grovel through String arrays if you want to capture all of them. When implementing a Servlet, often what you want is some combination of request parameters, cookies and header; rarely do you need all of them at once. However what you get is one big über-object that has everything. Enjoy!

One final beef with the ServletRequest class is that getting path parameters out is an absolute nightmare. If you're building REST resources you really really care about the path as it is the way to identify resources. The poor support the ServletRequest class provides for this is simply shocking. Here's a String — you parse it and figure out what the hell the segments are.

In contrast, Jersey has a much looser philosophy with how requests and responses are handled. First, the monolithic request and response objects go away. Instead your methods provide the narrowest possible interface, expressing only what they need in exacting terms. This is done by making extensive use of Java annotations to mark up simple method parameters. For example, if you have a resource that needs a request parameter, use the @QueryParam annotation. Need a header? Just use @HeaderParam. Interested in cookies? Use the @CookieParam annotation. Here's an example:

public MyResult getMyResult(@QueryParam("name") String name) {
        ...
    }
    

This does away with a tremendous amount of "busy-work". You want the "name" parameter? By god you're going to get it—no intermediate objects to reach into and pull stuff out of.

A really nice side-effect of this is that writing tests for your resources become so much cleaner than using the more general Servlet API. Instead of setting up the monolithic request and response objects, you simply pass in the bits you want.

What about the response side? I think that it's in this area that we really see a fundamental philosophical shift emerge. In an earlier time the textbook answer to how to design an API like this would be to have each request method return some kind of superclass or interface. This would allow us to use polymorphism to vary the response, but keep our type-safety.

Jersey takes a different approach based on real-world needs. Whereas the Servlet API is intended to keep everyone equally happy by making sure everyone suffers the same amount of pain, Jersey lets you vary what you return—no special markers, no special configuration. If you have a simple case where you want to serialize an object as the entity response, just return the object.

What if you need to fiddle with the response some more? Maybe set some headers or alter the status code? In this case your method returns a Response instance. Now this may sound monolithic and perhaps, under the covers, it is. What saves it from degenerating into the Servlet API is the fact that you build a Response object with only as much as you need.

In the Servlet API you might have to set a moderately complicated response like this:

public void doGet(HttpServletRequest req, HttpServletResponse resp) {
      // do some stuff
      resp.addHeader("Expires", computeExpires());
      resp.setStatus(201);
      writeResponse(resp.getWriter(), new MyStuff());
    }
    
    private void writeResponse(Writer writer, MyStuff stuff) {
      // do whatever you have to do to serialize your object
    }
    

In Jersey it looks like this:

public Response getMyStuff() {
      MyStuff stuff = new MyStuff();
      return Response.created(stuff).expires(computeExpires()).build();
    }
    

The amount of code probably comes out to be the same, but the fact that you can build the response in a single line feels really good to me. The amount of vertical space dedicated to response-building is much more proportional to it's conceptual space in the method in Jersey than the Servlet API.

Now I realize that comparing Jersey to the Servlet API is perhaps unfair. The Servlet API is really designed to operate a layer below where Jersey plays. But I think the same arguments apply to a majority of the other popular web frameworks. The models are all essentially the same.

So here's where the philosophical sea-change comes in. To vary the response type to do the right thing means that after a decade of existence, people are finally embracing Java's reflection capabilities. Up until pretty recently this was considered the domain of the lunatic fringe.

The primary argument against reflection has always been that it's too slow. While it's true that their can be a penalty under certain circumstances, the modern perspective is that it's an acceptable price to pay for a kinder, gentler API. Simply put, this level of dynamism is essential for reducing the amount of boilerplate and busywork.

I think if other frameworks like Rails, Django and Merb hadn't gained so much traction in the last few years Java would have continued on it merry way. However I think the popularity of those alternatives has forced the Java greybeards to learn a thing or two. Strict static type-safety can become a burden and using reflection to vary behavior instead of polymorphism can make for some very clean APIs.

After spending a few days with Jersey I find myself a little re-energized to work with Java again. We're even considering Groovy as a way to cut down Java's chubby syntax for another small productivity win. Of course, my personal preference would be to do this JRuby since I, personally, don't find Groovy to be terribly compelling on its own. But since Jersey is so dependent on annotations, using JRuby is a non-starter.

Who knows? This could be a complete fiasco and another frustrated attempt at tilting windmills. I'm already a little suspicious of the "enterprisey" way this thing gets configured. However if nothing else, it's refreshing to see some new ideas make their way into Java after all these years.

Now you may ask yourself, "for the love of God, why would you want to do something like this?" Well, at work we do a lot of internal web services. For my particular team, we've found a real sweet-spot by using an embedded Jetty server sitting right next to a BDB instance. There are no extra processes or packages to manage (e.g. apache or a RDBMS). However we were becoming dissatisfied with our current web layer which is a homegrown REST framework that sits on top of the Servlet API. So in a fit of rage, I decided to see if I could stuff Merb in the middle of this mess.

You may also be asking yourself, "why not use the jruby-rack gem directly?" The answer is that the jruby-rack gem makes a lot of assumptions about how you want to run your application. First it assumes that you're cool with packaging things up as a WAR (which I'm not) and, secondly, that your application is primarily a Rails/Merb application. In my case, for better or worse, our app is really a BDB application with a Merb app glommed onto the side for web visibility.

The Solution

I can't take complete credit for this solution. If I hadn't found Jan Berkel's post on putting Rails in Jetty I would have never figured out how to stuff Merb in there. To give yourself some context, take a look at that post first. Then take a look at the "Merb-ified" version of the same recipe below. Both solutions assume that you're configuring Jetty within JRuby.

server = org.mortbay.jetty.Server.new
    thread_pool = org.mortbay.thread.QueuedThreadPool.new
    thread_pool.min_threads  = 5  # adjust as needed
    thread_pool.max_threads  = 50
    server.set_thread_pool(thread_pool)
    context = Context.new(nil, "/", Context::NO_SESSIONS)
    context.add_filter("org.jruby.rack.RackFilter", "/*", Handler::DEFAULT)
    context.set_resource_base(Environment.resolve)
    context.add_event_listener(MerbServletContextListener.new)
    context.set_init_params(java.util.HashMap.new('merb.root'=>; Environment.resolve,
        'merb.environment' => 'production',
        'public.root' => Environment.resolve('public'),
        'gem.path' => Environment.resolve('gems'),
        'org.mortbay.jetty.servlet.Default.relativeResourceBase' => '/public',
        'jruby.max.runtimes' => '1'))
    context.add_servlet(ServletHolder.new(DefaultServlet.new), "/")
    server.set_handler(context)
    server.start
    

Tweaking

At first blush our performance seemed to be pretty lacking. This required two tweaks: putting Merb in "production" mode and dealing with poor I/O due to logging. In the previous snippet you will notice that we set the merb.environment to production. Yes we lose the quick dev turnaround, but since there is a lot of Java in this project we usually have to recompile anyway which requires a restart anyway (phooey).

As for the I/O issue, a little digging revealed that shutting up Merb as much as possible would help reduce the amount of JRuby-level IO. In our config/init.rb we configure logging like so:

Merb::Config.use { |c|
      c[:environment]         = 'production',
      c[:framework]           = {},
      c[:log_level]           = :warn,
      c[:log_file]            = Merb.root / "logs" / "merb.log",
      c[:use_mutex]           = false,
      c[:session_store]       = 'cookie',
      c[:session_id_key]      = '_facet-store_session_id',
      c[:session_secret_key]  = '49411912879b879e13f89a9280c0f6aaa2e3ab58',
      c[:exception_details]   = true,
      c[:reload_classes]      = false,
      c[:reload_templates]    = false
    }
    

Here we set the environment to "production" again (yes, you need to do both). Also we upped the log level to "warn" which significantly reduced the amount of logging merb does. With these tweaks in place we found that the Merb port of our service was operating within about 80% of the level of performance we were getting from our pure-Java solution.

Benchmarking was done by running httperf tests against the resources we expose and comparing both the number of requests per second and the average response time. Given that the options for generating XML, HTML and JSON were all so much easier than what we were doing in the servlet version, we were willing to live with the performance hit.

That seemed like a great plan until I realized just how tangled some of the patches were. It isn't difficult for this to happen. As you extend the functionality of a system, you often refine earlier work. This was especially true in my case since we were introducing JRuby to a previously Java-only project and so there was a lot of experimentation and refinement. What I was really trying to do was re-write my commit history to separate the changes. Small cleanup commits could be collapsed with others to make the entire commit-set something that my teammates could easily understand.

Getting Started

The most basic kind of re-writing you can do is simply amending your last commit. On the command line this is accomplished with git commit --amend. The bigger rewrite-hammer is git rebase -i . This command will pop off all of your commits to the point specified by , provide you with a control file to edit and then re-apply your patches as directed.

The control file (a term I just made up) looks kinda like this:

pick 1cea777 Initial introduction of Merb.
    pick 5fe7a19 Favor XML over HTML as the "default" content-type.
    pick 6791134 Cleaned up the 'views' directory, out with the old, in with the new.
    pick ae6adac Put lots of back-navigation links to make it nice 'n' easy to use.
    pick 1da31ca Removed last vestiges of the RestServlet-related configuration and code.
    pick b9fe3b9 Added error views, updated error-handling and improvised content dispatch.
    pick bf92292 Routing cleanup. This is much more pleasant to read.
    pick ec74c63 An attempt to get RSpec working with Maven and JRuby.
    
    # Rebase 92ddc87..ec74c63 onto 92ddc87
    #
    # Commands:
    #  p, pick = use commit
    #  e, edit = use commit, but stop for amending
    #  s, squash = use commit, but meld into previous commit
    #
    # If you remove a line here THAT COMMIT WILL BE LOST.
    # However, if you remove everything, the rebase will be aborted.
    

The top section lists all of your patches, one per line. You can edit this section to do any of the following: * Reorder the commits (just move lines up or down) * Edit a commit (replace "pick" with "edit" or simply "e") * Remove a commit (just remove the line) * Merge with a previous commit (replace "pick" with "squash" or "s")

Think of this list like a program of execution. Once you save the file and return control to git, it will attempt to execute this program. I say attempt here because sometimes git runs into conflicts when it comes to re-ordering patches.

Editing Commits

When you mark a commit for editing (with "edit" or "e") git will attempt to apply all patches up to, and including that commit, and stop. This threw me at first because for some reason I had the unreasonable expectation that somehow the changes in the last commit would only be applied to the working tree and, perhaps, the index. Instead, that commit is fully applied, but all commits past that are pending. To add to my confusion, there isn't an easily accessible marker to indicate that you are in the middle of rebase (unless you frequently scan the .git directory as a matter of habit). At this point you could amend the commit (with git commit --amend) or insert other commits.

Sometimes I use this just to fix up the commit message. When I'm working on a new feature that has a lot of trial-and-error I tend to mark the first commit message with "WIP" to remind myself to review that patch and the ones following to see if I can clean them up. This is an area where git really shines. In a system like Subversion your audience (other coders who look at your changes) end up walking through whatever little mini-epic you've composed as you try things out, add things and remove things. This makes for some difficult reading for consumers of those patches and so a lot of folks tend to stack up big changes and then send in the big über-commit.

The problem with building the mega-patch is that you don't have a lot of scaffolding under you while you are building it. If you go off and explore something that doesn't turn out right, you generally have to do a lot of manual reverting. This simply sucks and is a waste of time. With git I can work in lots of incremental commits, then go back and edit them into something sensible once I'm ready to push changes.

Once you've finished doing whatever edit you want to do for that commit, simply type git rebase --continue and the remainder of the "script" will execute. If any other "edits" are in the pipeline, the process will repeat itself until the rebase is completed.

Squashing Commits

I looooove squashing commits in git. For any moderately complicated work, it's rare that I get it right the first time. Usually as I go along I find some mistake I made or find a refinement that cleans up the original work. As often as not, it's usually a couple of commits away so amending the last commit isn't going to help me. But hey, no worries! I simply commit the change and leave a log message for myself to merge it with another commit. Then, when I run the interactive rebase, I can simply move this commit up the list and change it from "pick" to "squash" (or "s") and let git merge the two commits.

When rebasing encounters a "squash" it will apply the changes in both commits. If successful, git will prompt you with a commit message file that includes the original commit messages of both commits. You can choose either, both or neither of these and save the file to continue.

Resolving Conflicts

Sometimes when applying a commit, git will run into conflicts and bail out on a merge. Whenever this happens to me I always have a little moment of panic as if I've broken something, but fear not—you can fix this. When this happens git tries to stage as many of the changes as it can. Any conflicts are left unstaged and need to be edited (look for the standard conflict markers), and then re-staged into the index. When you commit, git should show you the original commit message of the patch it was attempting to apply, which you can edit or keep. Once the commit succeeds, rebasing should automatically continue.

The Big Red Button

Sometimes you may just give up on a rebase. Either you can't commit the time to it, you don't really want to go through with it, or your patience has reached its limit. At any point, before the rebase has completed, you can execute git rebase --abort and your working tree, index and commit-log will all be returned to the state prior to starting the rebase. Think of this as the big red "stop" button for rebasing.

Playing Fast and Loose with Branches

If you come from other SCM systems like Subversion, you tend to treat branches as expensive and something you only use on occasion. With git, branches are cheap and easy like drinks in Tijuana. Be fearless! Not sure about some exploration? Make a branch!

If rebasing makes you nervous, and you're not entirely confident that aborting will save you, I suggest creating a branch on which to rebase. Simply create a new branch from where you're at with git checkout -b . You'll be immediately switched to that branch with your commit log looking exactly like the branch you came from. Here you can rebase, edit, remove, add and generally go crazy with your commits.

Once you've got your commits where you want them it's simply a matter of getting them back into whatever you consider your "main" branch to be and pushing any changes upstream. This works really well when you have rendez-vous point like a Github repo or an SVN server (we do a lot of git on top of Subversion at work). Assuming that you've been adding changes on your master branch, you might create a new branch off of master that you might call exp. On the exp branch you might rebase and wreak all sorts of havoc on the commit log. Once you have your commits where you want them, switch back to your master branch and reset it to point to the head of your upstream repo. If you're using git all the way, simply merge your exp branch over. If you're running git on top of Subversion, simply cherry-pick the commits on the exp branch (in order!!!) and push the changes upstream.

It's hard to stress how important it is to shift your mindset into thinking in terms of sharing patches. Treat your commits like individual, digestible changes. Even if you're working by yourself or in a small team, I still think there's value in being disciplined with your commits.

Instead, this is just a humble little notice about a humble little gem I put together today. It's called daemon-spawn and despite its simply terrifying name, it's really here to help all mankind. You see, I've been working like mad to stuff Merb smack-dab in the middle of an embedded Jetty project I've been working on. One of the last things I needed was a decent daemon-launcher/management gem-thingie to make it happen.

I cast about for an existing solution and found each a little lacking. The daemons gem had the executable name hard-wired to the output log name and didn't give me a clean way to specify additional arguments to JRuby (unless I wrote another wrapper script, to which I say "boo, hiss"). Then I looked at simple-daemon which seemed really promising. It was really really close to what I wanted but didn't extend very well as it required more and more class-methods. Yuck. I looked at daemon_generator, but it was very Rails-y and wanted to generate a bunch of code for me, which I didn't need. So I did what any honest, hard-working Ruby-dork does, and made my own!

It's simple—dead simple. Wanna see how simple? Here's a real-live echo server with daemon support:

#!/usr/bin/env ruby
    
    require 'daemon-spawn'
    require 'socket'
    
    class EchoServer << DaemonSpawn::Base
    
      attr_accessor :server_socket
    
      def start(args)
        port = args.empty? ? 0 : args.first.to_i
        self.server_socket = TCPServer.new('127.0.0.1', port)
        port = self.server_socket.addr[1]
        puts "EchoServer started on port #{port}"
        loop do
          begin
            client = self.server_socket.accept
            while str = client.gets
              client.write(str)
            end
          rescue Errno::ECONNRESET => e
            STDERR.puts "Client reset connection"
          end
        end
      end
    
      def stop
        puts "Stopping EchoServer..."
        self.server_socket.close if self.server_socket
      end
    end
    
    EchoServer.spawn!(:working_dir => File.join(File.dirname(__FILE__), '..'),
                      :log_file => '/tmp/echo_server.log',
                      :pid_file => '/tmp/echo_server.pid',
                      :sync_log => true,
                      :singleton => true)
    

But what if you have non-Ruby code you want to daemonize? Well my friends, that's what Kernel#exec is for and it works like a champ. See the README for the full details. And of course to view the README, you have to install the gem which means you have daemon-spawn in the bowels of your machine! Mwaaa haa haa haa! Oops…I've said too much…

In all seriousness though, I would like to thank the powers-that-be at work who were very gracious to let me open-source this. You should start seeing more of this kind of stuff from Evri soon. As always, your feedback, comments, critiques and patches are welcome.

We have a RubyGem written to handle registration and renewal that hides the HTTP and XML message bodies away from the user. You simply create a client, setup your initial registration and tell it to keep you registered.

require "rubygems"
    require "radar_love"
    client = Radar::Client.new("http://radar-dev")
    service = client.create('foobar', 'http://foobar:1234')
    service.keep_registered # fires up background thread
    

That last line is implemented with a Ruby thread that loops indefinitely, sleeping and then renewing the registration lease. But a funny thing happened while implementing this. When we just fired up irb and tried to run this part (without doing any other work), the re-registration thread never executed. Man, I had heard that MRI threads were "broken", but this is completely non-functioning!

Then I remembered a very handy page I ran across once about MRI Threading. This page is worth spending a little time with, but essentially because MRI threads are so-called "green threads" they aren't really giving you true concurrent processing. Instead they are merely a context-switching mechanism, and the circumstances under which those switches can happen are described in that spec page.

In the case of our little irb session, the re-registration thread only began executing when we did something in the main thread. We weren't executing anything in the main thread that triggered one of these context-switches (remember, we're merely sitting at an irb prompt waiting for the next line). So getting the runtime to execute a context-switch merely required us to do something in the main thread:

loop do
      puts "Howdy!"  
      sleep 5
    end
    

You may shake your head and mutter something derisive about this "hack". However, in reality, requiring your main thread to do something isn't terribly burdensome. If you didn't have any work to do in your main thread, you'd have to ask yourself why you created a separate thread in the first place!

You may also think that since MRI threads don't provide true concurrency, they're worthless. One major limitation of green threads in MRI is that no matter how many you start, they will only execute on one processor. If you have a large multi-core machine, MRI threads will not be able to take advantage of them.

However, that doesn't mean threads don't have their place in MRI environments. In the first example I mentioned (tailing logs and publishing summaries) we use a separate thread for the publishing activity. We could have done this entire action in a single loop, but the major downside is that we would essentially be relying on new lines in our log to appear to "crank" the mechanism forward. If we go a long time before we see another log line, our summary task will fail to execute.

IO.popen("tail -F /var/log/some.log").each do |line|
      update_statistics(line)
      if Time.now >= next_report_time
        # we might not get here for a while unless the
        # log lines keep coming
        report_statistics
      end
    end
    

It would certainly be possible to read from the file with a timeout that is based on how much time is left before the next reporting period. However at that point the code starts to get a little cluttered, so we go with the threaded approach only to take advantage of its context-switching properties. In our case, this is a perfect solution for what we're trying to accomplish.

If you come from a Java or .NET background, you may find that MRI threads fail to measure up to threading in those environments. It's absolutely true that MRI does not provide the same robust threading mechanisms that those languages do (JRuby, and perhaps IronRuby, being special cases). It doesn't mean that threads in MRI are worthless, you just need to understand them properly to know when to use them.

Meet Magit from Alex Vollmer on Vimeo.

It's not meant to be an exhaustive survey of magit (check the docs for all the details), but to show off some of the cool features this mode has. I found that spending just a little time with docs and learning this mode has already paid off in terms of increasing my productivity. Enjoy!

Changes:

1.0.1 / 2009-01-06

• Fixed a bug where generating help resulted in an infinite-loop

In moochbot, your main transaction screen has a standard tabbed-interface. Each tab is a different view of all your transactions. In a tabbed display you can only show one view at a time so it helps the user when you can provide some hints in the non-selected tabs. Anything that helps them figure out whether or not they want to click on something without actually having to click on it is, in my opinion, a great help.

So I wanted to add a number in the tab to indicate how many items were there. The most naïve way to implement this would be to issue three different SQL statements for the counts. However somewhere, in the back of my mind, I remembered that ActiveRecord has a feature known as the counter cache. The basic idea is to hook some additional code into the lifecycle of ActiveRecord's associations to update a single column in the parent record as you add and remove child records.

Like an iceberg, the bulk of this feature lay deep below the surface. The actual view-layer changes were minimal, but I had to jump through some hoops to get the counter-cache working correctly.

In moochbot, a User model object has multiple Transaction records. Each Transaction points at two separate User records: one for the lender and one for the borrower. All of a user's transactions are stored in the TRANSACTIONS table, each record differentiated by the STATE column.

In ActiveRecord-land we express these relationships in the model with three has_many relations for the User: one for items they are lending, one for items they are borrowing and all closed transactions. The first two are fairly straight-ahead:

class User < ActiveRecord::Base
    
      has_many(:lent_items,
               :class_name => "Transaction",
               :foreign_key => "lender_id",
               :conditions => ["state IN (?)",
                               %w(started lent returned overdue disputed)])
    
      has_many(:borrowed_items,
               :class_name => "Transaction",
               :foreign_key => "borrower_id",
               :conditions => ["state IN (?)",
                               %w(started lent returned overdue disputed)])
    
      has_many(:completed_items,
               :class_name => "Transaction",
               :finder_sql => 'SELECT * ' +
               'FROM transactions ' +
               'WHERE state IN (\'aborted\', \'finished\') AND ' +
               '(borrower_id = #{id} OR lender_id = #{id})')
    end
    

However the third relationship requires some custom SQL because we want all records that are in either the "finished" or "aborted" state and where the user is either the lender or the borrower. I looked into doing this with a simple :conditions option on the has_many relationship, but couldn't figure out how to specify the ID of the user.

One really important thing to recognize here is that the SQL is quoted with single-quotes. If the SQL is specified in double-quotes, the interpolation is evaluated too early and the id value is not the user record. Putting it in single-quotes defers evaluation until the proper time. I wish this were documented a little better because I was completely stuck until I stumbled across this post on the RailsBlaster blog. I had to enable debug-level logging for ActiveRecord to see that I was getting skooky IDs in my final SQL string.

To add a counter cache to the User record, you declare a :counter_cache option on the reciprocal belongs_to relationship. This seemed counter-intuitive to me since if I didn't already have one in place, I'd have to add one. It seemed more obvious to me to put it in the has_many relationship but that ain't the way ActiveRecord rolls. So the next step was to update the belongs_to mappings in the Transaction class:

class Transaction < ActiveRecord::Base
    
      belongs_to(:lender,
                 :class_name => "User",
                 :foreign_key => "lender_id",
                 :counter_cache => :lent_items_count)
    
      belongs_to(:borrower,
                 :class_name => "User",
                 :foreign_key => "borrower_id",
                 :counter_cache => :borrowed_items_count)
    end
    

The final step was to create a migration that would add the counter cache columns to the USERS table. Note that not only do we add the columns, but we also update everyone's counters.

class AddCounterCacheToUsers < ActiveRecord::Migration
      COLUMNS = [:lent_items_count,
                 :borrowed_items_count,
                 :completed_items_count]
    
      def self.up
        COLUMNS.each do |c|
          add_column :users, c, :integer, :default => 0
        end
    
        User.reset_column_information
    
        User.find(:all).each do |user|
          User.update_counters(user.id,
                               :lent_items_count => user.lent_items.length,
                               :borrowed_items_count => user.borrowed_items.length,
                               :completed_items_count => user.completed_items.length)
        end
      end
    
      def self.down
        COLUMNS.each do |c|
          remove_column :users, c
        end
      end
    end
    

So there it is. Hopefully that helps the next poor sod that runs into that same problem.

I've been a football fan since I was about five years old. I have studied this game for almost thirty years, so forgive me if I bore you all with a trite football analogy. This is the old saw known as "fast vs. quick". Players like Barry Sanders or Walter Payton were "quick", but not necessarily fast. In general, "fast" guys are wide receivers and corner-backs (the guys that cover the receivers). Receives generally have further to run than running backs, so they need what is called "breakaway speed" which is enough raw power to get away from the defender covering them.

Running backs however, need a combination of strength and the ability to move their bodies very quickly in a very small space. The difference between a gain of one or two yards and a breaking-off a twenty-yard run is usually a matter of inches. The runner has to be in just the right place at just the right moment and get just the right kind of blocking from his teammates. Quickness is the ability to shift weight, turn the body and plant a foot just a fraction of second faster than the next guy. It's what separates the good from the truly great.

Hall of Fame running backs rarely survive on pure speed alone. Heck, most backs that were "burners" in college are usually forced to play another position at the professional level. Those that try to rely on pure brawn to break tackles generally don't last long (see, Eddie George, Priest Holmes or any back that has played in Denver in the last decade). The really great ones have that ethereal quality known as "quickness".

So, I was thinking lately about "agile" in the software world and what it really means. The term agile, isn't clear enough—it could be fast or it could be quick. But which really matters? Which is the quality you really want to have?

My answer? You want the agility associated with quickness, not raw speed. Being "quick" in software might look like: * New requirements are handled in-stride and rarely, if ever, with The Big Re-Write * The time to produce a meaningful feature is measured in hours or days, not weeks * The time to produce a meaningful release is measured in days or weeks, not months * The system almost never looks like the original design

OK, so what would "fast" look like? Well… * Releases are buggy * Few if any tests * Code diarrhea * Quality is inconsistent * There are lots of dependencies * New requirements are an exercise in hammering reality into fixed model

Actually, none of those sound particularly "fast" to me.

When folks complain about the ineffectiveness of "Agile", I sometimes wonder if they aren't simply confusing fast with quick. There isn't any methodology or toolset I know of that will make you faster. Your raw speed is determined more or less by the skill of your team. Each member transfers a concept into code at a relatively consistent rate, so you simply aren't going to go any faster unless you change your team. However I think you can acquire new skills to improve your quickness.

He came from the .net side of things and had been doing contracting for quite awhile. Recently though, his shop closed up and he was between gigs. He hadn't been exposed to Ruby and thought it sounded kind of interesting. He was, as he put it, "a Windows-guy", but basically apologized for it saying that "everyone around here" runs Windows. Really? You mean like the four-out-of-five-laptops that are Macs in this café??!!! (Note to self: this fella doesn't seem too observant.)

So while I was trying to politely end the conversation to get back to my hacking and an IRC-meeting, he rambled on about his personal work philosophy. God knows how long he droned on, but his M.O. essentially boiled down to "Hey I don't care what I work on. Languages are all the same, why fuss over this one or that one?"

It stopped me in my tracks.

Are you kidding me?

Look, there is something laudable (in a kind of John-and-Yoko-stay-in-bed-for-peace way) about The Big World view where all the geeks get along and sing side by side at the great binary campfire. At a certain level arguing over this language vs. that language or this tool or that operating system is frivolous waste of time. Much of our "values" in these things are based merely on opinions, aesthetics and feelings. It's like mailing list arguments over whether or not the Enterprise could defeat the Galactica. Ladies and Gentleman on your left is Dork City…

But what this really told me more than anything else was that this guy was simply indifferent. You don't have to agree with my way of doing things, but when we talk about them I at least expect you to have an opinion and to be able to back that up with some reasoning. This guy's indifference stemmed from the fact that he treated this like a job.

"Passion" is a word that is overused in this biz. Every time I'm in a debrief about some candidate and the notion of "passion" comes up I started checking the room for forks to shove in my eyes. I don't care about passion for The Company or The Idea. That's rah-rah B.S. that I have no stomach or patience for. What I do want to see in a co-worker is someone who cares about what they work on and how they do it. This is your day-to-day existence. This is what you spend a lot of waking hours doing, being away from your loved ones. You damn-well better care about how you want to spend that time.

If you wanted to fail spectacularly in a job interview with me, one way to score a boat-load of demerit-points quickly is give an indifferent answer to the question, "What do you like to work on?" When people shrug their shoulders and say, "Oh, I'll work on anything", it tells me that they don't care. The pay is the same, so why fuss over the details?

Now I understand that some interviewees won't give me a straight answer because they expect that I want the answer an automaton would give. There are certainly plenty of shops that just want "human capital" in much the same way generals in WWI wanted troops (see fodder, cannon). But even so, I'm not sure that's the kind of person I want on my team. Show a little back-bone for pete's sake! Do you do this job just because it pays the bills? Are you so beaten-down by the industry and your experiences that you have no hope of it ever getting better? Are you just trudging forward in your profession because you don't know what else to do?

I don't care so much whether or not we agree on tools, languages or operating systems, but I hope to God you care about how you spend your time on this earth in this profession. Don't sleepwalk through it.

When Columbus reached the New World, he burned his ships. As a result his men were well-motivated.

This quote comes from the character of Marko Ramius, as played by Sean Connery, from one of my all-time favorite films, "The Hunt for Red October". I love that quote in particular because I always think of it whenever I decide to tweak my process, my work environment or my tools. For all but the most trivial changes, you usually have to immerse yourself in it. So I usually "burn my ships" by leaving behind the old way of doing things to really burn the new way into my brain.

Sometimes the new way turns out to not be so great. That's fine because that's a satisfactory result of the experiment. If the "new way" turns out to suck, there's just as much to learn (if not more) from a real crash-and-burn than success.

The first big "ship-burning" I did was very early in my career. I had been a few months into my professional developer career doing Java and decided that I really liked this Unix thing. So I completely wiped out my Windows NT work box and put one of the first versions of Red Hat on there. I figured if I was going to get good at Linux I was going to have to just jump in the deep end of the pool and learn to swim.

It turned out to be an absolutely wonderful decision. Throughout the years I moved through several distributions: Red Hat to Mandrake to Gentoo to Ubuntu. Each new distribution taught me something more about Linux than I knew before (especially true for Gentoo).

I've done the same trick with editors too. When I first started out I used vi for my Java development. I tried using the first IDEs (JBuilder, etc.) but they were all terrible. Plus, at that time a serious Java IDE for Linux was still a twinkle in Erich Gamma's eye. I had a co-worker who was an XEmacs power-user and it looked like he got a lot done so I figured I'd give it a go—especially since I had on-site technical support. This was my first experience with Emacs and I never really got beyond basic editing. I wrote a couple of macros but that was about it.

Then NetBeans rolled around and I was more than happy to start using it. I had a decent intellisense feature and built-in Ant integration, both of which were a significant step forward to me. I ran with that for a couple of years until I started a new job where everyone was using this new-fangled thing called Eclipse. I tried to hold-out with NetBeans for a while, but after watching some others pull off some seriously cool tricks in Eclipse I knew it was time to give it a go. I've ridden that Eclipse horse for damn-near six years and I'd be willing to put my knowledge of Eclipse-editing arcana up against anyone.

Several years laster, on the operating system front, another big change was coming. I had been a die-hard Linux fan since my switch back in the Old Days, but by winter of 2005 I had run into a tangled knot of Linux-related headaches that really sapped the fun out of it for me. Between home servers and 64-bit laptop (in 2005!) I spent more time just trying to keep my machines up to date and running than actually getting anything productive done. However, it wasn't until I went to my first Seattle Ruby Brigade meeting (when everyone there had a Mac) that I seriously considered switching operating systems.

This was the same time that I discovered Ruby. What attracted me to the language was not only the beauty of the syntax, but the underlying philosophies of pragmatism, simplicity and no-bullshit attitude. There seemed to be a similar attitude in Mac users and I found this approach appealing. This is not a knock on Linux. I love Linux, I just don't want to use it for my desktop OS anymore. I don't have the patience to fiddle with things that much anymore. I have lots of ideas and I'll never get anywhere if I get bogged-down with irrelevant minutiae and yak-shaving.

When I switched to the Mac I got myself a license for TextMate which has definitely been the editor-of-choice for lots of Rubyists. However, as I blogged about before, it's now time to get off the TM-train and switch (back) to Emacs.

So what's the point of all of this? Am I just thrashing? Am I desperately throwing myself at each and every new tool on the block? You might think so, but I actually have some discriminating taste. I do try out a lot of tools because I'm always looking for some kind of edge, anything that I can leverage to get stupid busy-work out of the way. As I get older I have less and less patience for this sort of thing for productivity taxes. I just want to get something important done.

So, from time to time, a little alarm goes off inside my brain to remind me that it's time for a change. Sometimes it's something as simple as changing my terminal or editing font. Other times it's things like learning a new language. Regardless, all of these things keep me from getting mentally lazy. Once I get mentally complacent, it's game over. So consider if perhaps it's time to "burn some ships" just keep things interesting.

Good heavens, some day I may even do something as crazy as switch to a Dvorak keyboard layout. That just might make my head explore—in a good way.

Yesterday the city of Seattle was paralyzed by the fear of snow (which arrived today). As a result downtown was a ghost-town. That meant that we could get sandwiches from Salumi (one of Seattle's culinary gems) in less than the usual hour-long wait.

Even on a "quiet" day, the process still took about thirty minutes to order, receive and pay for our sandwiches. As we walked back to the office we started chatting about ways they could be faster. They could make sure that people are paying as quickly as they are getting their sandwiches made. They could make sure that the full menu is displayed further back in the line so customers know what they want as they arrive.

We even ranked some of the local sandwich shops in terms of speed. If you want an incredible sandwich and have the time, Salumi is hard to beat. Next up would be our beloved Tat's. Finally, you can go to a chain place, like Jimmy John's, and get a decent corporate sandwich in unbelievable time.

So then I asked myself the question, "why doesn't Salmui try to go any faster?" Wouldn't their customers be happier? The lines are ridiculous at times and you can blow a whole lunch-hour just on getting a sandwich. The place is tiny too. It's near-impossible to eat it there because of the wait for a table. Wouldn't they like a bigger location to service more customers?

And then it hit me: they don't go any faster and they don't grow any bigger because it wouldn't make them any happier to do so. The folks behind the counter clearly love what they're doing. It's not that they're indifferent to their customers, it's that they seem to be doing quite well the way things are and don't feel a need to "grow". I think there's something very laudable in that attitude.

Anyway, one day it occurred to me that it would be funny (to me at least) to have a very simple, single-serving site that told you whether a certain song or artist was, in fact, Freedom Rock. Note the capital letters. I'm not talking Toby Keith "freedom rock" with waving flags in the background and lots of shiny pickup trucks. I'm talking about the rather odd, rag-tag collection of songs that made it to this album. Try playing with the site a bit, then check out the Urban Dictionary's entry on Freedom Rock.

This leads to one of the pillars of my personal philosophy:

Amuse yourself first, worry about your audience later.

I wrote this primarily because it make me laugh…a lot. I don't know why. It's dumb, it's juvenile and it's totally worth the $10 I paid to register the domain-name.

But stupid-hippy jokes don't make up the whole picture. The other half of this fun little excursion was figuring out how to do this with as little code as possible. At first I wrote this as a flat Merb application. Unfortunately I wanted to host this on Dreamhost and they are still on Ancient Ruby (read, 1.8.5) and the newest Merb wants 1.8.6. That was a deal-breaker. So then I took a look at Sinatra, and just like the bobby-soxers of the 40's I fell in love. Sigh…

If you haven't had a chance to look at Sinatra, you should do yourself a favor and check it out. They call it a DSL for the web, not a framework. It has the nice terse syntax of Camping, but has source that gives me half a chance to comprehend. It's short and sweet and gives enough things that will make sense to Rails-veteran, but with little extra cruft. The source is up on GitHub (with a shocking number of commits for such a simple project).

• Receiving email works much much better when you configure your MX records correctly
• Mixing ERb and HAML is a really really bad idea
• I'm probably using RubyGems in my Rails app in about the stupidest way possible
• Good dynamic form validation with JavaScript is actually quite tricky
• Squashing commits in Git is still pretty damn cool.

That's all for now. Good night and good luck.

The secret side-project that I've been referring to in various posts finally reached its Version One milestone tonight. I closed out the remaining V1 issues off of my to-do list and suddenly realized that I could turn it loose on the world. So, without further ado, I present to you: moochbot. Check it out and see what I've been up to all this time.

I'd love to hear any feedback folks have; what you love and what you hate. I have a ton of V2 features planned so moochbot will certainly continue to evolve. It's running on a single Slicehost slice so it's not real high-powered. Your patience is appreciated.

I'll have plenty of blog-material from this experience to write up later, but for now I'm just going to enjoy the satisfying feeling of getting something done. Enjoy!

For some reason I've always been a little bothered by this and have always felt like it should be avoided. I dislike this way of speaking about a system because of how easy it is to personalize code that someone has written. I think it's simply too easy to make a statement that sounds like a criticism of the person instead of the work. A criticism of the latter can be hard enough to take, but when it's misconstrued with a personal critique it's very easy for feelings to get hurt and team morale to take a nose-dive. In my experience it takes far longer to gain the trust of your teammates than it does to lose it.

The other reason I don't like it is that it reinforces ownership boundaries around particular parts of the code. In an ideal world your team members would be able to move fluidly about the codebase depending on the features to be implemented. In reality this can become difficult, but like a lot of thinks in life I think it's a good "stretch-goal". So when we speak in personal pronouns instead of component names it seems to reinforce these barriers (or maybe it simply highlights them.)

I've found that it's difficult to avoid talking about systems in these terms. I have to make a conscious effort to not do this in much the same way I have to remember to floss. Both seem like unnatural activities that require a lot of discipline to maintain. I can understand why I might be inclined to forget flossing regularly, but it's not as clear to me why all of us developers want to personify code in our daily speech. Any ideas?

Like a good boy I started out with the simplest thing that could work which was to simply call mailers in my model. However, I wanted to limit the number of activities performed during a request to keep the app feeling responsive. So I decided that asynchronous mail delivery was a "pre-launch" feature that I had to have.

I looked at a variety of background processing tools, including Bj, Starling/Workling, Spawn and AP4R. Each had its strengths and weaknesses but none of them felt like the right fit. My research criteria included: * Job persistence via the database * Something that could get a Rails environment cheaply * Runs outside of the Rails processes * Minimum fuss to get it running

In the end the one that hit the sweet-spot best was delayed_job. It had the DB persistence I was looking for, but didn't source the Rails environment for each worker and it was extremely simple to plumb it into my app.

Refactoring

The first step was creating the DelayedJob worker classes; one for each mail action. At first this turned into a big pile of five-line classes, so to keep things organized I put these all in app/models/jobs and put each class in the Jobs module namespace. This was better, but not good enough so the final step was putting all of the worker classes in a single file, app/models/jobs.rb.

The second step was to find every place in the model where I called my mailer classes directly and replace them with calls to enqueue the appropriate worker job.

Here is what things looked like at first:

class UserObserver < ActiveRecord::Observer
      def after_create(user)
        unless user.current_state == :latent or user.is_a?(Admin)
          UserNotifier.deliver_signup_notification(user)
        end
      end
    
      def after_save(user)
        if user.current_state == :promoted
          UserNotifier.deliver_signup(user)
        else
          UserNotifier.deliver_activation(user) if user.recently_activated?
        end
      end
    end
    

Then the UserObserver was refactored like this:

class UserObserver < ActiveRecord::Observer
      def after_create(user)
        unless user.current_state == :latent or user.is_a?(Admin)
          Delayed::Job.enqueue(Jobs::UserNotifierSignupNotificationJob.new(user.id))
        end
      end
    
      def after_save(user)
        if user.current_state == :promoted
          Delayed::Job.enqueue(Jobs::UserNotifierSignupNotificationJob.new(user.id))
        else
          Delayed::Job.enqueue(Jobs::UserNotifierActivationJob.new(user.id)) if user.recently_activated?
        end
      end
    end
    

With the following workers (abridged):

module Jobs
      class UserNotifierDisconnectJob < Struct.new(:user_id)
        def perform
          UserNotifier.deliver_disconnect(user_id)
        end
      end
    
      class UserNotifierResetPasswordJob < Struct.new(:user)
        def perform
          UserNotifier.deliver_reset_password(user)
        end
      end
    
      class UserNotifierSignupNotificationJob < Struct.new(:user)
        def perform
          UserNotifier.deliver_signup_notification(user)
        end
      end
    
      class UserNotifierStartDisconnectJob < Struct.new(:user_id)
        def perform
          UserNotifier.deliver_start_disconnect(user_id)
        end
      end
    end
    

Testing

Prior to switching to asynchronous processing, mail delivery was triggered within my models, either via Observers or as Procs attached to state transitions (I'm using the acts_as_state_machine plugin). Therefore my tests had loads of assertions that various state changes in the model resulted in direct email delivery. In the asynchronous model of course, that changes slightly. While the state change ultimately ends in mail delivery, it only happens indirectly.

So here I had a big pile of tests that asserted that poking the model in certain ways resulted in a mail delivery. In my unit-tests I really just wanted to test the interaction between the models and DelayedJob. After all, if something went hay-wire during mail delivery the culprit would likely be my new worker classes, not my model.

However, for my integration tests I still wanted to keep the assertions about actual mail delivery since that was an important part of the stories. I could easily do this by monkey-patching the DelayedJob::enqueue method to call the worker's perform method directly. In my unit-tests I monkey-patched the DelayedJob::enqueue method to work more like a mock object which added some inquiry methods to check that it had been invoked correctly.

In isolation this worked great, but running all the tests together resulted in a number of random failures. I've run into this enough times to recognize that some tests were somehow poisoning the run-time environment for the others. It turns out that my two approaches were incompatible with each other unless I was very diligent about cleaning everything up properly. I will admit with red-faced shame that I punted. I did the lamest thing one could possibly do and redefined DelayedJob::enqueue for all of my tests and kept all of my original assertions. I'm not proud of it, but it does work.

Running in Production

The next trick was getting this all running in a production environment and I needed to figure out how one or more workers would be started and kept running. While it's great to have this decoupled from the Rails environment, it means that it's a separate process that needs to be managed.

My solution was to use the daemons gem to create a couple of scripts. Then I used Tom Preston-Warner's god, to monitor my process. The scripts look like this:

#!/usr/bin/env ruby
    
    unless ARGV.size == 1
      $stderr.puts "USAGE: #{0} [environment]"
      exit 1
    end
    
    RAILS_ENV = ARGV.first
    require File.dirname(__FILE__) + '/../config/environment'
    
    Delayed::Worker.new.start
    

The "control" script looks like this:

#!/usr/bin/env ruby
    
    require "rubygems"
    require "daemons"
    
    def running?(pid)
      # Check if process is in existence
      # The simplest way to do this is to send signal '0'
      # (which is a single system call) that doesn't actually
      # send a signal
      begin
        Process.kill(0, pid)
        return true
      rescue Errno::ESRCH
        return false
      rescue ::Exception
        # for example on EPERM (process exists but does not belong to us)
        return true
      end
    end
    
    if ARGV.size == 1 and ARGV.first == "status"
      pidfile = "/var/run/delayed_job_worker.pid"
      if File.exists?(pidfile)
        pid = open(pidfile).readlines.first.strip.to_i
        if running?(pid)
          puts "delayed_job_worker is running (#{pid})"
        else
          puts "delayed_job_worker is NOT running (#{pid})"
        end
      else
        puts "delayed_job_worker is NOT running (none)"
      end
    else
      Daemons.run(File.dirname(__FILE__) + '/delayed_job_worker',
                  :backtrace => true,
                  :log_output => true,
                  :dir => File.dirname(__FILE__) + "/../tmp/pids",
                  :dir_mode => :normal,
                  :multiple => false)
    end
    

The first script is merely the smallest amount of chrome required to start a worker. Note that I'm using John Barnette's version of delayed_job which gives us that nice little worker riff. The second script is essentially the daemons wrapper around my worker. For a little extra goodness I added my own "status" command which can be handy for debugging.

Getting those to work properly took a little bit of testing by hand. Fortunately, the entire solution is really a series of layers applied on top of each other. Once you convince yourself that an inner-layer is working correctly you can move on to build the outer layers.

The next step was to create a proper god configuration. I have more than one thing to monitor on my setup so I have a master god configuration that includes sub-configurations. My "main" configuration is a simple one-liner: God.load "/etc/god/*.god" My application-specific configuration looks like this (with a few edits for public consumption):

RAILS_ROOT = "/var/www/moochbot/current"
    
    God::Contacts::Email.message_settings = {
      # your config goes here
    }
    
    God::Contacts::Email.server_settings = {
      # your config goes here
    }
    
    God.contact(:email) do |c|
      # your config goes here
    end
    

After setting up some default notification details, we get into the meat of defining our "watch":

God.watch do |w|
      w.name = "delayed_job_worker"
      w.interval = 10.seconds
      w.start = "#{RAILS_ROOT}/script/delayed_job_worker_control start -- production"
      w.stop = "#{RAILS_ROOT}/script/delayed_job_worker_control stop"
      w.restart = "#{RAILS_ROOT}/script/delayed_job_worker_control restart"
      w.start_grace = 10.seconds
      w.restart_grace = 10.seconds
      w.pid_file = "#{RAILS_ROOT}/tmp/pids/delayed_job_worker.pid"
    
      w.uid = "deploy"
      w.gid = "root"
      w.behavior(:clean_pid_file)
    

Next we need to define our transitions. My first attempt at this failed because I was missing these and my watched process was stuck in the "unmonitored" state. It's worth spending some time reading the docs on the homepage since at first glance the configuration wasn't obvious to me.

  # determine the state on startup
      w.transition(:init, { true => :up, false => :start }) do |on|
        on.condition(:process_running) do |c|
          c.running = true
        end
      end
    
      # determine when process has finished starting
      w.transition([:start, :restart], :up) do |on|
        on.condition(:process_running) do |c|
          c.running = true
        end
    
        # failsafe
        on.condition(:tries) do |c|
          c.times = 5
          c.transition = :start
        end
      end
    
      # start if process is not running
      w.transition(:up, :start) do |on|
        on.condition(:process_exits)
      end
    

Finally I specify some resource-consumption boundaries to make sure that my little worker daemon doesn't take over my box:

  w.restart_if do |restart|
        restart.condition(:memory_usage) do |c|
          c.above = 100.megabytes
          c.times = [3, 5]
        end
    
        restart.condition(:cpu_usage) do |c|
          c.above = 50.percent
          c.times = 5
        end
      end
    
      w.lifecycle do |on|
        on.condition(:flapping) do |c|
          c.to_state = [:start, :restart]
          c.times = 5
          c.within = 5.minute
          c.transition = :unmonitored
          c.retry_in = 10.minutes
          c.retry_times = 5
          c.retry_within = 2.hours
        end
      end
    end
    

So without further ado, let's start with the New York Times:

If you remember my original post, I favored the tabular displays generally over map-based views. My favorite part of this is the electoral vote tally on the left-hand side. To me, the most important information is the electoral vote tally and the Times has done a good job of highlighting this both by making it a larger font and putting it on the left-hand side. Then as my eye moves to the right I can drill down to state-by-state detail. The color codes provide a nice way to get a "feel" for the state of things at a glance.

Next stop, CNN:

What you don't see here is the completely unnecessary animation of the left-hand counters like some alarm-clock from the seventies. The other three panels "flip" to various local races. Instead of using space show multiple data-points, CNN chose to use time. This feel like a real waste of space and my time. When compared with the Times site, there is really no contest.

Let's move on to MSNBC. These guys get bonus points for including the Senate races in aggregated form—a nice touch if you want to take the pulse of all the races. This is a nice tight display of a lot of information.

Again, I could do without the map. It's not that I don't like maps (I like them quite a bit), but they don't tell me the most important information—the electoral college votes. It's interesting to see the geographic proclivities but it's essential that I know the electoral college votes. The geographically large, but sparsely-populated states in Big Sky Country and the Midwest distort our initial perception of the data. Their size belies their actual value in the race and so we have to constantly look elsewhere for electoral college vote totals. Unfortunately, this graph doesn't tell that story. I can't look at Ohio in this chart and understand its value in the race.

Next, the Washington Post:

This is pretty uninspiring and uninformative. Oh look! Pictures! Just in case you forgot what the candidates looked like. However, if we start clicking the tabs, we get a little more meat:

I'm not sure why the map is all-gray, but the I like the bar at the top. This seems to be the new trend this year in info-graphics. I think for highlighting the changes in the balance-of-power, the line chart does a good job.

Now we move onto USA Today…

That's it folks. That's all you get on the front page, up in the right-hand corner. I think that pretty much sums up the estimation the editors of McPaper have of their audience. Let's, for a moment, try to treat this with a serious eye. Could somebody please explain what the blue boxes on the right-hand side are for? Do you click them? What do they do? Why are they the only ones shown? Can you see others? This is what Edward Tufte would call "chart junk".

On to the once-venerable Wall Street Journal…

Same old map, so few details. I can't tell if the lack of titles on the shapes indicates that the Journal's editors assume their audience can actually tell the states apart, or that they just couldn't be bothered. Like the Washington Post we get the candidate's faces. While it fits the Journal's famous stodgy etchings theme, it doesn't bring much information to the table. Oh yeah, just in case you lack the skill to compare numbers, the big checkbox next to Obama's name serves to tell you who won. Sheesh.

One interesting note is that this is the first infographic of the major sites I surveyed that included results for any of the other candidates. It's interesting to note that just the simple quantity of digits gives you some idea of the magnitude of the votes the two major-party candidates get.

Check back in four years and we'll do it again!

Let's say that the processing of writing your program is like launching spacecraft. Ideally you would like to get from launch to cruising around in space as quickly as possible. The Star Trek universe solves this quite elegantly with the transporter. We don't put you in a box and launch it, we break your atoms apart and transmit them to another location! That's pretty cool, but maybe we're just not quite that cool yet. Another solution is one proffered by the Star Wars universe. A ship like the Millenium Falcon can leave just about any planetary atmosphere any time it damn well pleases without the use of special equipment. It just flies away. Not bad.

However, here on earth, our primitive space craft need a tremendous amount of disposable apparatus to reach escape-velocity. The proportion of useful vehicle (the shuttle) to the orbit-busting mechanism (the rocket boosters and fuel tanks) is a staggering 5.4:1 (based on liftoff weight).

Command-line parsing in code exhibits a similar disproportion. The interesting part of your app isn't the command-line parsing. Why should it take up such a disproportionate amount of space in your code? Those boosters quickly become "space-junk", once the launch vehicle has left Earth; expensive trash that is never to be used again.

Space-junk is dangerous for the next guy that wants to launch into space. It's also dangerous for the folks on the ground as it may decide to come crashing back down to earth. And those boosters have also been responsible for one of the worst disasters in NASA's history. If we could only get rid of those damn things the whole space program just might be a little better off. Unfortunately physics, and our current space-flight capabilities currently require them.

But our code is a different story. We don't have such physical barriers that handicap us. Any barriers we run into are usually of our own making. So why not try to reduce those as much as possible? Why say in more code, what you could say in less? Wouldn't that cut down on bugs? Wouldn't that ease the burden of maintenance? Wouldn't that reduce the amount of information overhead you have to maintain each time you revisit that code?

Command-line parsing is a stupid, menial task that should require very little attention. By extension, it should only be given a stupid, menial amount of code to make it work. We have big ideas! They shouldn't get bogged down by handling command-line options!

This is why I wrote Clip. Clip is an expression of the need to make the simple things simple, but no simpler. If you have modest command-line parsing needs, Clip rewards you with minimal investment. If you need something trickier, Clip allows you to say a little more to it and gain more benefits from it. You get to decide how much you want to engage—not the library.

This is one of the things I like about Ruby. The language is extremely flexible which gives me a lot of ways to "pack" ideas into code in a variety of ways. Having more than one way to do things isn't all that useful by itself. But it's essential when you want to write expressive code. Things like object literals, or one-line control-structure alternatives help me keep the lines of code proportional to the ideas they express.

This is also something I find challenging to do in Java. In languages like Java, even just creating a collection of objects requires quite a bit of code:

import java.util.*;
    
    public class Designer {
      public void makeItWork(List<Trash> trash) {
        // today's challenge: convert trash to wearable garments
        List<Garment> garments = new ArrayList<Garment>(trash.size());
        for (Trash t : trash) {
          Garment g = new Garment(t.getName());
          garments.add(g);
        }
    
        submitToJudges(garments);
      }
    }
    

In Java we often solve this by pushing all of that code into a private method that is named something meaningful. This works pretty well, but does tend to result in an explosion of "helper methods". Sometimes folks take the "cheap" route and simply prefix these riffs with explanatory comments like "convert each Trash into a Garment". I'm not a real big fan of this. Generally I don't care about the object conversion in the first place because the rest of the code is presumably doing something interesting with the Garments and I don't give a damn about the Trash.

So let's look at it in Ruby:

class Designer
      def make_it_work(trash)
        submit_to_judges(trash.map { |t| Garment.new(t.name) })
      end
    end
    

By my count there are five lines in the Java example (including the comment) just to convert trash to garments. In contrast I boiled that down to one line in Ruby. OK I could have done this in two lines if you think that's too much of a long-train. But I think there are couple of important points here: * The importance of the concept being expressed diminishes from left to right * The attention-span of the reader diminishes from top to bottom

The Ruby example beats Java on both counts. I don't waste a lot of the reader's attention span up-front on book-keeping details (in the vertical space) and I state the important thing I'm trying do (submit my top Foos) quickly (on the left). The details of which Garments I'm dealing with are merely a qualification of what I'm trying to do.

How you handle these two dimensions is greatly affected by both the language you use and the APIs you deal with. This is one of the reasons that I do not find the use of scripting languages for Java's Swing API all that compelling. Scripting languages like JRuby or Jython help me with the horizontal space, but don't do a damn thing for the vertical requirements. With an awful API like Swing I have to say a lot of words to make it go, regardless of the language I do it with.

Getting back to my dumb example, being on such a small scale this may not seem like a large impact. But multiplied several times over to match the size of most projects, this kind savings can really pile up. The difference in the amount of code required by these two approaches is manifested in a savings in cognitive investment required to grok these projects. This is the very essence of maintainability and sustainability. Anytime you can do more with less, you come out ahead.

At great peril to my own geek cred, I will say that this is why I find The Lord of the Rings to be such an awful piece of writing. It is so full of peripheral and non-essential information that finding the real story or characters requires extraordinary patience and concentration on the part of the reader. If Tolkien had been more concerned with the story and less with "world-building" I'll bet he could have gotten that story boiled down to a single book.

Now I realize that a lot of folks love the Tolkien books for the very reasons I criticize it. That's fine, that is an argument about aesthetics, not facts. However I would strongly argue that "world-building" in your code is a bad idea. I think you're much more likely to build a decent piece of software if you pack your ideas tightly like a William Gibson novel than as a sprawling "trilogy" of epic code. Go ahead, prove me wrong. I double dog-dare ya.

OK, so by this point any credibility I may have had is gone. Look at the size of a post about saying more with less. In the hope that you might be lazy and like to skip to the end: Do as much as you can…with as little as possible.

Since switching to the Mac over a year and half ago, I've used TextMate as my non-Java editor. TextMate is great text editor. It's relatively simple to extend, has a very active community and only takes a little investment before a user gets productive. But I'll admit that in the last month or so my eyes started wandering from TextMate—I felt like maybe I was ready to go back to Emacs.

Perhaps it's worth a little history first. Back in the Dark Ages of Java Development (anyone remember Gnu Server Pages?), I was a pretty hard-core XEmacs guy for my Java dev. At the time Java IDEs were painful exercises in waiting and crash-recovery. JBuilder, early NetBeans, Visual Cafe were all valiant attempts but terrible failures. So I rode with XEmacs until the IDEs caught up.

Then one day I met Eclipse. Unlike any other Java editing tool, Eclipse was much more than a text-editor. Rather than treating your code like text to be manipulated, Eclipse actually understood your code. This allows you to think in terms of manipulating Java syntax and symbols instead of simple text tokens and lines. For a static language like Java this is incredibly powerful since you can determine up-front all of the possible things you might want to write. This is why a full-on "intelli-sense" feature for dynamic languages is hard to do as well in static, compiled languages.

I spent the next five years learning just about every nook and cranny of Eclipse and would put my raw Java-writing skills up against anyone. Even now, I still use Eclipse for Java and I can't imagine going back to a plain text-editor—even a souped-up hot-rod like Emacs. This says less about the deficiencies of any text editors as it does about the requirements for developing in Java. Writing Java without all of the code-completion and refactoring tools an IDE like Eclipse has would simply exceed my pain threshold. So while Eclipse has become quite bloated lately and, when coupled with tools like Maven, is an exercise machine-servitude, it will be a cold day in hell before I give it up for Java. (NB: I plan on giving up Java-dev first.)

Now I'm writing a lot more non-Java than I did a year ago. That's what brought me to TextMate. The integration for Ruby, HTML, CSS, JavaScript and shell was outstanding. I bought the TextMate book and was on my way to editor ass-kicking.

Unlike my Java days, now I identify myself less with a single language. Back in the day I was a Java Programmer™ dad-gummit. It was The One Language To Rule Them All and it served me well. Do you remember those days? Yeah, that was also the time when certified Oracle dudes were writing their own checks. But as time went on, I noticed that those Oracle guys pretty much knew how to do one thing and if and when the Day of Database Reckoning ever came, these guys were gonna be left standing in the cold wondering what the hell happened. I think that day has come and those dudes are becoming the equivalent of COBOL programmers—relegated to propping up legacy systems and talking wistfully about "the good ol' days".

So it got me to thinking about what I needed to do to survive in this business. Clearly I was going to have to evolve. So I started looking around for some inspiration and came across Ruby. Right now it's my favorite language, but it's not the only game in town. Ruby borrows a lot of features from other languages and a lot of other Ruby-ists are also spending time with other languages like Haskell, Erlang, io, and Lisp just to get exposure to other ideas. Moving away from a single-language focus to broader interests has been an important part of moving from rookie to journeyman. Evolve or die, right?

So how does this relate to Emacs? Well, like a lot of folks, I've been interested in learning more about functional programming. The problem is, I simply haven't had the time to crack open the Erlang book and bend my brain in new ways. Don't get me wrong, it's on the list, but it's going to be a while before I get to it. So I was eager to dip my toe in FP without a major commitment. Enter ELisp.

Before I dropped out of college, my brief career in academic computer science was spent with Scheme. My time in school didn't end well and so I've associated Scheme (and by extension, Lisp) with that particularly unpleasant time in my life. I've only recently overcome the shakes when I see car or cdr. But a little time and some professional success has healed old wounds. In my first go-round with Emacs, I was really just using it as a power-editor. The internals of Emacs pretty much escaped me and in my mind there was a pretty major barrier between an ELisp user and an ELisp writer.

This time I've made a concentrated effort to learn more about how Emacs works. I've been racing through the O'Reilly Emacs book and this time it makes sense. I don't know what the hang-up was before, but this time it all seems so logical! The entire editor is simply a collection of ELisp functions and variables built on top of other functions and variables. It's all self-documented and incredibly dynamic. Wanna try something out? Just put a little Lisp in your *scratch* buffer and evaluate that sucker. Hell, in Eclipse I'd have to download the freakin' source and compile it or swim in XML writing an extension. That's incredibly powerful—the barrier to "trying stuff out" is mind-bogglingly low.

So while I've definitely take a short-term hit by trading in TextMate for Emacs, but I'm reaching productivity-parity pretty quickly. A couple of modes/libraries I've found that I've really liked are:

• Steve Yegge's JS2 mode
• Rinari, for Rails
• Org-mode
• Magit for Git integration

I plan on spending some quality time with some of the various snippets libraries too to port over some of the templates that TextMate uses. For something like Rails development, which is about the most idiomatic programming I can imagine, having short-hand snippets is a major productivity boost.

So why go through this? After all, switching editors can be like converting from Catholicism to Judaism for some. Well, I think I'm doing this because:

• I've already overcome the initial steep-learning curve of basic editing in Emacs
• I get a nice, no-commitment introduction to functional programming
• I get a great environment to learn new stuff in
• With a little effort and a willingness to learn, I can make this sucker to anything
• Emacs has quite a track-record. It's close association with Unix is no accident.
• Emacs customization is easy to write and a snap to track in SCM

I can't say that I'll use Emacs for everything. I'm getting into Cocoa development too and it looks like doing that outside of XCode is a fool's errand. But that's okay. I don't need a single tool to do everything for me, just a handful of tools that help me get a variety of things done. That said, I'm pretty sure I'll be solving a lot more problems in Emacs than I ever did in any other single tool.

The second-generation acolytes turned a very pragmatic set of principles and practices into an esoteric scripture that could never meet the needs of reality. The "agile" moniker makes me cringe almost as much as "best practices". Both are cavalier substitutes for critical thinking, too often thrown about in the hopes that their mere mention will work magic. Ugh.

A real unfortunate side-effect of the backlash is the disregard some have for automated testing. Before I go any further, let me be the first to say that, like any other recommendation or practice, automated testing is not a panacea. I shouldn't have to say this. It's painfully obvious, like explicitly stating up front that I am against beating up defenseless children. Who is?

However, just because automated testing isn't a cure-all doesn't mean that:

• testing isn't worth doing
• testing isn't hard
• sometimes testing is so hard that it isn't worth doing

Note that last point. That there is what we in the biz call pragmatism. Yes kids, there are times when the cost of getting sufficient test-coverage is enormously expensive. Assuming that economics are part of your project somehow, you need to pay attention to this (which is why code coverage metrics can be dangerous in the hands of the nuance-free thinker).

Critics of automated testing like to point out that if developers write buggy code, how can they not write buggy tests? Assuming that it's as impossible to write perfect tests as it is to write perfect code, is there a point to writing the test? There is if the benefits you get from testing exceed the cost of developing and maintaing them.

Test-Driven Design/Behavior-Driven Design

The TDD/BDD philosophies are nearly as abstract and misunderstood as the entire concept of testing itself. So rather than parrot the original manifestos, let me break down the tangible benefits of driving your code with tests. The first misconception is the notion that you write all the tests first then you write all the code. This works almost as well as gathering all the requirements up front first and then writing all the code. I believe the technical term is "Waterfall". This is just dumb. Honestly folks, use your brains. Only a complete rookie would think that's a sensible practice to adopt.

Putting testing before writing code provides no magical effect that radically increases the quality of your code. Where testing does improve your code is when you use them as part of the code-building process. I like to use unit-tests as a form of mental scaffolding while I develop.

Breaking It Down

Any non-trivial problem needs to be broken down in some way. Few of us have the mental capacity to keep an entire solution in our heads. Even if we could, it generally doesn't last beyond the period in which the solution is developed. And even if you could do this, chances are few of your teammates would be able to. So at worst, breaking a problem down is a pro-social practice; at best it's a tool to ensure that you really understand what's going both now and in the future.

Generally when you break a problem down, you get into the business of assigning responsibilities to various components. This is a fractal pattern that starts at the highest-level architecture and repeats itself all the way down to the lowest-level classes, modules and functions of your system. It must be an inherent part of human-nature—we simply love putting things in boxes. Perhaps this is because that in doing so, we can hang a simple tag on a collection of related concepts as a form of shorthand. That shorthand allows us to talk about things in an efficient way. Imagine a design conversation where you couldn't name anything, but had to describe a piece over and over by its details. Yuck. So yeah, boxes are important.

So where does testing fit in? Testing allows you to write automated assertions about each of those boxes. It's like a logic game where you can start draw conclusions based on what you already know. It's like being able to say, "If this test is passing and that test is passing, the only thing that could torpedo this is if that doesn't work". I'll come back to this later when I talk about troubleshooting.

What unit-testing does not cover here, are the connections between the boxes. It should be obvious that it's entirely possible to have a totally sweet set of unit-tests that never fail that cover each piece of your solution in full and still have bugs or represent and incorrect solution.

This is because unit-tests only cover the boxes, not the connections.

Tests that cover the entire system are the best way to exercise the connections. They cover the boxes too, but they cover the integration (hence the name) of the boxes and connections. Since unit-tests can't cover the system as a whole, are they still worth writing? You bet. Keep reading…

Efficient Troubleshooting

When something goes wrong, you enter a different frame of mind from programming. Now you're in the detective business. You start with some scraps of information (stack traces, error messages, late-night pages) and with a combination of logic, guile and intuition (hopefully) find the root cause.

I could do an entire post on root-cause analysis, but let it suffice to say that most folks don't ask enough questions when troubleshooting. Often the conclusion people come to is an observation of a symptom rather than a thoughtful, system-aware diagnosis. In these situations it's a good idea to consider the 5 Whys Approach.

How do unit-tests help with troubleshooting? Like I said earlier, unit-tests are really good at testing the boxes, but not the connections. Let's assume that your unit-test accurately covers the box that you believe is the root of the issue you're seeing. Given that, there is a handful of conclusions you can draw: * The problem isn't really in this box * The problem is the connection coming into or going out of this box * Your understanding of the box is incorrect

That's it. It has to be one of those. Unit-tests allow you to confirm that your box is working correctly. If it's being invoked or interacted with in a way you didn't expect, it's generally a trivial exercise to write a test that does use your box in that way. If it passes then you probably need to keep looking.

If you're pretty confident that your box is good, then checking the connections is the next thing to look at. It's just as likely that the output of your box is being misused as the input to it. Integration tests are usually the best mechanism to catch this, but are generally much more expensive to write and maintain. For this reason they generally don't exhaust all of the possibilities. But with a real-world bug you now have a candidate for a new integration test. I can't stress enough the benefits of using bugs as opportunities to strengthen your test-suites. This is as true for integration tests as it is for unit-tests. It's generally not practical to write tests for every situation. So we use our intellect to make educated guesses that cover the most likely situations. When you observe funny behavior in your system, consider it a gift to your understanding. It's a golden opportunity to add a test for a real behavior, not a speculative one that may or may not occur.

So let's consider a world in which we have no tests. When something goes haywire you have to start from first principles every time. You have no mechanisms to get any leverage on the issue. Instead we have to rely on understanding the entire solution and holding it in our head at once. What do you think the chances are that we'll effectively troubleshoot an issue under these conditions?

Finding Smells

It is simply no fun to work on low-quality code day after day. As anyone who has every read Zen & The Art of Motorcycle Maintenance can attest, "quality" is a pretty squishy word. But in concrete terms, quality code means: * You enjoy working on it (or at least don't hate it) * You can expand its capabilities with reasonable effort * Somebody else can work on it too

I find the process of test-driving the development of code is a great opportunity to find "smells" in your code. These "smells" generally detract from code quality. The best catalog of these smells that I've encountered is the Refactoring book by Fowler and Beck. You don't need to read the book for a detailed description of the mechanics of refactoring, but it is worth reading to learn the terminology of "code smells".

At a minimum tests are the first "customers" of your code. If it's painful to test your code, you probably haven't factored it very well. Paying attention to this while your test-driving the development is a great way to keep your code high-quality. To really benefit from this, you need experience, a highly-developed sense of aesthetics and intuition. If you keep at it, you will gain these.

Note that I don't equate code quality with "success". There are plenty of systems out there that are successful that are not built on what I would call quality code. That's fine. I've made a conscious choice not to work on those systems. I spend an awful lot of my waking hours working on code—personally I don't want to waste that time being frustrated. I want a sense of accomplishment and satisfaction when I'm done for the day. Honestly, I'd trade success for quality code just about anytime. I expect to be doing this for a long time, so why not enjoy it?

Safety Net for Refactoring

When you need to extend or enhance a system, having automated tests serve as a great safety-net for refactoring. If you're confident in the efficacy of your unit-tests, you can be confident when you make changes and your tests still pass. You can refactor with a higher level of confidence when your tests keep passing. Of course there are no guarantees that you haven't introduced some kind of regression bug. But let's face it folks, the testing game is one of probabilities. It's too expensive to make sure you're right all of the time—instead shoot for being right most of the time.

There will always be times when something goes wrong and the only way to diagnose and fix it is to walk through the code painstakingly, line by line. No amount of testing will ever make that go away. My goal is to reduce that as much as possible. It's inevitable, but it doesn't have to be the norm.

Consider the case where you don't have any tests while you refactor. Your code/build/test/debug cycle gets a whole lot longer and more expensive. Because of that, you do it less often. Which means that when you do go through the cycle you have more things that can go wrong and more stuff you have to hold in your head. That increases both the likelihood that you will mis-diagnose a problem and that you will write more bad code to fix the latest bug.

I think this is how really twisted code is born. It almost reads like a blow-by-blow account of the developer's mounting frustration. The pace and phrasing of the code gets more and more frenetic: more odd comments, more TODOs and FIXMEs littered throughout, more long procedural stanzas. It becomes obvious that they were trying to get this done as quickly as possible, and that's generally not the path to quality.

Confidence

When it comes human-nature and software I'm a pessimist. I think we're really good at the intuitive parts and achieving the "ah-ha" moment. We are not good at holding giant complicated systems in our heads. There are people who can do that, but it's not the norm. For this reason, I think it's extremely foolish to build a software organization that relies on raw mental capacity. This definition of "smart" is one that can't scale and is unsustainable.

Automated testing is one teeny-tiny way we can offload the need for that capacity to a better long-term storage mechanism. While I write tests my mind is very focused on a particular part of the system. As I flesh out the tests I fill out my understanding of the problem-space at that time. Now that it's written down in code, I can safely leave it there and let my mind free up that space to do other things.

If I've written my tests well, I can get back into that focused frame of mind when I need to (e.g. for new features or bug-fixes). Without these, I have to spend a lot more time understanding a larger system just to get "up to speed". The tests can serve as an index into the design of the system. This is particularly effective when you have been paying attention to code smells and have been refactoring aggressively—your tests will map effectively to the underlying concepts of the system design.

This kind of confidence has a real tangible benefit. Without it I'm easily distracted by small details and dead-ends. As soon as I have more I have to worry about at once, I have less of a chance of succeeding at the task at hand in that moment. When I have such a thing in place, I have an instant force-multiplier in my hand. I can get something done. I don't have to repeat the entire experience of developing the code the first time. Instead I can leverage that experience to be more productive the next time I come back to that part of the system.

Conclusion

So there you have it. That's where I stand.

Is testing hard? Yes.

Does it catch everything? No.

Is it still worth doing? I think so.

This was causing all sorts of grief for me in some JavaScript I was trying to write. At first I tried to go with the flow and fix my JS, but it got really hacky really fast. So I went on a little source-code spelunking to figure how to make the problem go away.

In actionpack-2.1.0 there is a Proc attached to the ActionView::Base.field_error_proc class attribute. It's not documented in the RDoc, but this is also a writable attribute which means I can shut the damn thing up. Here's what it looks like in the original file, GEM_HOME/actionpack-2.1.0/lib/action_view/helpers/active_record_helper.rb:

require 'cgi'
    require 'action_view/helpers/form_helper'
    
    module ActionView
      class Base
        @@field_error_proc = Proc.new{ |html_tag, instance| "<div class=\"fieldWithErrors\">#{html_tag}</div>" }
        cattr_accessor :field_error_proc
      end
      ...
    end
    

My solution was to create a little file in config/initializers named field_errors.rb with this text:

ActionView::Base.field_error_proc = Proc.new do |html_tag, instance|
      html_tag
    end
    

Et voila! Just a simple pass-through with no more fancy-pants markup. Putting stuff like this in separate files in the initializers directory keeps your config and environment files from getting out of hand.