Going paperless? Get a Canon P-215 ($285) or Canon P-208 ($170), PDFScanner for the Mac ($15), and get going.

We have a great little Swedish-made birch wood filing cabinet that we use to store our paperwork: bank statements, legal documents, that stuff. However, the filing cabinet is almost full (the USA likes paperwork), and I’ve refused to get another one, because more paper means more space and more waste. So, I decided to go paperless: scan everything, store them as PDFs, and shred the paper after it’s scanned.

There are some services (such as Outbox in the USA) that will do this for you. Call me oldskool, but they’re just not my thing: I don’t like the idea of other people reading my mail, any more than I like the idea of other people reading my email. I’d also prefer not to pay a subscription every year, though honestly, at ~$10/month, these services aren’t that expensive if you decide that you really hate doing this yourself. So I decided to see if I could find a solution that did what I wanted:

• scan stuff fast: 5 pages+ per minute, with double-sided (a.k.a. duplex) scanning;
• scan stuff in bulk: at least 10 pages at a time, and save out selected pages to different documents;
• OCR everything: save as text-searchable PDF (searchable by Spotlight on OS X);
• and, the biggest problem with scanners: have software that wasn’t a pain in the arse to use.

Doesn’t seem that onerous, does it? Well, I’ve actually been looking for a solution for a long time now…

The Fujitsu ScanSnap

I’ve had the legendary Fujitsu ScanSnap S300M for years. The ScanSnap series are the kings of document scanners: duplex, fast, sheet feeder; pick three. Unfortunately, they suffered from one large problem: you had to use Fujitsu’s own software to do the scanning, because they didn’t have TWAIN drivers, which is the standard scanner driver interface on OS X and Windows. This means that on a Mac, you couldn’t use Image Capture, Preview, or Photoshop to scan things, and were limited by what Fujitsu’s software could do. Double unfortunately, Fujitsu’s software is what you’d expect from a company that makes printers and scanners: not great. The more expensive ScanSnaps support OCR, but the S300M didn’t, and I couldn’t cobble together a satisfactory workflow that used the Fujitsu software and integrate it with an OCR program. I tried hacking together Automator scripts, custom AppleScripts, shell and Python stuff, DevonThink Pro Office (which has native support for the ScanSnap) and couldn’t find something satisfactory.

Taking a different route inspired by Linux geekdom, I did try TWAIN SANE for OS X, which takes the hundreds of drivers available for the Linux SANE scanning framework, and presents them as TWAIN devices for OS X. Unfortunately, the driver for the S300M didn’t work: sane-find-scanner would find it just once, and never find it again, and scanimage never worked. I would’ve loved to debug it and fix it, but that “life” thing keeps getting in the way. So, death to all scanners with a proprietary interface. What other options are there?

The Canon P-215

Thankfully, Canon’s entered the market with their very silly-named, yet totally awesome Canon imageFORMULA P-215 Scan-tini Personal Document Scanner. It’s like the Fujitsu ScanSnap S300M, but better in every way. It scans 15ppm instead of the ScanSnap’s 8ppm; has a 20-page sheet feeder instead of 10; can be powered from a single USB port; and most importantly, it’s TWAIN-compliant. The P-215 is $285 on Amazon. The Wirecutter, a fantastic gadget review site, agrees that the P-215 is the most awesomest portable scanner around.

There’s also the Canon P-208 for about $100 less, which is basically the P-215 lite: smaller and slower (about the ScanSnap’s speed), but still TWAIN-complaint. I use the P-215, but see no reason why the P-208 would be significantly worse for what I do. I probably would’ve got the P-208 if it were $170: at the time I bought them, the P-215 was $270 and the P-208 was $230, and I thought the $40 was worth the extra features. $170 is a much better price than $270, though, so look into the P-208 seriously if you’re considering this.

The scanner is designed so that it presents itself as both a scanner and a USB drive when you plug it in, with the USB drive containing the scanning software, so you’re never without it. Clever. The software that comes with the Canon is actually “not that bad” as far as scanning software goes… which still means it’s pretty craptastic. The Wirecutter does a good job of reviewing the software, so I won’t review it here, except that to say that my standards for quality software is probably higher than the Wirecutter’s. However, since the P-215 is TWAIN-compliant, you can use any software you want to scan stuff. So, what’s some good scanning software?

PDFScanner

Thankfully, one coder from Germany was sick of all the crappy scanning software out there, and wrote his Own Damn Scanning Software, creatively named PDFScanner. PDFScanner is just plain excellent. If you have a scanner at all, just go buy it. It’s a measly $15, and I guarantee you that it’s orders of magnitude better than the tosspot scanning software that you got with your scanner.

• It does OCR.
• It does deskewing.
• It detects blank pages and removes them from the scan.
• You can re-order and remove pages that you scanned. (OMG! Wow!)
• It’s multithreaded so it OCRs and deskews multiple pages at once, and puts all those cores in your Mac to work. Oooer.
• It has a “fake duplex” mode, so that if your scanner doesn’t support duplex scanning, you can scan the first side of all your pages, then the second side of all your pages. Cool.
• You can select some pages out of the 20 you scanned, save just those pages as a single PDF, then remove them from the scan. Imagine that.
• You can select different compression levels when saving the PDF.
• It can import existing PDFs and OCR them.

Paired with PDFScanner, the P-215 will quickly munch through your paper documents so that you can happily shred them afterwards. The shredding is satisfying.

Did I mention that PDFScanner is $15? It’s $15. I hope the guy makes 10x as much money from it as the fools who write the retarded scanning software that comes with scanners.

Recommendations

I scan stuff at 300dpi: a one-page document is around 800KB. You may want to scan stuff at higher resolutions if you’re paranoid about reproducing them super-well if you need to re-print them. I don’t think it’s necessary.

I have a folder creatively named “Filing Cabinet” that I throw all my documents into. (Lycaeum is also a cool name if you’re an Ultima geek.) The top-level folder structure inside there mostly resembled the physical filing cabinet: “Banks”, “Cars”, “Healthcare”, etc, and works well. One of the nice things about a digital filing cabinet is that you’re not limited to just one level of dividers: just go create sub-folders inside your top-level folders, e.g. “Insurance” inside “Cars”. (Such advanced technology!) I include the date in the filename for most of my scans in YYYYMMDD format, but not all of them (e.g. I do for bank statements and car service appointments, but not for most work-related material).

Since all these documents contain some sensitive information, you do want to store them securely, but be able to conveniently access them. I store my stuff on Google Drive since I trust Google’s security (go two-factor authentication).

I expect the virtual filing cabinet to grow to 10-20GB for a few years of data, which is peanuts these days. I’m happy to pay Google the two-digit cost per year to have that much storage space in Google Drive.

Small Niggles

The workflow’s not perfect, but it really is close enough to perfect that I feel it’s about as streamlined as you can get. You do need to double-check that every physical page made it through to the final PDF correctly, which is just common sense. Don’t expect to scan 1000 pages, click Save and shred things without cross-checking.

Feeding in the paper into the Canon P-215 in the orientation that you’d expect means that you have to tell PDFScanner to scan in “Portrait (Upside Down)” mode instead of just “Portrait” mode, which seems a bit odd. (I’d blame this on the Canon TWAIN driver rather than PDFScanner, mind you.) No other side-effects besides having to pick the upside-down orientation, though.

Scanning in a mix of landscape and portrait documents doesn’t work if you want to OCR the whole batch, because PDFScanner will OCR the scan in its original orientation only, and won’t let you re-run OCR after you’ve rotated the page. This just means that you have to scan in portrait documents in one batch, and landscape documents in another batch. Not a big deal, especially since scanning a batch in PDFScanner is simply pressing the “Scan” button. I’ve emailed the PDFScanner author about this, and got a response back within a day saying that he’ll consider adding it to the next version. Maybe it’s already fixed.

It’d be a bonus for the P-215 to be wireless, instead of requiring a USB cable. However, the scanner’s so small and portable that I just grab it, scan stuff, then put it back on the shelf, so this isn’t a problem for me. If you really want, you can buy a fairly expensive $170 WiFi adapter for the P-215. At least you get a battery pack for your $170 too.

The Paperless Workflow In Practice

In practice, the workflow’s worked out quite well. I’m now rapidly churning through the entire filing cabinet’s worth of documents, and have shredded 99% of the paper. The workflow is near-perfect, with the very small caveats mentioned above.

I do keep a few physical documents, such as things printed on special paper (e.g. US social security card), birth certificates, etc. Those exceptions are extremely rare though; in total, I’ve managed to reduce a 80-90cm stack of paper (~3 feet) to about an inch. I love going paperless, and would highly encourage anyone who’s been thinking about it to get a Canon P-208 or P-215, PDFScanner, and make the leap. I said that the shredding is satisfying, right? It’s very satisfying.

I’ve been pretty dormant in my music for the past few years, but I have been working on two two mixes in my sparse spare time: Tes Lyric, a weird blend of electronica, classical and rock, and Stage Superior, a progressive house mix. They’re up on my music page now; enjoy!

I’ve been pretty dormant in my music for the past few years, but I have been working on two two mixes in my sparse spare time: Tes Lyric, a weird blend of electronica, classical and rock, and Stage Superior, a progressive house mix. They’re up on my music page now; enjoy!

I recently ran into some “interesting” behaviour when using lambda in Python. Perhaps it’s only interesting because I learnt lambdas from a functional language background, where I expect that they work a particular way, and the rest of the world that learnt lambdas through Python, Ruby or JavaScript disagree. (Shouts out to you Objective-C developers who are discovering the wonders of blocks, too.) Nevertheless, I thought this would be blog-worthy. Here’s some Python code that shows the behaviour that I found on Stack Overflow:

Since I succumb to reading source code in blog posts by interpreting them as “blah”¹, a high-level overview of what that code does is:

1. iterate over a list of strings,
2. create a new list of functions that prints out the strings, and then
3. call those functions, which prints the strings.

Simple, eh? Prints “do”, then “re”, then “mi”, eh? Wrong. It prints out “mi”, then “mi”, then “mi”. Ka-what?

(I’d like to stress that this isn’t a theoretical example. I hit this problem in production code, and boy, it was lots of fun to debug. I hit the solution right away thanks to the wonders of Google and Stack Overflow, but it took me a long time to figure out that something was going wrong at that particular point in the code, and not somewhere else in my logic.)

The second answer to the Stack Overflow question is the clearest exposition of the problem, with a rather clever solution too. I won’t repeat it here since you all know how to follow links. However, while that answer explains the problem, there’s a deeper issue. The inconceivable Manuel Chakravarty provides a far more insightful answer when I emailed him to express my surprise at Python’s lambda semantics:

This is a very awkward semantics for lambdas. It is also probably almost impossible to have a reasonable semantics for lambdas in a language, such as Python.

The behaviour that the person on SO, and I guess you, found surprising is that the contents of the free variables of the lambdas body could change between the point in time where the closure for the lambda was created and when that closure was finally executed. The obvious solution is to put a copy of the value of the variable (instead of a pointer to the original variable) into the closure.

But how about a lambda where a free variable refers to a 100MB object graph? Do you want that to be deep copied by default? If not, you can get the same problem again.

So, the real issue here is the interaction between mutable storage and closures. Sometimes you want the free variables to be copied (so you get their value at closure creation time) and sometimes you don’t want them copied (so you get their value at closure execution time or simply because the value is big and you don’t want to copy it).

And, indeed, since I love being categorised as a massive Apple fanboy, I found the same surprising behaviour with Apple’s blocks semantics in C, too:

You can see the Gist page for this sample code to see how to work around the problem in Objective-C (basically: copy the block), and also to see what it’d look like in Haskell (with the correct behaviour).

In Python, the Stack Overflow solution that I linked to has an extremely concise way of giving the programmer the option to either copy the value or simply maintain a reference to it, and the syntax is clear enough—once you understand what on Earth what the problem is, that is. I don’t understand Ruby or JavaScript well enough to comment on how they’d capture the immediate value for lambdas or whether they considered this design problem. C++0x will, unsurprisingly, give programmers full control over lambda behaviour that will no doubt confuse the hell out of people. (See the C++0x language draft, section 5.1.2 on page 91.)

In his usual incredibly didactic manner, Manuel then went on to explain something else insightful:

I believe there is a deeper issue here. Copying features of FP languages is the hip thing in language design these days. That’s fine, but many of the high-powered FP language features derive their convenience from being unspecific, or at least unconventional, about the execution time of a piece of code. Lambdas delay code execution, laziness provides demand-dependent code execution plus memoisation, continuations capture closures including their environment (ie, the stack), etc. Another instance of that problem was highlighted by Joe Duffy in his STM retrospective.

I would say, mutability and flexible control flow are fundamentally at odds in language design.

Indeed, I’ve been doing some language exploration again lately as the lack of static typing in Python is really beginning to bug me, and almost all the modern languages that attempt to pull functional programming concepts into object-oriented land seem like a complete Frankenstein, partially due to mutability. Language designers, please, this is 2011: multicore computing is the norm now, whether we like it or not. If you’re going to make an imperative language—and that includes all your OO languages—I’ll paraphrase Tim Sweeney: in a concurrent world, mutable is the wrong default! I’d love a C++ or Objective-C where all variables are const by default.

One take-away point from all this is to try to keep your language semantics simple. I love Dan Ingall’s quote from Design Principles Behind Smalltalk: “if a system is to serve the creative spirit, it must be entirely comprehensible to a single individual”. I love Objective-C partially because its message-passing semantics are straightforward, and its runtime has a amazingly compact API and implementation considering how powerful it is. I’ve been using Python for a while now, and I still don’t really know the truly nitty-gritty details about subtle object behaviours (e.g. class variables, multiple inheritance). And I mostly agree with Guido’s assertion that Python should not have included lambda nor reduce, given what Python’s goals are. After discovering this quirk about them, I’m still using the lambda in production code because the code savings does justify the complexity, but you bet your ass there’s a big comment there saying “warning, pretentous code trickery be here!”

1. See point 13 of Knuth et al.’s Mathematical Writing report.

UPDATE: There’s a lot more subtlety at play here than I first realised, and a couple of statements I’ve made above are incorrect. Please see the comments if you want to really figure out what’s going on: I’d summarise the issues, but the interaction between various language semantics are extremely subtle and I fear I’d simply get it wrong again. Thank you to all the commenters for both correcting me and adding a lot of value to this post. (I like this Internet thing! Other people do my work for me!)

Update #2

I’ve been overwhelmed by the comments, in both the workload sense and in the pleasantly-surprised-that-this-provoked-some-discussion sense. Boy, did I get skooled in a couple of areas. I’ve had a couple of requests to try to summarise the issues here, so I’ll do my best to do so.

Retrospective: Python

It’s clear that my misunderstanding of Python’s scoping/namespace rules is the underlying cause of the problem: in Python, variables declared in for/while/if statements will be declared in the compound block’s existing scope, and not create a new scope. So in my example above, using a lambda inside the for loop creates a closure that references the variable m, but m’s value has changed by the end of the for loop to “mi”, which is why it prints “mi, mi, mi”. I’d prefer to link to the official Python documentation about this here rather than writing my own words that may be incorrect, but I can’t actually find anything in the official documentation that authoritatively defines this. I can find a lot of blog posts warning about it—just Google for “Python for while if scoping” to see a few—and I’ve perused the entire chapter on Python’s compound statements, but I just can’t find it. Please let me know in the comments if you do find a link, in which case I’ll retract half this paragraph and stand corrected, and also a little shorter.

I stand by my assertion that Python’s for/while/if scoping is slightly surprising, and for some particular scenarios—like this—it can cause some problems that are very difficult to debug. You may call me a dumbass for bringing assumptions about one language to another, and I will accept my dumbassery award. I will happily admit that this semantics has advantages, such as being able to access the last value assigned in a for loop, or not requiring definitions of variables before executing an if statement that assigns to those variables and using it later in the same scope. All language design decisions have advantages and disadvantages, and I respect Python’s choice here. However, I’ve been using Python for a few years, consider myself to be at least a somewhat competent programmer, and only just found out about this behaviour. I’m surprised 90% of my code actually works as intended given these semantics. In my defence, this behaviour was not mentioned at all in the excellent Python tutorials, and, as mentioned above, I can’t a reference for it in the official Python documentation. I’d expect that this behaviour is enough of a difference vs other languages to at least be mentioned. You may disagree with me and regard this as a minor issue that only shows up when you do crazy foo like use lambda inside a for loop, in which case I’ll shrug my shoulders and go drink another beer.

I’d be interested to see if anyone can come up an equivalent for the “Closures and lexical closures” example at http://c2.com/cgi/wiki?ScopeAndClosures, given another Python scoping rule that assignment to a variable automatically makes it a local variable. (Thus, the necessity for Python’s global keyword.) I’m guessing that you can create the createAdder closure example there with Python’s lambdas, but my brain is pretty bugged out today so I can’t find an equivalent for it right now. You can simply write a callable class to do that and instantiate an object, of course, which I do think is about 1000x clearer. There’s no point using closures when the culture understands objects a ton better, and the resulting code is more maintainable.

Python summary: understand how scoping in for/while/if blocks work, otherwise you’ll run into problems that can cost you hours, and get skooled publicly on the Internet for all your comrades to laugh at. Even with all the language design decisions that I consider weird, I still respect and like Python, and I feel that Guido’s answer to the stuff I was attempting would be “don’t do that”. Writing a callable class in Python is far less tricky than using closures, because a billion times more people understand their semantics. It’s always a design question of whether the extra trickiness is more maintainable or not.

Retrospective: Blocks in C

My C code with blocks failed for a completely different reason unrelated to the Python version, and this was a total beginner’s error with blocks, for which I’m slightly embarrassed. The block was being stack-allocated, so upon exit of the for loop that assigns the function list, the pointers to the blocks are effectively invalid. I was a little unlucky that the program didn’t crash. The correct solution is to perform a Block_copy, in which case things work as expected.

Retrospective: Closures

Not all closures are the same; or, rather, closures are closures, but their semantics can differ from language to language due to many different language design decisions—such as how one chooses to define the lexical environment. Wikipedia’s article on closures has an excellent section on differences in closure semantics.

Retrospective: Mutability

I stand by all my assertions about mutability. This is where the Haskell tribe will nod their collective heads, and all the anti-Haskell tribes think I’m an idiot. Look, I use a lot of languages, and I love and hate many things about each of them, Haskell included. I fought against Haskell for years and hated it until I finally realised that one of its massive benefits is that things bloody well work an unbelievable amount of the time once your code compiles. Don’t underestimate how much of a revelation this is, because that’s the point where the language’s beauty, elegance and crazy type system fade into the background and, for the first time, you see one gigantic pragmatic advantage of Haskell.

One of the things that Haskell does to achieve this is the severe restriction on making things immutable. Apart from the lovely checkbox reason that you can write concurrent-safe algorithms with far less fear, I truly believe that this makes for generally more maintainable code. You can read code and think once about what value a variable holds, rather than keep it in the back of your mind all the time. The human mind is better at keeping track of multiple names, rather than a single name with different states.

The interaction of state and control flow is perhaps the most complex thing to reason about in programming—think concurrency, re-entrancy, disruptive control flow such as longjmp, exceptions, co-routines—and mutability complicates that by an order of magnitude. The subtle difference in behaviour between all the languages discussed in the comments is exemplar that “well-understood” concepts such as lexical scoping, for loops and closures can produce a result that many people still don’t expect; at least for this simple example, these issues would have been avoided altogether if mutability was disallowed. Of course mutability has its place. I’m just advocating that we should restrict it where possible, and at least a smattering of other languages—and hopefully everyone who has to deal with thread-safe code—agrees with me.

Closing

I’d truly like to thank everyone who added their voice and spent the time to comment on this post. It’s been highly enlightening, humbling, and has renewed my interest in discussing programming languages again after a long time away from it. And hey, I’m blogging again. (Though perhaps after this post, you may not think that two of those things are good things.) It’s always nice when you learn something new, which I wouldn’t have if not for the excellent peer review. Science: it works, bitches!

I recently ran into some “interesting” behaviour when using lambda in Python. Perhaps it’s only interesting because I learnt lambdas from a functional language background, where I expect that they work a particular way, and the rest of the world that learnt lambdas through Python, Ruby or JavaScript disagree. (Shouts out to you Objective-C developers who are discovering the wonders of blocks, too.) Nevertheless, I thought this would be blog-worthy. Here’s some Python code that shows the behaviour that I found on Stack Overflow:

Since I succumb to reading source code in blog posts by interpreting them as “blah”¹, a high-level overview of what that code does is:

1. iterate over a list of strings,
2. create a new list of functions that prints out the strings, and then
3. call those functions, which prints the strings.

Simple, eh? Prints “do”, then “re”, then “mi”, eh? Wrong. It prints out “mi”, then “mi”, then “mi”. Ka-what?

(I’d like to stress that this isn’t a theoretical example. I hit this problem in production code, and boy, it was lots of fun to debug. I hit the solution right away thanks to the wonders of Google and Stack Overflow, but it took me a long time to figure out that something was going wrong at that particular point in the code, and not somewhere else in my logic.)

The second answer to the Stack Overflow question is the clearest exposition of the problem, with a rather clever solution too. I won’t repeat it here since you all know how to follow links. However, while that answer explains the problem, there’s a deeper issue. The inconceivable Manuel Chakravarty provides a far more insightful answer when I emailed him to express my surprise at Python’s lambda semantics:

This is a very awkward semantics for lambdas. It is also probably almost impossible to have a reasonable semantics for lambdas in a language, such as Python.

The behaviour that the person on SO, and I guess you, found surprising is that the contents of the free variables of the lambdas body could change between the point in time where the closure for the lambda was created and when that closure was finally executed. The obvious solution is to put a copy of the value of the variable (instead of a pointer to the original variable) into the closure.

But how about a lambda where a free variable refers to a 100MB object graph? Do you want that to be deep copied by default? If not, you can get the same problem again.

So, the real issue here is the interaction between mutable storage and closures. Sometimes you want the free variables to be copied (so you get their value at closure creation time) and sometimes you don’t want them copied (so you get their value at closure execution time or simply because the value is big and you don’t want to copy it).

And, indeed, since I love being categorised as a massive Apple fanboy, I found the same surprising behaviour with Apple’s blocks semantics in C, too:

You can see the Gist page for this sample code to see how to work around the problem in Objective-C (basically: copy the block), and also to see what it’d look like in Haskell (with the correct behaviour).

In Python, the Stack Overflow solution that I linked to has an extremely concise way of giving the programmer the option to either copy the value or simply maintain a reference to it, and the syntax is clear enough—once you understand what on Earth what the problem is, that is. I don’t understand Ruby or JavaScript well enough to comment on how they’d capture the immediate value for lambdas or whether they considered this design problem. C++0x will, unsurprisingly, give programmers full control over lambda behaviour that will no doubt confuse the hell out of people. (See the C++0x language draft, section 5.1.2 on page 91.)

In his usual incredibly didactic manner, Manuel then went on to explain something else insightful:

I believe there is a deeper issue here. Copying features of FP languages is the hip thing in language design these days. That’s fine, but many of the high-powered FP language features derive their convenience from being unspecific, or at least unconventional, about the execution time of a piece of code. Lambdas delay code execution, laziness provides demand-dependent code execution plus memoisation, continuations capture closures including their environment (ie, the stack), etc. Another instance of that problem was highlighted by Joe Duffy in his STM retrospective.

I would say, mutability and flexible control flow are fundamentally at odds in language design.

Indeed, I’ve been doing some language exploration again lately as the lack of static typing in Python is really beginning to bug me, and almost all the modern languages that attempt to pull functional programming concepts into object-oriented land seem like a complete Frankenstein, partially due to mutability. Language designers, please, this is 2011: multicore computing is the norm now, whether we like it or not. If you’re going to make an imperative language—and that includes all your OO languages—I’ll paraphrase Tim Sweeney: in a concurrent world, mutable is the wrong default! I’d love a C++ or Objective-C where all variables are const by default.

One take-away point from all this is to try to keep your language semantics simple. I love Dan Ingall’s quote from Design Principles Behind Smalltalk: “if a system is to serve the creative spirit, it must be entirely comprehensible to a single individual”. I love Objective-C partially because its message-passing semantics are straightforward, and its runtime has a amazingly compact API and implementation considering how powerful it is. I’ve been using Python for a while now, and I still don’t really know the truly nitty-gritty details about subtle object behaviours (e.g. class variables, multiple inheritance). And I mostly agree with Guido’s assertion that Python should not have included lambda nor reduce, given what Python’s goals are. After discovering this quirk about them, I’m still using the lambda in production code because the code savings does justify the complexity, but you bet your ass there’s a big comment there saying “warning, pretentous code trickery be here!”

1. See point 13 of Knuth et al.’s Mathematical Writing report.

UPDATE: There’s a lot more subtlety at play here than I first realised, and a couple of statements I’ve made above are incorrect. Please see the comments if you want to really figure out what’s going on: I’d summarise the issues, but the interaction between various language semantics are extremely subtle and I fear I’d simply get it wrong again. Thank you to all the commenters for both correcting me and adding a lot of value to this post. (I like this Internet thing! Other people do my work for me!)

Update #2

I’ve been overwhelmed by the comments, in both the workload sense and in the pleasantly-surprised-that-this-provoked-some-discussion sense. Boy, did I get skooled in a couple of areas. I’ve had a couple of requests to try to summarise the issues here, so I’ll do my best to do so.

Retrospective: Python

It’s clear that my misunderstanding of Python’s scoping/namespace rules is the underlying cause of the problem: in Python, variables declared in for/while/if statements will be declared in the compound block’s existing scope, and not create a new scope. So in my example above, using a lambda inside the for loop creates a closure that references the variable m, but m’s value has changed by the end of the for loop to “mi”, which is why it prints “mi, mi, mi”. I’d prefer to link to the official Python documentation about this here rather than writing my own words that may be incorrect, but I can’t actually find anything in the official documentation that authoritatively defines this. I can find a lot of blog posts warning about it—just Google for “Python for while if scoping” to see a few—and I’ve perused the entire chapter on Python’s compound statements, but I just can’t find it. Please let me know in the comments if you do find a link, in which case I’ll retract half this paragraph and stand corrected, and also a little shorter.

I stand by my assertion that Python’s for/while/if scoping is slightly surprising, and for some particular scenarios—like this—it can cause some problems that are very difficult to debug. You may call me a dumbass for bringing assumptions about one language to another, and I will accept my dumbassery award. I will happily admit that this semantics has advantages, such as being able to access the last value assigned in a for loop, or not requiring definitions of variables before executing an if statement that assigns to those variables and using it later in the same scope. All language design decisions have advantages and disadvantages, and I respect Python’s choice here. However, I’ve been using Python for a few years, consider myself to be at least a somewhat competent programmer, and only just found out about this behaviour. I’m surprised 90% of my code actually works as intended given these semantics. In my defence, this behaviour was not mentioned at all in the excellent Python tutorials, and, as mentioned above, I can’t a reference for it in the official Python documentation. I’d expect that this behaviour is enough of a difference vs other languages to at least be mentioned. You may disagree with me and regard this as a minor issue that only shows up when you do crazy foo like use lambda inside a for loop, in which case I’ll shrug my shoulders and go drink another beer.

I’d be interested to see if anyone can come up an equivalent for the “Closures and lexical closures” example at http://c2.com/cgi/wiki?ScopeAndClosures, given another Python scoping rule that assignment to a variable automatically makes it a local variable. (Thus, the necessity for Python’s global keyword.) I’m guessing that you can create the createAdder closure example there with Python’s lambdas, but my brain is pretty bugged out today so I can’t find an equivalent for it right now. You can simply write a callable class to do that and instantiate an object, of course, which I do think is about 1000x clearer. There’s no point using closures when the culture understands objects a ton better, and the resulting code is more maintainable.

Python summary: understand how scoping in for/while/if blocks work, otherwise you’ll run into problems that can cost you hours, and get skooled publicly on the Internet for all your comrades to laugh at. Even with all the language design decisions that I consider weird, I still respect and like Python, and I feel that Guido’s answer to the stuff I was attempting would be “don’t do that”. Writing a callable class in Python is far less tricky than using closures, because a billion times more people understand their semantics. It’s always a design question of whether the extra trickiness is more maintainable or not.

Retrospective: Blocks in C

My C code with blocks failed for a completely different reason unrelated to the Python version, and this was a total beginner’s error with blocks, for which I’m slightly embarrassed. The block was being stack-allocated, so upon exit of the for loop that assigns the function list, the pointers to the blocks are effectively invalid. I was a little unlucky that the program didn’t crash. The correct solution is to perform a Block_copy, in which case things work as expected.

Retrospective: Closures

Not all closures are the same; or, rather, closures are closures, but their semantics can differ from language to language due to many different language design decisions—such as how one chooses to define the lexical environment. Wikipedia’s article on closures has an excellent section on differences in closure semantics.

Retrospective: Mutability

I stand by all my assertions about mutability. This is where the Haskell tribe will nod their collective heads, and all the anti-Haskell tribes think I’m an idiot. Look, I use a lot of languages, and I love and hate many things about each of them, Haskell included. I fought against Haskell for years and hated it until I finally realised that one of its massive benefits is that things bloody well work an unbelievable amount of the time once your code compiles. Don’t underestimate how much of a revelation this is, because that’s the point where the language’s beauty, elegance and crazy type system fade into the background and, for the first time, you see one gigantic pragmatic advantage of Haskell.

One of the things that Haskell does to achieve this is the severe restriction on making things immutable. Apart from the lovely checkbox reason that you can write concurrent-safe algorithms with far less fear, I truly believe that this makes for generally more maintainable code. You can read code and think once about what value a variable holds, rather than keep it in the back of your mind all the time. The human mind is better at keeping track of multiple names, rather than a single name with different states.

The interaction of state and control flow is perhaps the most complex thing to reason about in programming—think concurrency, re-entrancy, disruptive control flow such as longjmp, exceptions, co-routines—and mutability complicates that by an order of magnitude. The subtle difference in behaviour between all the languages discussed in the comments is exemplar that “well-understood” concepts such as lexical scoping, for loops and closures can produce a result that many people still don’t expect; at least for this simple example, these issues would have been avoided altogether if mutability was disallowed. Of course mutability has its place. I’m just advocating that we should restrict it where possible, and at least a smattering of other languages—and hopefully everyone who has to deal with thread-safe code—agrees with me.

Closing

I’d truly like to thank everyone who added their voice and spent the time to comment on this post. It’s been highly enlightening, humbling, and has renewed my interest in discussing programming languages again after a long time away from it. And hey, I’m blogging again. (Though perhaps after this post, you may not think that two of those things are good things.) It’s always nice when you learn something new, which I wouldn’t have if not for the excellent peer review. Science: it works, bitches!

There comes in the career of every motion picture that final occasion when all the artistry, all the earnest constructive endeavor of all the man-power and genius of the industry, and all the capital investment, too, must pour through the narrow gate of the projector on its way to the fulfilment of its purpose, the final delivery to the public.

The delivery is a constant miracle of men and mechanism in the projection rooms of the world’s fifty thousand theatres. That narrow ribbon, thirty-five millimetres, flowing at twenty-four frames a second through the scintillating blaze of the spot at the picture aperture and coursing by at an exactingly-precise 90 feet a minute past the light slit of the sound system, demands a quality of skill and faithful, unfailing attention upon which the whole great industry depends.

The projector lens is the neck in the bottle through which all must pass. The projectionist presiding over that mechanism is responsible for the ultimate performance upon which we must all depend.

The projector must not fail, and more importantly still, the man must not fail or permit it to waiver in its performance. It is to the tremendous credit of the skill of the modern projectionist that perfect presentation of the motion picture upon the screen is today a commonplace, a perfection that is taken as a matter of course.

Adolph Zukor, Chairman of Paramount Pictures, 1935. It still applies as much today as it did back then.

There comes in the career of every motion picture that final occasion when all the artistry, all the earnest constructive endeavor of all the man-power and genius of the industry, and all the capital investment, too, must pour through the narrow gate of the projector on its way to the fulfilment of its purpose, the final delivery to the public.

The delivery is a constant miracle of men and mechanism in the projection rooms of the world’s fifty thousand theatres. That narrow ribbon, thirty-five millimetres, flowing at twenty-four frames a second through the scintillating blaze of the spot at the picture aperture and coursing by at an exactingly-precise 90 feet a minute past the light slit of the sound system, demands a quality of skill and faithful, unfailing attention upon which the whole great industry depends.

The projector lens is the neck in the bottle through which all must pass. The projectionist presiding over that mechanism is responsible for the ultimate performance upon which we must all depend.

The projector must not fail, and more importantly still, the man must not fail or permit it to waiver in its performance. It is to the tremendous credit of the skill of the modern projectionist that perfect presentation of the motion picture upon the screen is today a commonplace, a perfection that is taken as a matter of course.

Adolph Zukor, Chairman of Paramount Pictures, 1935. It still applies as much today as it did back then.

Pixar’s latest short film. I’m so proud and honoured to be working with such an amazing group of people.

Pixar’s latest short film. I’m so proud and honoured to be working with such an amazing group of people.

Thanks to several evil friends of mine, I started to take an interest in photography at the end of last year. I’ve always wanted to have a “real” camera instead of a point and shoot, so at the start of 2010, I bit the bullet and bought myself a Panasonic Lumix DMC-GF1, usually just called The GF1 amongst the camera geeks.

I tossed up between the GF1 and the then-just-released Canon EOS 550D (a.k.a. the Rebel T2i in the USA) for a long time. I figured that getting a compact camera would make me tote it around a lot more, and after ten months of using it, I think I was right. I recently went to a wedding in Sydney, and I literally kept the camera in my suit pocket instead of having to lug it around strapped to my body or neck. I definitely wouldn’t be able to do that with a Canon or Nikon DSLR. The camera’s so small with the kit 20mm f/1.7 lens that I stopped using the UV filter with it, because I didn’t like the 2-3mm that the filter added to the camera depth. Here’s a size comparison of the Nikon D3000 vs the GF1.

(Image stolen from dpreview.com’s review.)

I won’t write up a comprehensive review of the GF1 here; other sites have done that, in much more depth than I can be bothered to go into. If you’re after a good review, the three articles that swayed me to the GF1 in the end were DPreview’s review, and Craig Mod’s GF1 photo field test article and video tests. What follows is my own impressions and experiences of using the camera. The one-sentence summary: the GF1 perfect for a DSLR newbie like me, the micro four-thirds lens system it uses looks like it has enough legs that your lens investments will be good for the future, and learning photography with the GF1 is great and deserves a Unicode snowman: ☃.

The reason you want the camera is to use the 20mm f/1.7 lens. For the non-photography geeks, that means that it’s not a zoom lens, i.e. you can’t zoom in and out with it, and the f/1.7 means that it can take pretty good shots in low light without a flash. All the reviews of it are right: that lens is what makes the camera so fantastic. Do not even bother with 14-45mm kit lens. The 20mm lens is fast enough that you can shoot with it at night, and the focal length’s versatile enough to take both close-up/portrait shots (whee food porn), and swing out a bit wider for landscape photos or group photos. It’s no wide-angle nor zoom and it’s not a super-fast f/1.4, but it’s versatile enough and so tiny that you will end up using it almost all the time. It feels weird to get a DSLR and only have one lens for it, but the pancake 20mm lens is so damn good that it’s all you really need. The only thing it really can’t do at all is go super-zoomalicious, for wildlife/distance shots.

The 20mm non-zoom (a.k.a. “prime”) lens has another advantage: it teaches you to compose. Despite all the technology and all the geek speak, photography is ultimately about your composition skills. Prime lenses force you to move around to find the perfect framing for the shot you’re trying to do; I think learning with a prime lens moves your composition skills along much faster than it would if you were using a standard zoom lens. If you’re a photography beginner, like me, shoot with a prime. It’s a totally different experience than shooting with a zoom, and you learn a lot more. Plus, primes are cheap: the Canon 50mm f/1.8 is USD$100, and Canon’s near top-of-the-line 50mm f/1.4 is USD$350. The Canon 35mm f/2, for something that’s similar in focal length to the Panasonic 20mm prime, is USD$300. (You need to multiply the 20mm by 2 to convert between micro four-thirds and 35mm framing, so the actual focal length of the 20mm in traditional camera speak is 20mm*2=40mm.)

After playing it for a few months, you realise that the GF1 is a fun camera to use. The combination of the 20mm prime lens, the super-fast focus, the size, and the great UI design just begs you to take pictures with it. You feel like you’re wielding a real camera rather than a toy: one that wants you to shoot with it. It’s not imposing like a bigger DSLR so it doesn’t feel like the camera is with you all the time, but it’s not so small that you feel like you’re just snipping super-casually with something that’s cheap. And did I mention the excellent UI? It’s excellent. The better controls are a good reason to get the GF1 over its rivals, the Olympus EP series.

One big bonus: I’ve found that the full-auto mode (“iAuto” as Panasonic brands it) very rarely gets stuff wrong. This is useful if you hand the camera over to someone else who doesn’t know how to use DSLRs so that they can take a picture of you… or, like me, if you just don’t know quite what aperture/shutter speeds to use for the particular shot you’re taking. The full-auto just adds to the joy of using it. I usually shoot in full-auto or aperture priority mode, but honestly, I could probably shoot on full-auto mode all the time. I can’t recall a single occasion where it didn’t guess f/1.7 or a landscape correctly.

Do follow DPreview and Craig Mod’s advice and shoot RAW, not JPEG. Honestly, I’d prefer to shoot JPEG if I could, but RAW lets you turn some bad shots into good shots. I use it because gives you a second chance, not because I want to maximise picture quality. Here’s one photo that was accidentally taken with the wrong settings: I had the camera on full-manual mode, and didn’t realise that the shutter speed and ISO settings were totally incorrect.

However, since I shot it in RAW, I could lift up the exposure up two stops, pulled up the shadows and pulled down the highlights, and here’s the result:

Seriously, that’s just frickin’ amazing. Sure, that photo might not be super-awesome: it’s a little grainy, and it looks a bit post-processed if you squint right, but it’s still a photo of a precious memory that I wouldn’t have otherwise had, and you know what? That photo’s just fine. If I shot JPEG, I would’ve had no choice but to throw it away. RAW’s a small pain in the arse since the file sizes are far bigger and you need to wait a long time for your computer to do the RAW processing if you’ve taken hundreds of photos, but boy, it’s worth it.

I did finally buy a wide-angle lens for the GF1—the Olympus 9-18mm f/4-5.6—and have been using it a lot for landscape shots. I bought the Olympus 9-18mm over the Panasonic 7-14 f/4.0 because it was cheaper, and also smaller. I figured that if I was getting a GF1, it was because I wanted something compact, so I wanted to keep the lenses as small as possible. (Otherwise, if you don’t care about size, then a full-blown Canon or Nikon DSLR would probably serve you much better.) I’ve always wanted a wide-angle lens from the first day that I asked “how do those real estate agents make those rooms look so bloody large?”, so now I have one, woohoo. The next lens on my shopping will probably be the Panasonic 45-200mm. (Never mind the quality, feel the price!) Here’s a shot taken with the Olympus 9-18mm; click through to see the original picture on Flickr.

The main thing that I wish for in a future version of the camera would be image stabilisation. Panasonic follow the Canon path and put image stabilisation in the lens, rather than in the body. I think Olympus made the right decision by putting image stabilisation in the body for their compact DSLRs; you can keep the lenses smaller that way, and you then get image stabilisation with all your lenses instead of the ones that only support it explicitly, e.g. the 20mm f/1.7 prime doesn’t have image stabilision, boo. In-body image stabilisation just seems more in-line with the size reduction goal for micro four-thirds cameras. I’d love to get my hands on an Olympus EP for a week and shoot with the 20mm to see if image stabilisation makes a difference when it’s dark and the environment is starting to challenge the f/1.7 speeds.

The only other thing I wish for would be a better sensor. The GF1’s great up to ISO 800: at ISO 1600+, it starts getting grainy. 1600 is acceptable, and you can do wondrous things with the modern noise reduction algorithms that are in Lightroom if you really need to save a shot. Shoot at ISO 3200+ though, and it’s just too grainy. This is the main advantage that more traditional DSLRs have: their larger sensors are simply better than the GF1’s. I’ve seen shots taken with a Nikon D50 at ISO 6400 in the dark because a slower f/4 lens was being used, and bam, the shot comes out fine. Don’t even try to compare this thing to a Canon 5D Mk II. The GF1 just can’t do high ISO. Here’s an ISO 3200 shot, which is just starting to get a little too grainy. It’s fine for Facebook-sized images, but if you click through to the original, you’ll see it’s noisy.

But y’know, despite the two nitpicks above, the GF1 is a fine camera, and the 20mm f/1.7 lens is an amazing do-it-all lens that’s absolutely perfect to learn with. There’s really nothing else out there like it except for the Olympus EP range (the EP-1, EP-2 and EPL-1), which you should definitely consider, but get it with the 20mm f/1.7 lens if you do. I’ve had a total blast learning photography with the GF1, and I’ve captured hundreds of memories in the past year that made the investment completely worthwhile. I don’t think I’m at the point yet where I feel like I need another camera yet, but it feels good knowing that the micro four-thirds format will be around for a while so that I can use my existing lenses with future cameras I buy. If you’re interested in learning photography, the GF1 is a fantastic starting point.

Update: Thom Hogan did a comparison between the most popular mirrorless cameras: the Olympus E-PL1, the Panasonic GF1, Samsung NX100, and Sony NEX-5. It’s written for people who know photography rather than for novices, but basically, the GF1 came out on top, with the E-PL1 being recommended if you can live with the worse screen and the far worse UI. That’s pretty much exactly my opinion, too.

Thanks to several evil friends of mine, I started to take an interest in photography at the end of last year. I’ve always wanted to have a “real” camera instead of a point and shoot, so at the start of 2010, I bit the bullet and bought myself a Panasonic Lumix DMC-GF1, usually just called The GF1 amongst the camera geeks.

I tossed up between the GF1 and the then-just-released Canon EOS 550D (a.k.a. the Rebel T2i in the USA) for a long time. I figured that getting a compact camera would make me tote it around a lot more, and after ten months of using it, I think I was right. I recently went to a wedding in Sydney, and I literally kept the camera in my suit pocket instead of having to lug it around strapped to my body or neck. I definitely wouldn’t be able to do that with a Canon or Nikon DSLR. The camera’s so small with the kit 20mm f/1.7 lens that I stopped using the UV filter with it, because I didn’t like the 2-3mm that the filter added to the camera depth. Here’s a size comparison of the Nikon D3000 vs the GF1.

(Image stolen from dpreview.com’s review.)

I won’t write up a comprehensive review of the GF1 here; other sites have done that, in much more depth than I can be bothered to go into. If you’re after a good review, the three articles that swayed me to the GF1 in the end were DPreview’s review, and Craig Mod’s GF1 photo field test article and video tests. What follows is my own impressions and experiences of using the camera. The one-sentence summary: the GF1 perfect for a DSLR newbie like me, the micro four-thirds lens system it uses looks like it has enough legs that your lens investments will be good for the future, and learning photography with the GF1 is great and deserves a Unicode snowman: ☃.

The reason you want the camera is to use the 20mm f/1.7 lens. For the non-photography geeks, that means that it’s not a zoom lens, i.e. you can’t zoom in and out with it, and the f/1.7 means that it can take pretty good shots in low light without a flash. All the reviews of it are right: that lens is what makes the camera so fantastic. Do not even bother with 14-45mm kit lens. The 20mm lens is fast enough that you can shoot with it at night, and the focal length’s versatile enough to take both close-up/portrait shots (whee food porn), and swing out a bit wider for landscape photos or group photos. It’s no wide-angle nor zoom and it’s not a super-fast f/1.4, but it’s versatile enough and so tiny that you will end up using it almost all the time. It feels weird to get a DSLR and only have one lens for it, but the pancake 20mm lens is so damn good that it’s all you really need. The only thing it really can’t do at all is go super-zoomalicious, for wildlife/distance shots.

The 20mm non-zoom (a.k.a. “prime”) lens has another advantage: it teaches you to compose. Despite all the technology and all the geek speak, photography is ultimately about your composition skills. Prime lenses force you to move around to find the perfect framing for the shot you’re trying to do; I think learning with a prime lens moves your composition skills along much faster than it would if you were using a standard zoom lens. If you’re a photography beginner, like me, shoot with a prime. It’s a totally different experience than shooting with a zoom, and you learn a lot more. Plus, primes are cheap: the Canon 50mm f/1.8 is USD$100, and Canon’s near top-of-the-line 50mm f/1.4 is USD$350. The Canon 35mm f/2, for something that’s similar in focal length to the Panasonic 20mm prime, is USD$300. (You need to multiply the 20mm by 2 to convert between micro four-thirds and 35mm framing, so the actual focal length of the 20mm in traditional camera speak is 20mm*2=40mm.)

After playing it for a few months, you realise that the GF1 is a fun camera to use. The combination of the 20mm prime lens, the super-fast focus, the size, and the great UI design just begs you to take pictures with it. You feel like you’re wielding a real camera rather than a toy: one that wants you to shoot with it. It’s not imposing like a bigger DSLR so it doesn’t feel like the camera is with you all the time, but it’s not so small that you feel like you’re just snipping super-casually with something that’s cheap. And did I mention the excellent UI? It’s excellent. The better controls are a good reason to get the GF1 over its rivals, the Olympus EP series.

One big bonus: I’ve found that the full-auto mode (“iAuto” as Panasonic brands it) very rarely gets stuff wrong. This is useful if you hand the camera over to someone else who doesn’t know how to use DSLRs so that they can take a picture of you… or, like me, if you just don’t know quite what aperture/shutter speeds to use for the particular shot you’re taking. The full-auto just adds to the joy of using it. I usually shoot in full-auto or aperture priority mode, but honestly, I could probably shoot on full-auto mode all the time. I can’t recall a single occasion where it didn’t guess f/1.7 or a landscape correctly.

Do follow DPreview and Craig Mod’s advice and shoot RAW, not JPEG. Honestly, I’d prefer to shoot JPEG if I could, but RAW lets you turn some bad shots into good shots. I use it because gives you a second chance, not because I want to maximise picture quality. Here’s one photo that was accidentally taken with the wrong settings: I had the camera on full-manual mode, and didn’t realise that the shutter speed and ISO settings were totally incorrect.

However, since I shot it in RAW, I could lift up the exposure up two stops, pulled up the shadows and pulled down the highlights, and here’s the result:

Seriously, that’s just frickin’ amazing. Sure, that photo might not be super-awesome: it’s a little grainy, and it looks a bit post-processed if you squint right, but it’s still a photo of a precious memory that I wouldn’t have otherwise had, and you know what? That photo’s just fine. If I shot JPEG, I would’ve had no choice but to throw it away. RAW’s a small pain in the arse since the file sizes are far bigger and you need to wait a long time for your computer to do the RAW processing if you’ve taken hundreds of photos, but boy, it’s worth it.

I did finally buy a wide-angle lens for the GF1—the Olympus 9-18mm f/4-5.6—and have been using it a lot for landscape shots. I bought the Olympus 9-18mm over the Panasonic 7-14 f/4.0 because it was cheaper, and also smaller. I figured that if I was getting a GF1, it was because I wanted something compact, so I wanted to keep the lenses as small as possible. (Otherwise, if you don’t care about size, then a full-blown Canon or Nikon DSLR would probably serve you much better.) I’ve always wanted a wide-angle lens from the first day that I asked “how do those real estate agents make those rooms look so bloody large?”, so now I have one, woohoo. The next lens on my shopping will probably be the Panasonic 45-200mm. (Never mind the quality, feel the price!) Here’s a shot taken with the Olympus 9-18mm; click through to see the original picture on Flickr.

The main thing that I wish for in a future version of the camera would be image stabilisation. Panasonic follow the Canon path and put image stabilisation in the lens, rather than in the body. I think Olympus made the right decision by putting image stabilisation in the body for their compact DSLRs; you can keep the lenses smaller that way, and you then get image stabilisation with all your lenses instead of the ones that only support it explicitly, e.g. the 20mm f/1.7 prime doesn’t have image stabilision, boo. In-body image stabilisation just seems more in-line with the size reduction goal for micro four-thirds cameras. I’d love to get my hands on an Olympus EP for a week and shoot with the 20mm to see if image stabilisation makes a difference when it’s dark and the environment is starting to challenge the f/1.7 speeds.

The only other thing I wish for would be a better sensor. The GF1’s great up to ISO 800: at ISO 1600+, it starts getting grainy. 1600 is acceptable, and you can do wondrous things with the modern noise reduction algorithms that are in Lightroom if you really need to save a shot. Shoot at ISO 3200+ though, and it’s just too grainy. This is the main advantage that more traditional DSLRs have: their larger sensors are simply better than the GF1’s. I’ve seen shots taken with a Nikon D50 at ISO 6400 in the dark because a slower f/4 lens was being used, and bam, the shot comes out fine. Don’t even try to compare this thing to a Canon 5D Mk II. The GF1 just can’t do high ISO. Here’s an ISO 3200 shot, which is just starting to get a little too grainy. It’s fine for Facebook-sized images, but if you click through to the original, you’ll see it’s noisy.

But y’know, despite the two nitpicks above, the GF1 is a fine camera, and the 20mm f/1.7 lens is an amazing do-it-all lens that’s absolutely perfect to learn with. There’s really nothing else out there like it except for the Olympus EP range (the EP-1, EP-2 and EPL-1), which you should definitely consider, but get it with the 20mm f/1.7 lens if you do. I’ve had a total blast learning photography with the GF1, and I’ve captured hundreds of memories in the past year that made the investment completely worthwhile. I don’t think I’m at the point yet where I feel like I need another camera yet, but it feels good knowing that the micro four-thirds format will be around for a while so that I can use my existing lenses with future cameras I buy. If you’re interested in learning photography, the GF1 is a fantastic starting point.

Update: Thom Hogan did a comparison between the most popular mirrorless cameras: the Olympus E-PL1, the Panasonic GF1, Samsung NX100, and Sony NEX-5. It’s written for people who know photography rather than for novices, but basically, the GF1 came out on top, with the E-PL1 being recommended if you can live with the worse screen and the far worse UI. That’s pretty much exactly my opinion, too.

Filesystems are fundamental things for computer systems: after all, you need to store your data somewhere, somehow. Modern operating systems largely use the same concepts for filesystems: a file’s just a bucket that holds some bytes, and files are organised into directories, which can be hierarchical. Some fancier filesystems keep track of file versions and have record-based I/O, and many filesystems now have multiple streams and extended attributes. However, filesystem organisation ideas have stayed largely the same over the past few decades.

I’d argue that the most important stuff on your machine is your data. There are designated places to put this data. On Windows, all the most important stuff on my computer should really live in C:\Documents and Settings\Andre Pang\My Documents. In practice, because of lax permissions, almost everyone I know doesn’t store their documents there: they splatter it over a bunch of random directories hanging off C:\, resulting in a giant lovely mess of directories where some are owned by applications and the system, and some are owned by the user.

Mac OS X is better, but only because I have discipline: /Users/andrep is where I put my stuff. However, I’ve seen plenty of Mac users have data and personal folders hanging off their root directory. I’m pretty sure that my parents have no idea that /Users/your-name-here is where you are meant to put your stuff, and to this day, I’m not quite sure where my dad keeps all his important documents on his Mac. I hope it’s in ~/Documents, but if not, can I blame him? (UNIX only gets this right because it enforces permissions on you. Try saving to / and it won’t work. If you argue that this is a good thing, you’re missing the point of this entire article.)

One OS that actually got this pretty much correct was classic Mac OS: all system stuff went into the System folder (which all the Cool Kids named “System ƒ”, of course). The entire system essentials were contained in just two files: System, and Finder, and you could even copy those files to any floppy disk and make a bootable disk (wow, imagine that). The entire rest of the filesystem was yours: with the exception of the System folder, you organised the file system as you pleased, rather than the filesystem enforcing a hierarchy on you. The stark difference in filesystem organisation between classic Mac OS and Mac OS X is largely due to a user-centric approach for Mac OS from the ground-up, whereas Mac OS X had to carry all its UNIX weight with it, so it had to compromise and use a more traditionally organised computer filesystem.

As an example, in Mac OS X, if you want to delete Photoshop’s preferences, you delete the ~/Library/Preferences/com.adobe.Photoshop.plist file. Or; maybe you should call it the Bibliothèque folder in France (because that’s what it’s displayed as in the Finder if you switch to French)… and why isn’t the Preferences folder name localised too, and what mere mortal is going to understand why it’s called com.adobe.Photoshop.plist? On a technical level, I completely understand why the Photoshop preferences file is in the ~/Library/Preferences/ directory. But at a user experience level, this is a giant step backwards from Mac OS, where you simply went to the System folder and you trashed the Adobe Photoshop Preferences file there. How is this progress?

I think the fundamental problem is that Windows Explorer, Finder, Nautilus and all the other file managers in the world are designed, by definition, to browse the filesystem. However, what we really want is an abstraction level for users that hides the filesystem from them, and only shows them relevant material, organised in a way that’s sensible for them. The main “file managers” on desktop OSs (Finder and Windows Explorer) should be operating at an abstraction level above the filesystem. The operating system should figure out where to put files on a technical (i.e. filesystem) level, but the filesystem hierarchy should be completely abstracted so that a user doesn’t even realise their stuff is going into /Users/joebloggs.

iTunes and iPhoto are an example of what I’m advocating, because they deal with all the file management for you. You don’t need to worry where your music is or how your photos are organised on the filesystem: you just know about songs and photos. There’s no reason why this can’t work for other types of documents too, and there’s no reason why such an abstracted view of the filesystem can’t work on a systemwide basis. It’s time for the operating system to completely abstract out the filesystem from the user experience, and to turn our main interaction with our documents—i.e. the Finder, Windows Explorer et al—into something that abstracts away the geek details to a sensible view of the documents that are important to us.

One modern operating system has already done this: iOS. iOS is remarkable for being an OS that I often forget is a full-fledged UNIX at its heart. In the iOS user experience, the notion of files is completely gone: the only filenames you ever see are usually email attachments. You think about things as photos, notes, voice memos, mail, and media; not files. I’d argue that this is a huge reason that users find an iPhone and iPad much more simple than a “real” computer: the OS organises the files for them, so they don’t have to think that a computer deals with files. A computer deal with photos and music instead.

There are problems with the iOS approach: the enforced sandboxing per app means that you can’t share files between apps, which is one of the most powerful (and useful) aspects of desktop operating systems. This is a surmountable goal, though, and I don’t think it’d be a difficult task to store documents that can be shared between apps. After all, it’s what desktop OSs do today: the main challenge is in presenting a view of the files that are sensible for the user. I don’t think we can—nor should—banish files, since we still need to serialise all of a document’s data into a form that’s easily transportable. However, a file manager should be metadata-centric and display document titles, keywords, and tags rather than filenames. For many documents, you can derive a filename from its metadata that you can then use to transport the file around.

We’ve tried making the filesystem more amenable to a better user experience by adding features such as extended attributes (think Mac OS type/creator information), and querying and indexing features, ala BFS. However, with the additional complexity of issues such as display names (i.e. localisation), requiring directory hierarchies that should remain invisible to users, and the simple but massive baggage of supporting traditional filesystem structures (/bin/ and /lib/ aren’t going away anytime soon, and make good technical sense), I don’t think we can shoehorn a filesystem browser anymore into something that’s friendly for users. We need a filesystem abstraction layer that’s system-wide. iOS has proven that it can be done. With Apple’s relentless progress march and willingness to change system APIs, Linux’s innovation in the filesystem arena and experimentation with the desktop computing metaphor, and Microsoft’s ambitious plans for Windows 8, maybe we can achieve this sometime in the next decade.

Filesystems are fundamental things for computer systems: after all, you need to store your data somewhere, somehow. Modern operating systems largely use the same concepts for filesystems: a file’s just a bucket that holds some bytes, and files are organised into directories, which can be hierarchical. Some fancier filesystems keep track of file versions and have record-based I/O, and many filesystems now have multiple streams and extended attributes. However, filesystem organisation ideas have stayed largely the same over the past few decades.

I’d argue that the most important stuff on your machine is your data. There are designated places to put this data. On Windows, all the most important stuff on my computer should really live in C:\Documents and Settings\Andre Pang\My Documents. In practice, because of lax permissions, almost everyone I know doesn’t store their documents there: they splatter it over a bunch of random directories hanging off C:\, resulting in a giant lovely mess of directories where some are owned by applications and the system, and some are owned by the user.

Mac OS X is better, but only because I have discipline: /Users/andrep is where I put my stuff. However, I’ve seen plenty of Mac users have data and personal folders hanging off their root directory. I’m pretty sure that my parents have no idea that /Users/your-name-here is where you are meant to put your stuff, and to this day, I’m not quite sure where my dad keeps all his important documents on his Mac. I hope it’s in ~/Documents, but if not, can I blame him? (UNIX only gets this right because it enforces permissions on you. Try saving to / and it won’t work. If you argue that this is a good thing, you’re missing the point of this entire article.)

One OS that actually got this pretty much correct was classic Mac OS: all system stuff went into the System folder (which all the Cool Kids named “System ƒ”, of course). The entire system essentials were contained in just two files: System, and Finder, and you could even copy those files to any floppy disk and make a bootable disk (wow, imagine that). The entire rest of the filesystem was yours: with the exception of the System folder, you organised the file system as you pleased, rather than the filesystem enforcing a hierarchy on you. The stark difference in filesystem organisation between classic Mac OS and Mac OS X is largely due to a user-centric approach for Mac OS from the ground-up, whereas Mac OS X had to carry all its UNIX weight with it, so it had to compromise and use a more traditionally organised computer filesystem.

As an example, in Mac OS X, if you want to delete Photoshop’s preferences, you delete the ~/Library/Preferences/com.adobe.Photoshop.plist file. Or; maybe you should call it the Bibliothèque folder in France (because that’s what it’s displayed as in the Finder if you switch to French)… and why isn’t the Preferences folder name localised too, and what mere mortal is going to understand why it’s called com.adobe.Photoshop.plist? On a technical level, I completely understand why the Photoshop preferences file is in the ~/Library/Preferences/ directory. But at a user experience level, this is a giant step backwards from Mac OS, where you simply went to the System folder and you trashed the Adobe Photoshop Preferences file there. How is this progress?

I think the fundamental problem is that Windows Explorer, Finder, Nautilus and all the other file managers in the world are designed, by definition, to browse the filesystem. However, what we really want is an abstraction level for users that hides the filesystem from them, and only shows them relevant material, organised in a way that’s sensible for them. The main “file managers” on desktop OSs (Finder and Windows Explorer) should be operating at an abstraction level above the filesystem. The operating system should figure out where to put files on a technical (i.e. filesystem) level, but the filesystem hierarchy should be completely abstracted so that a user doesn’t even realise their stuff is going into /Users/joebloggs.

iTunes and iPhoto are an example of what I’m advocating, because they deal with all the file management for you. You don’t need to worry where your music is or how your photos are organised on the filesystem: you just know about songs and photos. There’s no reason why this can’t work for other types of documents too, and there’s no reason why such an abstracted view of the filesystem can’t work on a systemwide basis. It’s time for the operating system to completely abstract out the filesystem from the user experience, and to turn our main interaction with our documents—i.e. the Finder, Windows Explorer et al—into something that abstracts away the geek details to a sensible view of the documents that are important to us.

One modern operating system has already done this: iOS. iOS is remarkable for being an OS that I often forget is a full-fledged UNIX at its heart. In the iOS user experience, the notion of files is completely gone: the only filenames you ever see are usually email attachments. You think about things as photos, notes, voice memos, mail, and media; not files. I’d argue that this is a huge reason that users find an iPhone and iPad much more simple than a “real” computer: the OS organises the files for them, so they don’t have to think that a computer deals with files. A computer deal with photos and music instead.

There are problems with the iOS approach: the enforced sandboxing per app means that you can’t share files between apps, which is one of the most powerful (and useful) aspects of desktop operating systems. This is a surmountable goal, though, and I don’t think it’d be a difficult task to store documents that can be shared between apps. After all, it’s what desktop OSs do today: the main challenge is in presenting a view of the files that are sensible for the user. I don’t think we can—nor should—banish files, since we still need to serialise all of a document’s data into a form that’s easily transportable. However, a file manager should be metadata-centric and display document titles, keywords, and tags rather than filenames. For many documents, you can derive a filename from its metadata that you can then use to transport the file around.

We’ve tried making the filesystem more amenable to a better user experience by adding features such as extended attributes (think Mac OS type/creator information), and querying and indexing features, ala BFS. However, with the additional complexity of issues such as display names (i.e. localisation), requiring directory hierarchies that should remain invisible to users, and the simple but massive baggage of supporting traditional filesystem structures (/bin/ and /lib/ aren’t going away anytime soon, and make good technical sense), I don’t think we can shoehorn a filesystem browser anymore into something that’s friendly for users. We need a filesystem abstraction layer that’s system-wide. iOS has proven that it can be done. With Apple’s relentless progress march and willingness to change system APIs, Linux’s innovation in the filesystem arena and experimentation with the desktop computing metaphor, and Microsoft’s ambitious plans for Windows 8, maybe we can achieve this sometime in the next decade.

As GLaDOS would say, It’s been a long time. How have you been?

I was invited a few weeks ago to keynote at /dev/world 2010, a conference for Mac developers in Australia. It was my first-ever keynote, and you know, inspirational talks turn out to be kinda harder to give than technical talks. For those who didn’t attend, the talk intended to address the two most frequent questions I get asked about Pixar (“how did you get there?” and “what do you, uhh, actually do?”), and provide some insight into Pixar’s culture.

Two videos that I referred to in the talk and I think are an absolute must-see—whether you’re an engineer, CEO, manager, designer, artist or otherwise—are

• Ed Catmull’s talk at Stanford Business School: Keep Your Crises Small (~40 minutes, minus question time)
• Randy Nelson’s Learning and Working in the Collaborative Age (~10 minutes)

They complement Steve Jobs’s amazing commencement speech at Stanford in 2005. The number of insightful, genuine sound bites you can take away from those three talks are off the charts.

For all those who attended my keynote, I hope it was worthwhile. Thank you to everyone in the audience for giving me such a warm reception and for making me feel back at home, and thank you to the AUC for putting on a great conference.

Also, new website look, oooooo. It almost looks like it was made after 2000. Hopefully this will mean more regular updates on my blog than once per decade. I guess we’ll find out!

As GLaDOS would say, It’s been a long time. How have you been?

I was invited a few weeks ago to keynote at /dev/world 2010, a conference for Mac developers in Australia. It was my first-ever keynote, and you know, inspirational talks turn out to be kinda harder to give than technical talks. For those who didn’t attend, the talk intended to address the two most frequent questions I get asked about Pixar (“how did you get there?” and “what do you, uhh, actually do?”), and provide some insight into Pixar’s culture.

Two videos that I referred to in the talk and I think are an absolute must-see—whether you’re an engineer, CEO, manager, designer, artist or otherwise—are

• Ed Catmull’s talk at Stanford Business School: Keep Your Crises Small (~40 minutes, minus question time)
• Randy Nelson’s Learning and Working in the Collaborative Age (~10 minutes)

They complement Steve Jobs’s amazing commencement speech at Stanford in 2005. The number of insightful, genuine sound bites you can take away from those three talks are off the charts.

For all those who attended my keynote, I hope it was worthwhile. Thank you to everyone in the audience for giving me such a warm reception and for making me feel back at home, and thank you to the AUC for putting on a great conference.

Also, new website look, oooooo. It almost looks like it was made after 2000. Hopefully this will mean more regular updates on my blog than once per decade. I guess we’ll find out!

I feel like there’s been four stages to my life. The first stage was being a youngling at primary school: I don’t remember much from there except that I fantasised about handball being an olympic sport. The second stage was the PC demoscene, where I grew interested in many things that I love today about computing: art, music, and my first experience with a community and culture that you could love and immerse yourself in. The third stage was my twenties: an introduction to university, Linux, coding, the Mac, Haskell, research conferences, industry conferences, the working life, and balancing it all with healthy doses of relaxation, food and the beautiful world that Sydney had to offer. The fourth stage was tearing myself away from that fairly sheltered life and my emotional base, and moving to San Francisco.

I’ve been here for six months. It’s felt like two years. It has been a truly wonderful experience: making new friends, learning a new culture that’s both significantly but subtly different, and doing it all without my family nearby, who’ve been my anchor and support for the past three decades. Part of the motivation was proving to myself that I could make it on my own: prove myself worthy in the eyes of my peers, be social enough to make genuine friends here who I cared about and who cared about me, living on my own and simply paying the rent. Part of the motivation was to shake things up from a cruisy life in Sydney and experience new things. I’m glad to report that the experiment’s going pretty well so far.

San Francisco is a city of immense contrast. For every stupid hipster who thinks that owning a Prius absolves them of their environmental debt to society, there are remarkable individuals who understand and challenge the daunting realism of politics, lobbying, energy, transformity and limits to growth. For every poser upstart get-rich-quick guy chasing after VC funding for Facebook apps, there are the quiet anonymous developers at Apple, Google, and startups you’ve never heard of who work on all the amazing technologies that the entire world takes for granted today. The Tenderloin, so unpleasant to walk through, has some of the very best restaurants and bars that the city has to offer. The nouveau shiny high-rises of South Beach contrast with the destitute run-down feel of western SoMa, only a few blocks away.

It’s a make-or-break city: rents are insanely high despite the rent control laws, and there’s no lower-middle class population here because either you’re flying high, or you’re not flying at all. It’s natural selection in action: either you keep up with the pack and continue being successful, or you fall and become left behind. And so, in contrast to the relaxed lifestyle of Sydney, San Francisco is full of ambition. While it lacks the non-stop pace of New York or the late-night industry of Detroit and Chicago, the people here want to change the world, and they have the innovation, the smarts and the determination to do so.

The tech industry here is simply amazing. Despite being here for half a year, I’m still floored when I go to a party and every person I meet there ends up being a Web designer, or a coder, or a sysadmin, or a DBA, or a network engineer, or a manager of a bunch of coders, or a VC funding a tech company, or a lawyer or accountant or marketing or PR person working for a tech company, or a level designer or artist working for a games company. Even the girls. It boggles me. It’s like the entire Bay Area simply exists to build software and iPhones and tech solutions. I was truly daunted in the first few months to find out that everyone around me was, well, just like me. A few months ago, I was at my favourite little tea shop in San Francisco decompressing and minding my own business, when three people sat down next to me and started talking about VGA BIOS exploits. (Turns out that they work for VMware.) I mean, seriously?

I wouldn’t say that I’m totally acclimated to the Bay Area yet, and perhaps I never will be. Visiting Australia just a month ago reminded me just how different the two cities are in their lifestyles. People are always doing something in San Francisco: there’s so many interesting people there that you feel like need to divide your time between groups, let alone having time to yourself. Even the serious introverts there are out on most schoolnights. The people here are always switched on; even at a party, there’s an air of networking going on and the feeling of opportunities to be seized. You almost always end up talking shop at any event, because people here are defined by what they do: one of the very first questions you’re usually asked is “Where do you work?” or “What do you do for a living?”. In Sydney, asking that question so soon would just be a little bit weird. You usually save that for far later in the conversation, when you’re running out of things to say to the pretty girl to try to hook up with her. (And don’t even get me started about the American dating scene.)

And so, for all the wonderful parks, bars, tacos, restaurants, pirate shops and museums of the city; the incredible beauty and varied terrain of the North Bay; the charm and chilled suburbia of North Berkeley in the East; and the innovation and serenity of Silicon Valley just south, I still miss Sydney and the culture I grew up with for twenty years. I don’t mean that in a yearning way or mean to imply that San Francisco is somehow inadequate, because it rocks: I’m having a wonderful time experiencing new things, and it was the right decision to move here. This is where I should be at this stage in my life. Sydney will always be where my heart is, but right now, San Francisco is home, and it’s as fantastic as I hoped it would be.

I feel like there’s been four stages to my life. The first stage was being a youngling at primary school: I don’t remember much from there except that I fantasised about handball being an olympic sport. The second stage was the PC demoscene, where I grew interested in many things that I love today about computing: art, music, and my first experience with a community and culture that you could love and immerse yourself in. The third stage was my twenties: an introduction to university, Linux, coding, the Mac, Haskell, research conferences, industry conferences, the working life, and balancing it all with healthy doses of relaxation, food and the beautiful world that Sydney had to offer. The fourth stage was tearing myself away from that fairly sheltered life and my emotional base, and moving to San Francisco.

I’ve been here for six months. It’s felt like two years. It has been a truly wonderful experience: making new friends, learning a new culture that’s both significantly but subtly different, and doing it all without my family nearby, who’ve been my anchor and support for the past three decades. Part of the motivation was proving to myself that I could make it on my own: prove myself worthy in the eyes of my peers, be social enough to make genuine friends here who I cared about and who cared about me, living on my own and simply paying the rent. Part of the motivation was to shake things up from a cruisy life in Sydney and experience new things. I’m glad to report that the experiment’s going pretty well so far.

San Francisco is a city of immense contrast. For every stupid hipster who thinks that owning a Prius absolves them of their environmental debt to society, there are remarkable individuals who understand and challenge the daunting realism of politics, lobbying, energy, transformity and limits to growth. For every poser upstart get-rich-quick guy chasing after VC funding for Facebook apps, there are the quiet anonymous developers at Apple, Google, and startups you’ve never heard of who work on all the amazing technologies that the entire world takes for granted today. The Tenderloin, so unpleasant to walk through, has some of the very best restaurants and bars that the city has to offer. The nouveau shiny high-rises of South Beach contrast with the destitute run-down feel of western SoMa, only a few blocks away.

It’s a make-or-break city: rents are insanely high despite the rent control laws, and there’s no lower-middle class population here because either you’re flying high, or you’re not flying at all. It’s natural selection in action: either you keep up with the pack and continue being successful, or you fall and become left behind. And so, in contrast to the relaxed lifestyle of Sydney, San Francisco is full of ambition. While it lacks the non-stop pace of New York or the late-night industry of Detroit and Chicago, the people here want to change the world, and they have the innovation, the smarts and the determination to do so.

The tech industry here is simply amazing. Despite being here for half a year, I’m still floored when I go to a party and every person I meet there ends up being a Web designer, or a coder, or a sysadmin, or a DBA, or a network engineer, or a manager of a bunch of coders, or a VC funding a tech company, or a lawyer or accountant or marketing or PR person working for a tech company, or a level designer or artist working for a games company. Even the girls. It boggles me. It’s like the entire Bay Area simply exists to build software and iPhones and tech solutions. I was truly daunted in the first few months to find out that everyone around me was, well, just like me. A few months ago, I was at my favourite little tea shop in San Francisco decompressing and minding my own business, when three people sat down next to me and started talking about VGA BIOS exploits. (Turns out that they work for VMware.) I mean, seriously?

I wouldn’t say that I’m totally acclimated to the Bay Area yet, and perhaps I never will be. Visiting Australia just a month ago reminded me just how different the two cities are in their lifestyles. People are always doing something in San Francisco: there’s so many interesting people there that you feel like need to divide your time between groups, let alone having time to yourself. Even the serious introverts there are out on most schoolnights. The people here are always switched on; even at a party, there’s an air of networking going on and the feeling of opportunities to be seized. You almost always end up talking shop at any event, because people here are defined by what they do: one of the very first questions you’re usually asked is “Where do you work?” or “What do you do for a living?”. In Sydney, asking that question so soon would just be a little bit weird. You usually save that for far later in the conversation, when you’re running out of things to say to the pretty girl to try to hook up with her. (And don’t even get me started about the American dating scene.)

And so, for all the wonderful parks, bars, tacos, restaurants, pirate shops and museums of the city; the incredible beauty and varied terrain of the North Bay; the charm and chilled suburbia of North Berkeley in the East; and the innovation and serenity of Silicon Valley just south, I still miss Sydney and the culture I grew up with for twenty years. I don’t mean that in a yearning way or mean to imply that San Francisco is somehow inadequate, because it rocks: I’m having a wonderful time experiencing new things, and it was the right decision to move here. This is where I should be at this stage in my life. Sydney will always be where my heart is, but right now, San Francisco is home, and it’s as fantastic as I hoped it would be.

Just before I left Sydney, I gave one last talk at the revived Sydney Cocoaheads user group about Objective-C Internals. It’s similar to the presentation that I gave at fp-syd a few months ago about Objective-C and Mac OS X programming, but was tailored for a Mac audience rather than a functional programming audience. As a result, the Cocoaheads talk has a lot more detail on the object model, memory layout, and how message-sending works, and less info on higher-order messaging and language features (e.g. I didn’t talk about categories at all.)

If you’re a Mac coder, hopefully you’ll find something new in there. As always, drop me an email if you have any questions!

P.S. For all the voyeurs out there, the San Francisco move & Pixar are both going great! More news on that soon, too.

Just before I left Sydney, I gave one last talk at the revived Sydney Cocoaheads user group about Objective-C Internals. It’s similar to the presentation that I gave at fp-syd a few months ago about Objective-C and Mac OS X programming, but was tailored for a Mac audience rather than a functional programming audience. As a result, the Cocoaheads talk has a lot more detail on the object model, memory layout, and how message-sending works, and less info on higher-order messaging and language features (e.g. I didn’t talk about categories at all.)

If you’re a Mac coder, hopefully you’ll find something new in there. As always, drop me an email if you have any questions!

P.S. For all the voyeurs out there, the San Francisco move & Pixar are both going great! More news on that soon, too.

I think my current plan of being self-employed is arguably working out pretty well. I get to work on RapidWeaver and LittleSnapper, two kick-ass products with a ton of users who love it. I’m friends with the lovely baristas and staff at my local café, where I normally work during the day. However, nothing beats taking a weekday off to chill out at my favourite café in Sydney, perhaps catching some waves at Bondi Beach and then playing some tennis afterwards, only to put my head down and code at night when the distractions are minimal. Life, as they say, is pretty peachy.

So, just before my 30th birthday, it was with both great apprehension and excitement that I made the decision to give up my current lifestyle and my current job. In five days, I move from the comfort of Sydney to magnificent San Francisco, to start work for a company that I’ve loved so much ever since I was a kid: Pixar.

To say that this was not where I expected to be in my life is quite an understatement. I was always the guy who thought that things would fall into place if he found the right girl, and that career would work itself out later. I fell in love hard when I was 21, was about this close to deciding to get married when I was 26, and when things went south, it took countless numbers of D&M talks with my close friends (thank you thank you thank you), another wonderful relationship with one of the most amazing people I know, and over three years to truly recover. I spent weeks in self-reflection pondering what life was about, what the day-to-day drudgery meant, and what I had accomplished during my second decade on Earth while so many of my friends were growing by leaps and bounds in their own relationships.

Meanwhile, my career was working out just fine. I worked on one project that will hopefully have the success it deserves with the release of Firefox 3.1 and perhaps take the lead in the important area of open-standards video on the Web. I’ve worked on other projects that are all tremendous successes in their respective markets, and along the way, I made a ton of genuine friends in the demoscene, Haskell, Linux and Mac OS X communities. Nonetheless, I still felt that I failed to meet my own expectations, since I’d never considered my career to be a measure of success. Despite the fantastic lifestyle that working remotely for Realmac afforded me, I was still restless, and still felt incomplete in my personal life.

Last year, I travelled a lot, not as a means of escapism, but because I had a ton of conferences to hit, and also wanted to visit some of my best friends who were now overseas. As I hopped from the UK to the USA to Singapore, I spent a lot of time alone, as travel does to you, and reflected on life. One day, I spent eight hours by myself in my favourite tea lounge in San Francisco, mixing feelings about the past decade and all its ups and downs: perhaps my expectations were too high, or perhaps I concentrated too much on things that would simply work themselves out. By the time I returned to Australia’s beautiful shores in December, I was exhausted from too much reflection, too much living out of a duffel bag on the road for months, and too much melancholy from thinking about the past and too much searching without answers.

However, something else happened in those months of travel: all of those worries slowly felt more and more like experience. I figured that, hey, my third decade was looming; either I start my thirties by continuing to be subtly haunted by those worries of the past decade, or I could treat those worries as learning experiences and forge a new outlook that relished any challenges the future could bring. As Steve Jobs says, “sometimes life hits you in the head with a brick. Don’t lose faith.” And so, just as I started to feel happy when I finally returned home, Pixar came knocking. I flew to San Francisco just a few days after arriving back in Sydney from months of travel; it would be my last trip for 2008. I did the interview thing, and a week later when I was back again in Sydney to wind down for Christmas, I got The Phone Call. When would I be able to move to San Francisco to start work there?

I thought hard about my current lifestyle of cafés and working remotely, and how I loved working at a small company like Realmac where I was directly responsible for the welfare of a much-loved application. I anguished over all the friends I had made in Australia and how much I’d miss them, and how much I’d miss my family. I thought about how I made my life here: how I went to school here, University here, met all my lovers here, and how this was, well, home. In the end, though, how could I turn down the opportunity of a lifetime, working at a company that married art and science so perfectly, and inspires so much love & passion in everyone? (Also, I hear there’s a lot of hot Asian-American girls in San Francisco.)

There’s no moral to this story: it just is, and I thought it needed telling while I still had the guts to tell it. So, as of next Wednesday, I leave my wonderful memories here to explore life in a new city. I’ve traditionally used this blog to communicate my thoughts on computing and technology, but I’ve always admired and enjoyed reading other people’s blogs that were much more personal (without being totally emo). Hopefully I’ll transform this blog a little to have a far more personal feel, so I can keep in touch with all the people I know & love around the world; thank you all for being a part of my life and enriching and defining me. See you on the other side of the Pacific!