High Leverage Development
March 26thDuring our third month of porting SftpDrive to OS X, it became clear that creating and maintaining a cross-platform codebase of high performance network and filesystem code would be far more effort than we had hoped. It wasn’t that the project was impossible, or even absurdly difficult. It just wasn’t any fun. Every time I looked at a #ifdef WIN32, it was even more clear the code was becoming much more tedious to debug and maintain.
It seems our work on Slingshot spoiled us. A few months working with Ruby, Objective-C and C# left us feeling happy and optimistic about programming—anything was possible! Needless to say, the tens of thousands of lines of procedural C in SftpDrive for Mac no longer brought about the same feeling of joy. It seemed unfair that hot-shot-web-developers, with their pretty MacBooks, got all the attention, and they got to use fun high-level languages like Ruby or Python. We were developing a truly useful piece of technology, but were stuck on Windows and spending more than 50% of our time dealing with “pedestrian” details like pointers, memory management, IRQLs, and IRPs.
Still, we fancied ourselves hardcore and kept at it even though it was hard (and sometimes boring). When it was time to write an auto updater for SftpDrive we spent hours upon hours searching on Google and MSDN trying to find a clean implementation that would work on a vanilla Windows 2000 installation. One option, WinInet was ridiculously ugly and verbose. Another option, WinHTTP, didn’t work on Windows 2000 GM. We ended up using libCURL. It was a ridiculous and frustrating waste of time.
We wanted to ‘import httplib’, and then just start making things happen. XKCD hits the nail on the head:
We couldn’t afford to keep spending time and energy writing software this way. Even if we could afford it, we didn’t want to spend our time this way. Web applications were being developed at an astounding pace in part because of centralized management and deployment (they never have to maintain different versions for Macs and PCs), but also because they were using modern interpreted languages. Web developers also used community-developed open-source projects when they needed some help on a routine problem. They didn’t have to reinvent the wheel at every turn, but instead focused on the core of their product. With high-level languages and good libraries, small teams can create great products at a rapid pace. We realized that we could write applications for the desktop in the exact same way.
We rewrote SftpDrive from top to bottom in Python, with a GUI in Objective-C. It’s called ExpanDrive, and it took 1/3rd the time that SftpDrive took to develop. We greatly leverage Python and and many open source projects—just like a web-developer. To minimize conflicts and to have the necessary control over the runtime environment, our build process extracts only the necessary bits from the full python distribution and packages it into the .app. We trim Python from 5000+ files to a few more than 400. Like many OS X apps, we use Sparkle.Framework to automatically distribute and install updates. We’re pushing out weekly updates which include more than just bug fixes. ExpanDrive has been a breeze to maintain and extend and the core remains perfectly cross platform.
Desktop applications aren’t dead, they’re just about to really get going.
March 26th, 2008 at 2:52 pm
Awesome story. I’m sure the web and the desktop will converge at some point. Let’s just hope it isn’t with silvercrap or air (adobe).
March 26th, 2008 at 9:52 pm
Do you use FUSE?
March 27th, 2008 at 1:40 pm
The only major problem with interpreted language is that it makes it really easier to steal the source of an app since it is not compiled.
March 27th, 2008 at 1:41 pm
This is indeed very much like web development. Our GUI’s are written in the same language (CSS) for each platform (browser), but require different very different rules. The desktop applications may be more ahead than we realize.
March 27th, 2008 at 1:45 pm
Jeedee: Not exactly - with Python as well as many others, you can compile and ship bytecode instead of source. It can still be decompiled, but shrug so can Java.
March 27th, 2008 at 2:22 pm
I totally dig the story, but one thing isn’t that clear to me.
Are you saying that now that you’ve:
(a) rewritten the core of the program in Python, and
(b) now it is a better cross paltform app?
So, are you planning on now taking the ExpandDrive codebase and porting it back to Windows using a different layer for the gui?
Just wondering. Good job on the program, though!
March 27th, 2008 at 3:34 pm
It usually takes less time to rewrite an app than to develop it for the first time.
March 27th, 2008 at 5:07 pm
How much of a downside is having to also ship the language environment with the app?
March 27th, 2008 at 5:38 pm
Outstanding! Am a happy ExpandDrive user. Good to hear Python getting used for system level stuff.
The desktop app I’ve been working on uses a slightly different architecture — Python, Django, Twisted, and SQLite, all rolled into a ‘microserver’ that runs in the background. The GUI can be anything you want as long as it talks HTTP. Cross-platform as all get out and the response is pretty decent. As you say, a joy to develop on.
BTW, that very XKCD cartoon is taped up on my wall :-)
March 27th, 2008 at 5:42 pm
“To minimize conflicts and to have the necessary control over the runtime environment, our build process extracts only the necessary bits from the full python distribution and packages it into the .app. We trim Python from 5000+ files to a few more than 400.”
Thanks very much for this very interesting post. Now, the above sentence could use a whole (geeky) other post, or even some separate source code sharing… you know, this whole “giving back to the community” ;-) No pressure, of course!
March 27th, 2008 at 8:19 pm
yeah，recently I have ported an MFC app to Mac OS X,and it leads me to realize how important it is to write cross platform codes.
March 28th, 2008 at 11:11 am
Great post and an outstanding work!
I’d also be interested in knowing how you made the trimmed down version of Python!
kudos
March 28th, 2008 at 2:21 pm
[...] the latest post on the Magnetk blog they make a good case for circumventing WIN32 and using a high level, cross [...]
March 28th, 2008 at 2:35 pm
@Mark: I’m not at all jealous of people who spent their days working on cross browser compatibility. Ugh. Still, desktop applications
@curios: That’s certainly in interesting idea, isn’t it?
@Charles, Emanuel: We’re going to do followup blog posts about some of the tools and techniques we’ve used, including how to create a dependancy extracted subset of python.
March 30th, 2008 at 11:13 am
We’re doing something similar over at Wooji Juice, using Cocoa and Python to make commercial Mac apps. I’ve written a post exploring our reasons in more detail, over at our website. The short version? Impedance mismatch — or the lack of it.
April 3rd, 2008 at 5:55 am
I couldn’t agree more. People can be very scathing of higher-level languages but to me it’s about the enjoyment of programming, if I didn’t care about that I would try to write everything in pure assembly.
April 6th, 2008 at 1:41 pm
[...] later on MacPorts, command-line magic and Python development on the Mac… Matthias posted this entry on Sunday, April 6th, 2008 at 8:41 pm. Posted in [...]
June 12th, 2008 at 5:27 pm
[...] an interesting side note, ExpanDrive is written in Python with only the GUI in [...]