Download the slides as a PDF document.
As is often the case when working on a startup, virtual or not, things don’t work out according to plan. I had the privilege to spend the past year working on Sled, from product inception to execution. At the same time we were launching Sled, I experienced some significant management changes at Yahoo! which eventually led to shutting it down. Continue reading
I’ve been obsessed with project management and personal productivity for a almost two decades. My experience ranges from tiny lists to gigantic project plans with hundreds of people and resources. In the past I’ve been a certified PMP and managed large engineering teams. What I’ve learned above all, is that we tend to overcomplicate everything.
Four years ago my startup Nouncer failed for many reasons, none of which had much to do with the product itself. Looking at where Twitter is now and how it evolved, it is a clear validation of my original vision. But even if I had gotten passed the challenges that doomed Nouncer, I still think it would have failed. It was just too complicated, too soon.
I’ve long considered Twitter’s biggest asset to be its 140 character limit. It completely democratized personal expression by making everyone as expressive and articulate. It also helped people communicate more by making their content small enough for casual, constant consumption.
A year ago I started thinking about applying this philosophy – empowerment through restrictions – to project management. I’ve started thinking about enterprise-scale problems and what a restrictive tool might look like. But no longer working on large scale enterprise projects, my attention shifted to personal productivity and “home projects”, and so Sled was born.
There wasn’t much of selection 6 months ago when I started coding Sled when it came to Node frameworks. Node itself provides very little. You create an HTTP server and get a callback when a new HTTP request comes in. Modern web applications require session management, request routing, and view rendering at the very least. These functions are provided by frameworks.
At the time, the only two popular frameworks were Connect and Express. Connect provides a basic middleware framework with some built-in facilities such as a static file server, route management, cookie parser, form-encoded and JSON-encoded body parser, and logging. Express, built on top of Connect, adds a much more robust routing facilities, view rendering and other goodies. I’m not doing these frameworks justice with this descriptions.
I use Express extensively and it is fantastic.
Express’ biggest asset is its maintainer, TJ Holowaychuk. I have seen first-hand how Express grew and matured into a stable and powerful framework. Express made it easy for me to write the application server used for login, registration, account management, and other static assert (developer pages, about) with very little extra effort. If you are going to develop a traditional web application in Node, you should probably start with Express.
I plan to open source some of the additional functionality I’ve added on top of Express to validate request bodies and query parameters, a flexible authentication configuration facility for routes, and a light layer to make it easier to building API servers. Given my focus on OAuth, I’m going to share my OAuth + Express experience in a followup post.
Express really shines when you combine it with Jade, another brilliant brainchild of Mr. Holowaychuk – a simple templating language for HTML which is easy to learn, easy to read, and unlike all the rest, doesn’t suck. We had to restrain ourselves from converting some static HTML files into Jade because once we started using it, we didn’t want to read actual HTML ever again.
Socket.io is Magic
I am willing to bet that a large percent of those taking a look at Node for the first time are doing it because of a Socket.io -powered demo they have seen. Socket.io provides a trivial-to-use server and client library for making real-time, streaming updates between a web server and browser client. It makes building cool real-time games a matter of hours, and it works on every crappy browser known using the best available option from Flash to WebSockets.
We use Socket.io to power Sled’s real-time features which include live updates to your shared list as changes are made. To put this in perspective, it took us one day to add real-time streaming updates to the API server, and another day or two to add it to the client. So in three days we got full, real-time updates going between multiple browsers. What used to be months of work when Google Docs was first introduced, is now trivial with Socket.io.
So yeah, it’s magic.
Socket.io comes from Guillermo Rauch and the team of superstars at LearnBoost. There are days when I have to wonder if these guys get anything done for their own startup, given the amazing open source projects they push out on a weekly basis. I’m bummed that I’m not the target audience for their product because looking at the technology it must be amazing. I consider them part of the Sled team, given just how much of their code we are using.
In fact, I appreciate their work so much, I’d like to get a small fundraiser going to send these guys to dinner or whatever else they feel like doing for fun. I hope you join me in showing our appreciation. Without their continued work and dedication, Node would be an amazing platform that is just too raw to use.
Thanks to Blaine Cook for the idea.
Working with CouchDB was fantastic. It took no time to learn the REST API and jump right into building the application. I had the first version of the base API ready in 2 days, including full database integration. Couch is a document store using JSON as the document format. Combined with an HTTP REST API, it makes Couch an ideal fit for Node. That’s exactly why I picked it.
I dismissed MySQL from the very beginning for two reasons. First, I had no idea what my schema would look like, and knew I was going to change it a lot over time, as well as allow unstructured data. Second, MySQL is a block database in nature (e.g. database deadlocks, atomic actions, etc.) and while you can use it with Node, it really wasn’t designed for this kind of environment. Also, at the time, there was no quality driver for Node. Oh, and I really wanted to play with a NoSQL database.
I started talking directly to the database but within a few hours switched to use Cloudhead‘s excellent cradle module. The module provides a light layer for managing the HTTP client calls, error handling, simple caching, and common macros for performing multiple database requests in a single function call. It is a very light abstraction layer (and it doesn’t abstract too much either). This worked very well for a few months. I didn’t use cradle’s built-in cache because of the expected size of my database and my constant tweaking of data manually.
Node is all about non-blocking, asynchronous architecture. This means any activity taking a long time to finish, such as file access, network communication, and database operations, are requested and put aside until the results are ready and returned via a callback function. Instead of asking to read a file and waiting for the operating system to come back with a file handler or buffer, the a callback function is invoked when the operation is completed, freeing the server to handle additional requests.
What gives Node a bit of a negative reputation is how this architecture affects its style of programming and the difficulty some people are having in getting used to it. When I first started, I described this convoluted style of coding like scratching your right armpit with your left hand by twisting your left arm over the shoulder, behind the neck, and under the back of the right armpit. There are days when it still feels like that, but at least now my arms are much more flexible.
For the past 6 months I’ve been spending most of my time building a new service called Sled. I’ll write more about Sled in the coming weeks, but for now all you need to know is that Sled is a collaborative list making tool for small groups.
I expect this to change very fast.
The next few posts are a random collection of things I’ve learned after 6 intense months of working with Node.