For the past 2 years I have had the pleasure (and luck) of working with node full time. It’s an amazing technology and a remarkable community. Oh, and it’s crazy fun. My focus this year was rethinking web services at Walmart Mobile, from the business layer all the way down to the tools and process we use. A significant part of this effort focused on developing hapi, a new web services framework for node.
But before I write my traditional ‘Introducing’ post, I wanted to first discuss the evolution that led us to build a whole new framework. To truly understand the judge a new framework, it is important to understand the context and objectives leading to its creation.
Walmart, like many big and small companies, is not in the business of creating and maintaining developer tools.
There is great value in doing open source and participating in the community. But without an active community, developing an open source project can quickly become a liability. When we started, the goal was to take an existing effort and simply enhance it to meet our goals. We didn’t set to build an entire stack from scratch. If you build it and nobody comes, it can get expensive and very isolating.
I’ve started using node working on Sled, a short-lived project at Yahoo! which was open-sourced as Postmile. Sled had a simple services architecture in which multiple clients communicated with an API server. At the time, the only real framework available was Express, and for a very early effort, it was quite good.
But it also required a lot of effort supplementing it with the necessary building blocks for a productive foundation. Most of the effort focused on changing the interaction between Express and the business logic, along with support for authentication, encrypted cookies, input validation, and other small features missing at the time.
When Sled ended and I moved on to Walmart, I started by extracting the parts of Postmile I liked as the foundation of our new services platform and called it hapi (for HTTP API server). It was still built on top of Express, and wasn’t much more than a bloated Express middleware – but it was easy to use and much faster to hack services on. It was a good start.
We used Express for a few months until we hit a wall.
Some of the features we needed required a cleaner isolation between the node HTTP server, the router (the part matching requests to handlers), and the actual business logic. Express tries to be an unopinionated framework, but in practice, there is no such thing. Every decision made imposes an opinion and a way of working with the framework, and Express comes with a lot of baggage from its Connect roots (req, res, next).
For example, we were trying to add batch functionality in which a JSON payload includes a list of chained endpoints, some parallel and some serial. Given the cost of round-trips in mobile, this was an important requirement. The problem was, because of Express’ middleware architecture and it’s strong coupling of the router to the node HTTP server, the only clean way of making internal API calls that truly replicate the result of calling each API individually from the outside is to make an actual HTTP call into the server. This gets real messy when using authentication and having to fake client credentials.
We also had some bad experience with Express’ lack of true extensibility model. Express was a pleasure and easy to use 2 years ago with a limited set of middleware and very little interdependencies among them. But with a long list of chained middleware, we found hard to debug problems when we simply changed the order in which middleware modules were being loaded.
Goodbye Express, hello Director, the simple but powerful router from Nodejitsu’s Flatiron project.
Replacing Express with Director took a day and the result was a cleaner interface between the layers. We enjoyed working with Director and the responsive support from its active community. By this time, there was little resemblance between our higher level framework API and Express’ since we wrapper the entire Express API with our own. We were able to quickly implement batch processing and other features and used Director successfully.
But as you can guess, that didn’t last long.
In order to achieve Director’s highly optimized routing tree, it had to make certain choices about how paths are structured and configured. Director offers a particular mix of static routes, parameterized routes, and some forms of wildcards / regular expressions.
In some ways, Director’s attempt to accommodation expectations from Express users ends up make it more complex for little gain. Director does not keep track of the path parameter names (only their order) which meant we had to hack that feature back in. We also had to work around Director’s way of providing extensibility hooks via before/after functions. After all, how framework are extended is one of the most significant architectural choices.
One of the main problems we had with Director was the difficulty in prevent route collisions – a critical feature when building a modular web services layer where different plugins register their endpoints with a central host. More on that in a followup post.
We didn’t set to create a whole new framework, but at the end of the day, it was easier and more appropriate than trying to make an existing solution work. Framework represent a philosophy encompassing both coding style and engineering process, and it’s hard to impose one on another.
hapi was created around the idea that configuration is better than code, that business logic must be isolated from the transport layer, and that native node constructs like buffers and stream should be supported as first class objects. But most importantly, it was created to provide a modern, comprehensive environment in which as much of the effort is spent delivering business value.
A formal hapi introduction is coming. If you can’t wait, check it out.