Converting Full Time Pay to Hourly Contract Rate

Multiple people asked me this week how much they should charge for an hourly development engagement, trying to figure out what’s fair. They all know their full time salary range but didn’t know how to translate that into self-employed hourly rate. This post is a quick explanation of the formulas I use. This is obviously not an industry standard or what your next employer (or employee) is going to use but you might find it a useful reference point. This is limited to US based employment. Needless to say, these formulas ignore the many employer tax differences of employing W-2 vs 1099 employees. They are also ignoring the most important factor which is the human element. We are talking about people hiring other people where unique skill sets, personalities, and economic realities can often mean more.

There are two types of hourly employees, those you can easily replace with full time people and those you can’t afford (or can’t convince) to work for you as W-2 employees. For the first group, the hourly rate is based on the employer cost of a full time person. Basically, the employer will want to keep its cost about the same regardless of the employee status.

Assuming:

  • 20% employer overhead cost over the employee cash compensation for benefits, taxes, and other expenses
  • 250 work days a year
  • 15 days paid vacation
  • 8 hours work day

The formula is:
Hourly rate = Total annual cash compensation / 1567

For the second group – people the employer can’t afford to pay full time or the kind of top talent they can’t convince to join full time – the formula is slightly different. There is an additional consideration. People doing short term contract work typically lose about 20% annually due to time in between jobs and the cost of finding work on a regular basis (a contractor factor of 0.80 – 20%productivity loss). Because they are top talent, the employer will have to pay for that loss. This gives us:

Hourly rate = Total annual cash compensation / 1253

The full formula is:

\displaystyle \frac{A * E}{(W - V) * H * C}

  • A – total annual cash compensation
  • E – employer overhead percent (1.0 is 100%)
  • W – work days a year
  • V – paid vacation days
  • H – hours a day
  • C – contractor factor (percent time employed)

This means:

  • An average developing making $120K a year would be able to get about $80/hour
  • A senior expert making $200K a year would be able to get about $160/hour

If the full time job come with equity (assuming a 25% annual vesting schedule of a publicly traded stock), you can add 25% of the equity value to the annual salary.

On Securing Web Session Ids

Someone asked: why does the Express session middleware add a hash suffix to the session id cookie? A great question. But first the obligatory disclaimer: like any security advice from someone who doesn’t know the specifics of your own system, this is for educational purposes only. Security is a complex and very specific area and if you are concerned about the security of your system you should hire an expert that can review your system along with a threat analysis and provide the appropriate advice.

Brute Force

Brute force attacks are those in which the attacker is trying to gain access to the system by making repeated requests using different credentials (until one works). The most common example is of an attacker who tries guessing a user password. This is why passwords should be long and avoid using dictionary words to make it harder to guess. Properly designed systems keep track of failed authentication requests and escalate the issue when it appears an attack is in progress. Passwords are not the only credential used in web authentication. The most common implementation includes a login page which upon successful authentication sets a session cookie on the client. The session cookie acts as a bearer token – whoever shows up with the token is considered to be the authenticated user. Setting a session cookie removes the need to enter your username and password on every page. However, this session cookie now acts as the sole authentication key and anyone who gains access to this key will gain access to the system. Cookies are, after all, just a simple string of characters. A session id guessing attack is a type of brute force attack. Instead of trying to guess the password, the attacker is trying to guess the session id and forge the authentication cookie. The attacker generates session ids and tries to make requests using those ids, in hope that they will match actual active sessions. For example, if a web application session ids are generated in sequence, an attacker can look up their own session id and based on that forge requests using nearby session id values. To protect against this attack we need to make guessing session ids impractical. Note I’m saying “impractical,” not “impossible”.

Impracticality

The first step is to make sure session ids are sufficiently long and non-sequential. Just like passwords, the longer the session id is, the harder it is to find a valid one by guessing. It is also critical that session ids are not generated using a predictable algorithm such as a counter because if such logic exists, the attacker is no longer guessing but generating session ids. Using a cryptographically secure random number generator to produce sufficiently long session ids is the best common practice. What’s “sufficiently long”? Well, that depends on the nature of your system. The size has to translate into an impractical effort to guess a valid session id. Another way to prevent an attacker from guessing session ids is to build integrity into the token by adding a hash or signature to the session cookie. The way the Express session middleware does this is by calculating a hash over the combination of the session id and a secret. Since calculating the hash requires possession of the secret, an attacker will not be able to generate valid session ids without guessing the secret (or just trying to guess the hash). Just like strong random session ids, the hash size must match the security requirements of the specific application it is meant to protect. This is because at the end, the session cookie is still just a string and open to guessing attacks. Session ids must be sufficiently long and impractical to guess. There are a few ways to accomplish this. The randomness and hashing techniques above are the two most common ways but not the only ones.

Layers

If we generate strong random session ids, do we still need the hash? Absolutely! The core security principal is layering. This is also known as not putting all your eggs in one basket. If you rely on a single source of security, you end up with no security at all if that single source fails. For example, what if someone finds a bug in your random number generator? What if they find a way to hack that part of your system and replace it? There are countless of known attacks exploiting exactly this – the generation of random numbers that turns out not to be so random after all. Combining a strong random session id with hash for integrity will protect against flaws in the random number generator. It will also protect against developer errors such as using the wrong random number generator function (e.g. the not so random method every system offers alongside the strong method). We all write bad code no matter how great our process is or how experienced we are. It is part of software engineering. This is why it is so important to layer your security. A moat is not enough, you also want a wall behind it, and probably some guards behind the wall. If you think using the wrong random function or a deep bug in OpenSSL are the only two issues here consider the common practice of monkey patching code in JavaScript and other dynamic languages. If someone anywhere in an entire application deployment messes with the global random facilities (for testing, logging, etc.) and breaks it (or it is part of a malicious code injection), session ids relying solely on randomness are no longer secure.

Alarms

An important difference between guessing passwords and guessing session ids is the fact that passwords are associated with an account (e.g. username). The account-password pair makes it easier to keep track of brute force attacks because it provides a relatively straightforward way to keep track of failed attempts. However, when it comes to session ids, it is not as simple because sessions expire and do not include an account context. This means an invalid session id could come from an expired session or from an attacker, but without additional data (e.g. IP address) it would be hard to tell the difference in a large scale system. By including an integrity component in the session id (via a hash or signature), the server can immediately tell the difference between an expired session, an unallocated session id, and an invalid session. Even if you just log invalid authentication attempts (and you should), you would want to log an expired session differently than an invalid one. Beside the security value of knowing the difference, it will also provide useful insight about how your users behave.

Hygiene

Credentials should expire and therefore session ids should have a finite lifespan (where duration is very much a system-specific value). While cookies come with an expiration policy, there is no way to ensure it is actually obeyed. An attacker can set the cookie expiration to any value without the server being able to detect it. A common best practice is to include a timestamp in every credential issued, which can be as simple as adding a timestamp suffix to the randomly generate session id. However, in order to rely on this timestamp, we must be able to verify it was not tampered with and the way to accomplish that is with a hash or signature. Adding a timestamp to the session id allows the server to quickly handle expired sessions without having to make an expensive database lookup. While this might sound unrelated to security, it actually is very much core to maintaining a secure application. A denial of service attack (or DoS) is an attack in which the attacker makes repeated requests with the sole purpose of consuming too much resources on the server and either shutting it down or making it inaccessible to others. If every request authentication requires a full database lookup at the application tier, an attacker can use forged session ids to stage a DoS attack with ease. By including an integrity component in the cookie, the server can immediately identify forged or expired credentials without any backend lookup cost.

Kill Switch

Sometimes things go wrong. And when they go very wrong, you need to have a way to immediately invalidate entire classes of sessions. Because generating a hash or signature requires a server-side secret or key, replacing the secret will immediately cause all session ids to fail validation. By using different secrets for different types of session ids, entire classes of sessions can be segregated and managed. Without such a mechanism, the application itself has to make a computed decision about the state of each session or perform mass database updates. In addition, in large distributed systems with database replication over different geographic locations, invalidating a session record in one location can take seconds and even minutes to replicate. This means the session stays active until the full system is back in sync. Compared to a self-describing and self-validating session id, the benefits are obvious.

General Purpose

An important feature of the Express session middleware is its support for user-generated session ids. This allows developer to deploy the middleware in an existing environment where session ids are generated by an existing entity which might reside on a completely different platform. Without adding a hash to the user-provided session ids, the burden of building a secure system moves from the expert (the module author) to the user (which is likely to be a security novice). Applying a hash is a much better approach than forcing an internal session id generator.

Crocodiles

Adding a hash to a strong random session id is not all you should do. Whether your moat can benefit from crocodiles is, again, a castle-specific decision. Without going too far from the topic, there are plenty of other layers you can add to your session management tier. For example, you can use two session credentials, one long lived (lasts as long as the session) and another short lived (good for minutes or hours). To refresh the short lived you use the long lived but by doing so, you are reducing the exposure of the long lived credential on the network (especially when not using TLS). Another common practice is to have one cookie with general information about the user (e.g. first name, recently viewed items, etc.) alongside the session and to then include something from that cookie in the hash to create a binding between the user’s active state to the authentication. It’s a way of bringing back the “username” into the workflow. To go even further, hashing can be replaced with signatures and cookie content can be encrypted (and then hashed or signed on top). The security verbosity must match the threat.

Last Words

If you take one thing away from this post I hope it is the concept of layering security. While math plays a significant role in security, it is far from the only tool. Measuring the odds of guessing a session id as the sole source of security is failing to recognize that security comes from a combination of defenses. I would also strongly advise against having the kind of academic debates focusing on a single aspect of a secure system in public (at least without the proper disclaimers). It is extremely misleading to narrow down the question to the point where it causing confusion and misinformation. Asking “is there a statistical benefit to hashing a strong random session id?” is harmful because it creates the false impression that this is the only consideration. It moves the discussion from the real world to that of an incomplete abstraction. As I hope I demonstrated above, there are a lot of reasons for including a hash beyond just making guessing impractical.

Introducing chairo, a hapi.js Microservices Plugin

Introduction

Over the past four years hapi grew to be the framework of choice for many projects, big or small. What makes hapi unique is its ability to scale to large deployments and large teams. As a project grows, so does its complexity – engineering complexity and process complexity. hapi’s architecture and philosophy handles the increased complexity without the need to constantly refactor the code or build meta-frameworks on top of it, while keeping the simple cases simple.

hapi being a web application framework is not concerned with how the actual business logic is implemented. It provides the developer with a few hooks (in the form of handlers and extensions) to implement its logic and largely stays out of what goes into these hooks. As the project complexity grows, so does the need to decouple functionality and distribute internal load. The hapi server becomes the outwards facing interface (either via an API or UI) while behind it an array of other technologies is used to break the monolithic business logic into smaller pieces (some of which can themselves use hapi).

As node extends deeper into the full system stack and is used to implement more and more core services all the way down to the database or file system, we need better tools to connect all these components together. While we can certainly use many standalone hapi servers for a distributed RESTful SOA, this might add complexity and overhead that is better addressed with other tools.

Microservices

The basic premise of microservices is to isolate business logic to its smallest components, each implemented separately and with a clear and simple interface. Complex solutions are then broken down into a set of small services which are composed together to provide the combined, orchestrated functionality.

The important part about microservices isn’t the deployment strategy which should be based on load and scale requirements (as well as policies and politics). The focus is on writing the code in a way to allow these services to be deployed as both a monolithic single executable and as many distributed processes based on the evolving needs of the environment in which they run. Such decisions represent a trade-off between software complexity and operational complexity.

A good microservices framework provides the tools to define these components and connect them together through a message bus which supports this range of deployment strategies. As the project grows, services can be moved, changed, or replaced with minimal impact of the rest of the architecture because they can live side-by-side with older versions.

Seneca

Seneca is a microservices framework from nearForm, a leading node consultancy based in Ireland. The nearForm team has been an early adopter for node and is an active member of the community (they organize the European NodeConf franchise among other activities).

The core feature of Seneca is the registration and invocation of actions through simple and powerful pattern matching. Each of these actions (which can be as simple as a single function) represents a microservice which in turn can invoke other actions. To reach another service, you just need to know it’s matching pattern regardless of where it is deployed.

var Seneca = require('seneca');

// Create instance
var seneca = Seneca();

// A microservice for loading a user record from a database
seneca.add({ record: 'user' }, function (message, callback) {
    db.load('user', message.id, callback);
});

// A microservice for information about today
seneca.add({ service: 'today' }, function (message, callback) {

    return callback(null, { date: (new Date()).toString(), weather: 'Sunny' });
});

// Invoking the two services
seneca.act({ record: 'user', id: '123' }, function (err, user) {
    seneca.act({ service: 'today' }, function (err, today) {
        console.log('Hi ' + user.name + '! It is a ' + today.weather + ' day today');
    });
});

And to make things easier, Seneca accepts string patterns as well using a loose JSON format:

seneca.act('record:user,id:123', function (err, user) {
    seneca.act('service:today', function (err, today) {
        console.log('Hi ' + user.name + '! It is a ' + today.weather + ' day today');
    });
});

The combination of the two services can be published as another service:

seneca.add('service:welcome', function (message, callback) {
    seneca.act({ record: 'user', id: message.id }, function (err, user) {
        seneca.act('service:today', function (err, today) {
            return callback(null, {
                message: 'Hi ' + user.name + '! It is a ' + today.weather + ' day today'
            });
        });
    });
});

seneca.act('service:welcome,id:123', function (err, result) {
    console.log(result.message);
})

The chairo plugin

Seneca is ideal for building microservices implementing the bits and pieces of the application business logic. However, its pattern matching routing interface is optimized for internal consumption and less for public exposure of these services. It would be unusual to expose Seneca actions direction as a public API. In addition, Seneca focuses on the backend architecture, not on interfacing with a front end experience (single page application or server-rendered views).

The new chairo (which means “happy” in ancient Greek) plugin brings the power of Seneca to hapi by bridging between these two frameworks and allowing developers to use the richness of serving web and API content via hapi while building their business logic using the Seneca microservices architecture.

chairo is registered with a hapi server like any other plugin using the hapi server.register() method. Once registered it decorates the server and request objects with a reference to the seneca instance initialized:

var Chairo = require('chairo');
var Hapi = require('hapi');

var server = new Hapi.Server();
server.connection();

// Pass options to the Seneca constructor
var senecaOptions = { log: 'silent' };

// Register plugin
server.register({ register: Chairo, options: senecaOptions }, function (err) {

    // Add a Seneca action
    var id = 0;
    server.seneca.add({ generate: 'id' }, function (message, next) {
        return next(null, { id: ++id });
    });

    // Invoke a Seneca action
    server.seneca.act({ generate: 'id' }, function (err, result) {
        // result: { id: 1 }
    });

    server.route({
	method: 'POST',
	path: '/id',
	handler: function (request, reply) {
            // Invoke a Seneca action using the request decoration
            request.seneca.act({ generate: 'id' }, function (err, result) {
            if (err) {
                return reply(err);
            }

            return reply(result);
        });
    }
});

hapi already provides its own version of actions using server methods. While server methods can be cached and used as handlers and prerequisites, they cannot be decoupled from the server implementation and must reside within the same process. The new server.action() method provided by chairo maps a Seneca action pattern to a hapi server method. This allows using Seneca actions anywhere server methods can be used with the Seneca flexibility of maintaining the actual business logic elsewhere.

var Chairo = require('chairo');
var Hapi = require('hapi');

var server = new Hapi.Server();
server.connection();
server.register(Chairo, function (err) {
    // Set up a Seneca action
    var id = 0;
    server.seneca.add({ generate: 'id' }, function (message, next) {
        return next(null, { id: ++id });
    });

    // Map action to a hapi server method
    server.action('generate', 'generate:id', { cache: { expiresIn: 1000 } });

    server.start(function () {
        // Invoke server method
        server.methods.generate(function (err, result1) {
            // Invoke the same server method
            server.methods.generate(function (err, result2) {
                // result1 === result2 (cached)
            });
        });
    });
});

In simple cases, all you want to do is map a Seneca action to a hapi endpoint and proxy the action result back. chairo adds a new reply() interface decorator reply.act() which sends back a handler response using the result of a Seneca action by specifying the action pattern.

server.route({
    method: 'POST',
    path: '/id',
    handler: function (request, reply) {
        // Reply using a Seneca action
        return reply.act({ generate: 'id' });
    }
});

In addition, the act handler shortcut is also provided:

server.route({
    method: 'POST',
    path: '/id',
    handler: { act: 'generate:id' }
});

For more complex cases where a hapi endpoint requires combining data from multiple source, some of which are based on Seneca actions, chairo provides the reply.compose() decorator which renders a template view using the provided template and context object. The context object combines regular object keys with top level keys with a $ suffix which are resolved into the corresponding Seneca actions matching the key’s value pattern.

// Set up a hapi view engine
server.views({
    engines: { html: require('handlebars') },
    path: '../templates'
});

// Add route
server.route({
    method: 'GET',
    path: '/welcome',
    handler: function (request, reply) {
        // Setup context with both Seneca actions and simple keys
        var context = {
            today$: 'service:today',
            user$: { record: 'user', id: 123 },
            general: {
                message: 'Welcome'
            }
        };

        // Reply with rendered view
        return reply.compose('example', context);
    }
});

Using the template ./templates/example.html:

<div>
    <h1>{{general.message}} {{user$.name}}!</h1>
    <h2>Today is {{today$.date}} and it's going to be a {{today$.weather}} day.</h2>
</div>

In addition, the compose handler shortcut is also provided:

server.route({
    method: 'POST',
    path: '/id',
    handler: {
        compose: {
            template: 'example',
            context: {
                today$: 'service:today',
                user$: { record: 'user', id: 123 },
                general: {
                    message: 'Welcome'
                }
            }
        }
    }
});

What’s Next?

The initial version of chairo is a very basic implementation of the Seneca features within the context of the hapi ecosystem. It maps the basic actions functionality and allows simple and elegant composition of API endpoints and web pages in hapi powered by existing or new Seneca deployments. When used with more advanced Seneca configuration, the actions can be moved to other processes, benefiting from the full power of a distributed microservices architecture.

Future versions of this plugin will look to incorporate more Seneca functionality such as data entities, make routing configuration simpler for a large distributed system, and combine the logging functionality of the two frameworks into a unified operations view.

Please give it a try and post questions, feedback, or issues.

The Best Kept Secret in the Node Community

It’s not that anyone is trying to hide this from you. It’s that those who have gone through the experience and know how incredible it is just assume it to be so obvious that it is not worth mentioning. If you have not been to a NodeConf event at Walker Creek Ranch you are passing on a rare opportunity to truly elevate your node game and connections. This is not a hyperbole.

My first NodeConf (at the very first NodeConf in Portland) was a typical ineffective event. I didn’t know anyone. No one really cared who I was and what I was working on. Yes, I could always name drop OAuth and other bullshit I worked on in the past but what I actually cared about – a node-based project called Sled – was of little interest to anyone. I also wasn’t part of the small group of people who ran the node project. I didn’t know any of them.

Conferences can be hard. They are not a natural place to meet people at a sufficiently deep level to build meaningful lasting connections. That’s not true for all conferences but I am sure your first few events felt pretty lonely (unless of course you went with people you knew in which case you stuck with them and missed the point as well). NodeConf at Walker Creek Ranch is different. Completely fucking different.

First, it is attended by pretty much all the internet famous node celebrities (and me). Second, we are all Mikeal’s hostages there. There is nowhere to go. There is nothing to do (other than hang out with people). There is barely any wifi so working on your laptop is kinda useless. This might sound terrible but here is the thing – everyone is equally stuck.

Hey look over there, it’s Isaac – yeah the guy who ran node for a while and created npm. Want to chat with him? Go ahead – get him! How fast can he possibly run away from you in those ridiculous shoes? Nope, you are not wrong, that’s Substack over there hacking away on some new crazy fucking tiny module shit. Ummmmmm – you guessed it, dshaw chilling out under the big oak tree (the same fucking tree the fucking domains API was conceived under – maybe this year will bury it there). Did a drone killing robot just run past you? I guess Raquel is up to no good again. And where’s that awful sound coming from?! Must be Nexxy throwing another rave party at the boogie barn.

Being successful in node – like most other emerging platforms – require a solid network. With io.js moving faster than most people can keep track of and the module landscape changing daily, it is absolutely essential to be both connected to others in the community and to have access to the people who can help you out. Getting a question answered about node or an npm module is dramatically easier when you had a drink with the author and can ping them directly on IRC or email. Making a personal connection matters a lot.

I can tell you without a doubt that my personal success with node and my community connection were both directly result of attending the very first NodeConf SummerCamp at Walker Creek Ranch. This is where I met all the people I have later worked closely with to make node a huge success at Walmart. That event was instrumental to my personal success and the success of my employer. Need proof I’m super successful? Well, I’m the only person who’s picture has ever been posted on the official nodejs.org website – after all, isn’t that how we measure success!

NodeConf this year is likely to be a smaller event. This means more quality time with people, more meaningful connections. If you are able to travel to CA for the event, you have to be an idiot not to. It’s really that simple. Here is a chance for you to spend time with the people who wrote a lot of the stuff you’re using, and the people who are likely to write it next. This is the place where so many of the second and third round of node leaders came from and you could (and should) be part of.

If this sounds like a bit of hero worship, it’s not. I am dropping names because these are all amazingly generous people who not only helped push node forward, but are also known for their kindness and welcoming attitude. They are all very busy and in other places can be hard to get hold of for a meaningful conversation. But at Walker Creek Ranch, we’re all just hanging around chillin’. The setting is so beautiful and relaxing that no one is treated differently. There are no private rooms, secret parties, or dinners where all the cool kids are hangin’ (except you).

This should be the easiest conference expense to justify to your boss. If you are doing node and have never been to a NodeConf event at the ranch, you are throwing away an opportunity to improve your skills, your network, your influence, and make a difference at the company you work for. It is a no brainer.

There are still tickets available and if you use my discount code you will get $50 off any ticket type. I expect to see you there next month!

On Leaving Walmart

It has been an exciting three and a half years, but it’s time to move on.

Looking back, there is much to be proud of. We have produced massive amounts of open source code that have been successfully adopted by dozens of companies. We created an open flow of information on our production experience (e.g. #nodebf) that played major role in increasing node adoption by the enterprise. It’s amazing how our small team of 18 people had an amplified influence over the future of node. I am extremely grateful for the trust I was given building and leading this exceptional team. This was very much a team effort all around.

Our biggest and most visible accomplishment has been the creation of the hapi framework and its community. It is very hard to predict how an open source project will work out, especially one created by a corporation (and even more so when that corporation is Walmart). hapi’s success clearly demonstrates that by embracing the community and openness from the start, companies can reap valuable rewards.

We never said “we want to work with the community” because we considered ourselves part of the community. Over the last year, the vast majority of work on hapi modules has been done by people outside of Walmart. In fact, the shift has been so dramatic that we changed the entire governance model last year to encourage and empower this transformation. At this point, Walmart is responsible for a very small share of the resources maintaining the codebase and the community.

This transformation has been so successful that it is no longer a Walmart project – and that’s a huge win for Walmart. Walmart gets to continue to benefit from their initial investment by having access to a best-in-class framework, custom-made to suit their needs, with little to none ongoing cost. They planted and nurtured a seedling and now get to share in the benefits of the tree, cost free.

There were two less visible accomplishments (but equally impressive) worth mentioning. The first is the amazing remote team we built which can serve as a model for other companies to follow. When I joined Walmart, attracting talent was a major challenge. People interviewed with us for the sole purpose of getting a competing offer to leverage against the company they actually wanted to join. But by reaching beyond the local Bay Area boundaries and showcasing our work and community participation, we quickly became one of the most sought after teams to join.

The second is the culture of quality we created. Learning from open source and community management best practices, along with extensive investment in testing tools, we developed an engineering workflow that has produced unmatched quality results. What is impressive about it is how quickly it was adopted by others outside of Walmart and the hapi community.

It has been gratifying to see our accomplishments celebrated by the community and to be able to share our success with others through open source and public sharing of information. Walmart has been consistently supportive of these dramatic cultural shifts in attitude towards the outside world more than any company I have previously worked for. I hope others will use this example to push for change in their own organizations.

As for hapi moving forward, nothing changes. I will continue to maintain hapi and participate in leading the community around it. As I mentioned above, hapi has been successfully transitioned out of Walmart over the last year and is fully owned by the community that supports it. No one owns any trademarks or has special rights to the code, names, logos, etc. It’s all under the same open license.

There is one person I have to thank by name and that’s Dion Almaer – you won’t find a more supportive, generous, and inspiring person to work for. It has been an amazing experience and I am grateful to everyone who took part – we share these accomplishments.

As for what’s next, I guess it’s time to find another adventure (yep, I’m looking).

The Node Version Dilemma

If you are using node for years or just starting, you are probably trying to figure out which version and distribution to use moving forward. The official Joyent distribution has two version 0.10 and 0.12, and the new community effort io.js has an active, almost weekly release schedule.

0.10 is the Current Safe Bet

If your production is running 0.10, keep it for now. If you are using node in production with a version older than 0.10, you should upgrade to the latest 0.10. Version 0.10 is by far the most stable, reliable, and well-understood release available today. We have been running 0.10 under heavy production load with extreme spikes for over 2 years. We started developing our production stack on 0.10.0 but it took until 0.10.21 for it to be production ready.

If you are not yet running node in production but are planning to go live in the next three months, use 0.10 for now. Upgrading to other versions later is going to be pretty simple (especially if you use a well maintained framework and published modules). It is unlikely that 0.12 or io.js are going to be production ready within that time-frame, and moving from 0.12 or io.js back to 0.10 is going to be painful (if at all possible).

If you are starting now, or if you are not planning on a significant load anytime soon (e.g. next three months), go ahead and start with the latest io.js. However, make sure to fully appreciate the risks associated. I consider building with io.js today similar to betting on node in the 0.2-0.4 days. It was clearly the future, but it was also experimental and unstable.

0.12 is DOA

The long awaited 0.12 release came about a year after it was originally expected. It represents significant improvements and is clearly the foundation of a future stable node release. However, given our experience with both 0.8 and 0.10, 0.12 will take about six months of active usage and development to reach similar levels of stability of the current 0.10 release.

I have already started seeing issues reported with 0.12 (and io.js which shares much of the code). Some of these issues are complex and involve changes in timing and event emitters that only appear in specific edge cases (these are no necessarily bugs, just very fine breaking changes). However, it is these edge cases that you should be most worried about when going to production since statistically, the larger your scale the higher the risk of hitting them.

I am skeptical about 0.12 reaching the required levels of stability given the resources available to work on it today (with the majority of core contributors focused on io.js). If 0.12 has a chance of moving forward, it is only once the foundation work is done and ready to support it, or via a merge with the io.js distribution. Either way, it is not a move worth making at this time.

io.js and the Future

There is little doubt that io.js represents the future of node. If you look at the work, people, and culture around it, it is pretty obviously heading in the right direction. I expect there to be only one prominent version of node within 6 to 9 months, either by making the official distribution obsolete or by merging with it. The question is when to make the switch.

The problem is that for most people, keeping track of what is going on with io.js right now is practically impossible. There is just so much activity going on. I would not call it noise because it is clearly well thought, well executed, and well communicated. Given that io.js was born out of the lack of progress on the official distribution, I am certainly not advocating slowing down or artificially blocking progress. What I am looking for is the organic maturity that is evident by the project naturally slowing itself down. And that can take a while.

Over the next few weeks and months, companies will come out and share their io.js deployment stories. What is important to remember is that just the fact someone deployed io.js in production doesn’t really means it is ready. The more meaningful story is when these companies share their experiences 3 months later, 6 months later, and a year later. When Walmart deployed 0.10 to production, it was extremely unstable, but the risk was very low given the overall architecture and mitigation available. It would have been a mistake to use that initial deployment announcement as an indication that your environment is equally risk tolerant and ready.

If you are in a position to take a measured risk and put io.js through the load of a production environment, you certainly should. We all depend on those early adopters. But when sharing that information with the community, make sure to provide the full picture, the technical details, and the reasons why you felt it was a low risk decision.

Why I Do Not Support a Node Foundation

I’ve been aware of the node foundation plans for a while. I have been part of the initial discussion group with Joyent back in May, was part of the node technical advisory board (for a bit), and had extensive discussions about this with pretty much every major players in the community. I have opted to keep my opinions offline until now because I didn’t want my (strongly held) positions to become the “opposition” and add more friction to what was already a pretty messy process. But now that the decisions have been made by both the io.js folks (to fork) and Joyent (to form a foundation), I am free to rant publicly.

For the sake of full disclosure, I am generally opposed to any foundation.

This comes from extensive first-hand experience with participating and forming similar foundations. I was an active participant of the early OpenID Foundation, I represented Yahoo in the formation of the OpenSocial foundation and wrote the intellectual property and working group process documents, I was a founding member and first president of the Open Web Foundation, and I had extensive engagement with the W3C organization. These experiences taught me that foundations are an unnecessary evil.

My main problem with foundations is that as soon as money is involved, the organization takes on a life of its own, and the mechanism will do anything to sustain itself. The first words out of IBM’s Todd Moore’s mouth about the foundation was that the next step is going to be to hire an executive director. I don’t know if Mr. Moore was expressing the position of the foundation or his own agenda but this is exactly the kind of misguided attitude that dooms such efforts.

Consider this – once you hire people to work for the foundation, these people’s livelihood depends on the foundation’s financial stability. This means they spend a significant amount of time raising funds and ensuring their paying members are happy and getting value out of it. No matter how much you try to balance the needs of the community the foundation was allegedly created to serve, it inevitably becomes a voice for its moneybags.

This is not to say all foundations are evil or unnecessary.

There are many examples of foundations that add value and support their communities effectively. The important distinction is what triggers the creation of the foundation and who are the main players behind it. In the node foundation case, the triggers were lack of technical progress on node and some concerns about ownership of the node trademarks. Both of these issues could have been quickly resolved without a foundation.

The great thing about the io.js effort is that it grew out of strong frustration with specific shortcomings in the Joyent process. Namely, the governance model, the lack of code of conduct, the sharp drop in contributions, and a release process that was predictable in its unpredictability. These concerns triggered the initial discussions about a foundation but when it came time to actually address them, the io.js community realized that all they needed to do was to simply focus on fixing things. They didn’t set a foundation, raised money, or registered marks. They simply created the work space needed to get shit done.

On the trademark side, Joyent has long claimed that their work to protect the node marks provided an important service to the community and to node. I disagree. I think trademarks should only be used to protect business interests, not to put someone in a benevolent position to decide what’s in the best interest of a community, especially one as diverse as node. I don’t think a node trademark adds any value. What exactly do we need protection from?

(As an aside, I’d like to point out that my disagreement with Joyent on the trademark policy and foundation plans does not take away from my gratitude and appreciation for everything they have done for node and the huge impact their support had on its success so far).

Node is a subset of the JavaScript community which is flourishing without any active trademark protection. Can you imagine what would have happened to the language and innovation (especially the recent work) if the Oracle corporation who owns the mark for JavaScript dictated to people what is a certified version of the language? I think there is value in someone registering important marks (and then not defending them) only so no one else can do it for evil purposes. Joyent owning the marks and not protecting them would be the ideal. While Oracle owns the JavaScript mark, I cannot find any record of that mark being used or enforced.

Now, I can understand why IBM, a company who never turned down an opportunity to exploit and make a buck wants a foundation. It’s how they manage their relationships with communities to promote their business agenda. I am not calling them evil – just that as a large corporation with a lot of history, I am confident their best interest is not aligned with mine. I would like to note that I am no hater of corporations – I’ve happily worked for Citi, Yahoo, and Walmart to name a few.

What I cannot understand is why the community should want a foundation. What will a foundation provide that we are not already doing a fantastic job at? I am hearing foundation supporters talk about events, sponsorship, marketing, and training. Sounds like a lot of people excited about potential funds flowing their way.

Node and JavaScript events are doing amazing with nothing but grassroots efforts all around the world. Companies are eager to sponsor node development by hiring full and part time developers to work on node and io.js. Node is one of the fastest growing technologies without anyone hiring an ad agency or paying for marketing. And between the free node schools effort and the paid offering of many node companies, along with a growing selection of books, training is well taken care of.

When my employer was approached to be a founding member of the foundation, I recommended they pass on the grounds that it adds no value to them. Walmart already employs two node core developers, along with almost 100 developers who use node on a daily basis and contribute significantly to open source. Walmart has also been a top sponsor of NodeConf for the last few years. Since they are not going to double their support, should Walmart direct all these funds to the foundation instead? How would that increase their influence and improve an already fantastic community?

(I do not speak for, or necessarily represent the position of the Walmart corporation).

The only real argument made so far in support of a foundation is the issue of controlling the trademarks. It could have been easily resolved by Joyent releasing them to the public domain and allowing the community and the market to sort things out. I can tell you for a fact that my employer would not have had any problems dealing with the “ensuing confusion and chaos”. Every other platform has multiple flavors competing, some open source and some commercial. What makes node so special it needs trademark protection?

Since the node foundation is a foregone conclusion, we’ll just wait and see what value it adds. Meanwhile, we should stay alert to make sure the sponsoring corporations are not fucking node up.

Got comments? I’m @eranhammer.

Notes on Managing Remote Teams

The node.js services team we built at Walmart received a lot of attention for our open source contributions and for pushing node forward in the enterprise. What gets little attention is the success we had in building and managing a distributed team.

The following notes are based on over six years of firsthand experience with remote teams during which I’ve spent 3 years as a remote employee and over 3 years building and managing a remote team. Since these notes are based on my personal experience alone, they do not assert any industry-wide conclusions on the effectiveness of remote work and remote teams. However, I think that these notes will help guide you in deciding whether remote work is suitable for your needs and culture, and how to be successful at it.

The Walmart Mobile Services team included between 2 to 20 people over three and a half years, both remote and office-based. This is what I learned.

Misguided perceptions

Many people, regardless of their actual experience with remote teams, have pretty strong opinions about their suitability and success. Many companies I talked raised largely unfounded concerns about remote workers that are based on anecdotal negative experiences. For example, I was told of a bad experience with one designer who failed to produce results when working from home and of an engineering manager who tried to manage a local team remotely.

The first thing to remember is that in any engineering team, about 10% will be underperformers regardless of their work location. Unless your experience is based on sufficient number of people and over a few years, it would be a mistake to reach any kind of conclusions. The second thing to remember is that hybrid teams – teams with both remote and local employees – rarely work out and require constant care. If your negative perception of remote teams falls into one of these two cases, you should reconsider.

Hybrid teams

Most remote employees are part of a hybrid team where some members work remotely and some are part of an office. This doesn’t work. In most cases, the ratio is heavily skewed towards the office group. The problem with managing hybrid teams is the inherit difficulty in enforcing remote culture within a common physical space. It is challenging to forcing people to use online tools to communicate with peers, even if they are sitting right next to them.

What usually happens is that when something goes wrong, the manager will walk over to someone in the office and discuss the problem. That discussion will grow to include more local people but will completely exclude the remote folks. Not only will this alienate the remote members, it will eliminate their ability to contribute to the solving the problem, add value, and participate until eventually they will be considered poorly performing employees.

The same problem applies to remote managers. Senior managers will often bypass the team remote manager and walk directly over to a local team member. When I first joined Walmart and managed a team where I was the only remote team member, I always found out about outages and problems a day or two later. I wasn’t given a chance to deal with issues because when something broke, everyone huddled locally and my lack of presence created the impression of being absent instead of simply working remotely.

Over time, I have found hybrid teams to be too difficult to manage. No matter how much we pushed to get everyone communicating online, regardless of their location, the office people always defaulted to getting up and walking over to their local peers instead of using the online communication tools. Upper management was never able to control their habit of walking over to the first available developer to look into issues for them. The only solution was to force everyone to be remote at least part of the time so they will change their habits and develop some empathy for their remote peers.

Productivity and cost

Our experience hiring about 20 node developers over the last 2 years showed that by building a remote team, we were able to hire better talent at lower cost compared to hiring the same team locally in the Bay Area:

  • Constant access to talent – we receive about 5 unsolicited employment requests from qualified developers a month (that’s a lot for a team of 20, and at Walmart). The candidates we interviewed already wanted to join the team, and with the exception of two people, we didn’t need to revisit an offer or lose someone to a competing offer. Our candidates mostly arrived from areas with either weak technology presence or limited options (jobs, industries, or technologies). Everyone was really eager to join the team.
  • Lower hiring costs – combining strong community outreach with remote positions produced a constant stream of candidates and removed the need to pay recruiters to source resumes. The quality of the people we were able to hire has been above industry average at annual cost of about 20% less on payroll costs compared with local wages. I don’t have office space cost figures but that adds to significant savings during growth periods.
  • 100% retention – over three years, not a single team member left. While this is certainly largely due to a great work environment and competitive pay, it is also due to a certain lock-in for remote employees, coming from markets with low availability of jobs or getting spoiled by the many benefits of working remotely and the limited availability of remote positions.
  • Extended coverage – our team is spread out over 4 time zones which means a normal 8 hours day is stretched to 12 hours. Add to that the flexible schedule options and the different work habits of people (morning people vs night people) and we have about 18 hours a day of team availability without asking people to work late or take night shifts.

Maintaining team cohesion

Remote teams lack the social glue that an office provides. This can make work very difficult during stressful times and especially for new hires. For a long time we had very little team cohesion. Most of the interactions were between individuals and me (as manager). This became a concern when we started growing the team. Here is what worked well for us:

  • Encourage everyone on the team to join a team nonsense channel. You can call it “random” or “general” or “nonsense” but the goal is to have a place where people can post jokes, silly pictures, abuse a bunch of chatroom bots, etc. It is critical that people interact with each other in casual, non-work related manner as much as they have serious conversations about work. An office provides that via water cooler chats and lunch breaks so for remote teams, we need to find other venues.
  • Organize a few face-to-face meetings for small groups. This should happen naturally based on business needs where it is helpful to fly in a few members of the team for a couple of days. By having a subset of the team meet, people can form strong one-on-one connections that are harder in a larger group. It also removed the need to organize a large offsite with a lot of preparations and content.
  • Setup an annual team offsite. And by offsite I don’t mean renting a hotel conference room and having everyone give a talk. I mean NodeConf. Find a community event that is not the office, that takes a few days, and that provides plenty of off time for people to hang out and chat. For the last 2 years, we flew the entire team out to CA for NodeConf. Having a non-work context with other people makes everyone more relaxed and against that backdrop, the team constantly found their way to hang out together. The presence of others made it easier not to always obsess about work.
  • We spend about $5000/year/person on travel costs which isn’t significantly higher than travel cost at top companies for Bay Area employees attending conferences and other work travel.

The personal impact

It’s hard to overstate the quality of life remote work provides. For most commuters, it eliminates anywhere between one and three hours of being in the car or train. Those extra hours means you can spend time with your family (having dinner with your kids every day is amazing) or develop a crazy hobby (like running a zoo, which was probably going too far). The flexibility also makes travel easier and allows you to take “days off” without actually missing work because the tools and expectations remove the need to be in one place all the time.

There is some downside. If you look for remote work because your area lacks opportunities, moving away from that job will be more difficult. The need to relocate in order to leave a remote position isn’t ideal (and many employers won’t pay for it). While this is true regardless at some point, it might be easier to move to a rich job market first, and then find a job. The longer you stay at a good remote position, the harder it is to pack up and move.

I heard a lot about the work discipline and strong ethics required to do remote work because of the temptation to sit home and play video games all day instead. I have not found this to be an issue for me or anyone on the team. I also don’t think being remote makes a difference. If you are lazy, you are going to find ways to surf the web all day even if you are in an office.

And last, I hear leaving the house and seeing others is something people like to do. It could be challenging for people to be physically disconnected from others. I find meetups, lunch with friends, and conferences as a good way to balance out the more insulated work environment at home. If you need the social energy of being with other people, and working out of a local co-working space isn’t available or for you, remote work might be a challenge.

A checklist

  • A team can be all remote or all local:
    • Hybrid doesn’t work.
    • Remote doesn’t mean no office – you can still have a place for people to come in when they want to but they must use remote tools all the time.
    • You don’t need to convert your entire organization to remote, but you have to do it in entire teams.
  • Don’t do remote to save money, do it to get better, happier talent:
    • Use payroll cost savings to cover team travel and events.
  • Leverage geographic diversity:
    • Spread over time zones for extended support.
    • Reach out to small tech communities for the best talent.
    • Diversify your team with access to people who don’t want to be another screw in the Bay Area tech machine.
  • Use the right tools:
    • Pick communication tools that work for your team and empower remote culture.
    • GitHub and Slack work really well.
  • Give it time:
    • Building remote culture that works well can take a few months (to a year), make sure to allow it to mature organically.

Got questions? I’m @eranhammer.

Before the Drama

I am going to comment on the recent node fork. Soon. I am not happy about it. I also don’t think it’s bad. I’ve been involved in the conversations with most sides since May and am in a unique position being (probably) the only “guy in the middle” that I think I can provide a perspective that is more complete than most. However, before I do that I would like to defuse the drama.

Given my position at Walmart and the fact that I knew a fork is highly likely for half a year, you can imagine I had a few internal conversations about node and its future inside and outside of Walmart. A large(st) enterprise has to ensure its investments are durable and sound. I shared the situation with my senior management and the message I delivered to them is the same one I am going to deliver to you now.

If there was no new release of node ever again, I would still use and recommend it. I understand people’s desire for faster releases and quicker availability of new JavaScript features but I consider these to be “rich people’s problems”. I spend most of my time writing and managing node development and I feel empowered and productive with the platform I have today.

Can things get better? Absolutely! But this concept of an evolving language and platform is pretty new. I have never imagined new features working a decade on Wall St. building high frequency trading systems in C++. The language barely changed (remember when template supported matured in 1998?). New features were mostly better optimizers and IDEs but not really the language or the platform. I am not being dismissive of progress, but I want to make sure people understand that the node we have today is pretty fucking awesome.

If you are in a decision making position and the recent events make you reconsider adopting node, don’t. Do it – you will not regret it. The current version of node is already fantastic. Again, it can get better, but after two Black Friday events running on this version of node at the biggest eCommerce scale (we did kick some ass this year against major competition) I can tell you without any hesitation that node is production ready today. Cross that. A year ago.

I also want to point out to all the delicate, sensitive souls out there who keep complaining about “all the drama” and “why can’t we all just get along” that the node community drama is amateur hour compared to other platforms. We don’t have lawsuits for hundreds of millions of dollars like Java. We don’t have key members of the project writing pages and pages of nasty blog posts calling the entire platform shit like Rails. We don’t have insane multinational standard bodies debating features of the platform over 10 years like C++. And we don’t start every mailing list response calling the new guy asking the question a fucking asshole like PHP.

I am not dismissing the importance of what is going on, but these events and the way they have evolved shows tremendous maturity and civility that I have not seen in other communities (and unlike most of the brilliant commentators on Hacker News, I have been writing code since 1983). All this drama is a healthy debate about the future of our platform and community and the way it has been handled is something to be proud of.

I am completely behind node. It might be called something else in the future, and there is probably going to be more than one server-side JavaScript platform (which is a good thing), but the foundation of running node-style code to build powerful server solutions is not going away. It is the future of the web.

Wide Open (or, Are You In?)

Earlier this year I confronted the painful realization that my baby framework grew into a mature ecosystem – one I no longer had the capacity to maintain on my own. It started with dragging open issues for more than a few days, to a growing pile of sticky notes on my monitor of ideas I’d like to try, to (and most problematic) no longer remembering how big chunks of the code work.

The problem is, how to successfully move from a one-man-show to a community driven project, without giving up on the stability, consistency, and philosophy of the framework.

Consensus-Dictator-Fork

I believe the only practical model for running a successful open source project is the Consensus-Dictator-Fork (CDF) model. It’s a fancy name for how most open source projects work. Decisions are made by consensus whenever possible. This usually covers 95% of the decisions by the simple mechanism of proposing a change and asking for strong objections. When strong objections are raised and consensus does not emerge, the project BDFL (benevolent dictator for life) makes the final call. If enough people object to that decision, they fork the project and create their own. It is a naturally self-balancing system.

CDF doesn’t solve the problem, but it provides the first building block.

Federation of Modules

The second building block is npm, the package management service. npm makes it trivial to break a large system into smaller modules, each with its own owners and publishing schedule. Instead of applying CDF to the entire ecosystem, I realized I can apply it individually to each module the framework consists of. Each module author has the autonomy to drive their module forward, allowing its dependents to vote with their feel (i.e. package.json files) and switch to another module if they don’t like the changes and fail to influence through consensus.

The smaller the surface area of the hapi module is, the more time I have to focus on the problems I care most about, and the more power is being delegated to other people to drive the many different areas of the framework forward. This is not just about which utilities we use, but about moving core functionality out of the main module. It redefines “core” from the top-level module people require, to the collection of modules used.

The Right People

Before splitting the hapi module into many smaller pieces, I had to identify who is going to take over these new modules. What profile am I looking for. After all, big chunks require more experience while smaller chunks increase the need to communicate. The answer to that was simple, but non-intuitive: everyone who is interested in contributing.

The reason this is non-intuitive is because we are stuck thinking about open source contributions and ownership in terms of merit and meritocracy. These are extremely unhelpful terms and for too many people they translate to “the rich keep getting richer”. Open source meritocracy is supposed to be about letting proven people lead, but in practice it creates a chicken-and-egg problem of making it virtually impossible for new leaders.

I wanted to make participation and leadership within the hapi.js community to be truly open to everyone who is committed to making a contribution. This requires two things: that no one feels excluded or unwelcomed (for any reason), and that everyone feels they are good enough to step up.

Rock Stars

Leading a successful open source project often translates to increased professional success, which in turn translates to more money, influence, freedom, and happiness. It is far easier to find a great engineering job with a famous open source project on your resume. Having a project on npm with 100K daily downloads can open doors 10 years of experience sometimes can’t. Numbers might not get you the job but they will get you through the door.

The best way to make open source sustainable is to make it rewarding. By extending the invitation to lead to others, we create opportunity and rewards that are otherwise very hard to achieve. It is far easier to take over an open source project with existing user base and traction than to start from scratch and battle for attention. The long tail is a lonely place.

Leap of Courage

So how do we convince people to make that leap? To take over a popular open source module and lead it, in public, where every mistake is visible and often amplified? How do we make them feel comfortable to try new things and take risks? And how do we trust them not to screw up? Simple – by having their back.

npm shrinkwrap is not just a command line tool. It is an instrument of social change. By locking down the proven, trusted, stable versions of the framework dependencies, module leads can make mistakes without devastating consequences. By putting safeguards in place to prevent instability, we can hand over core building blocks of the framework to people without demanding “proven track record” or “merit”. We no longer need to be exclusive in who we invite to join our round table, and if it really doesn’t work out, CDF to the rescue.

Support System

Putting risk controls is half the solution. We can’t throw people into the deep end with a lifesaver and expect them to get better at swimming. It is not enough to protect the ecosystem. We have to nurture and grow it. The other half of the solution is the new hapi.js mentoring program. By assigning new contributors and leads an experienced, one-on-one mentor, we increase their confidence and skill, and create a mutually rewarding environment where contributions get exponentially better.

The mentorship program is brand new, but it was based on extensive research and conversations with people to ensure a welcoming, safe, and diverse environment.

Making Room

With everything in place, I spend the last two months smashing hapi into about 20 new modules, all looking for a new lead maintainer. With version 7.1.0 we are now all set to embark on this new, wide open community strategy.

If you ever wanted to participate in a large, meaningful, and highly visible open source project, but did not feel confident (or safe), this is your cue. We are transforming a successful project on its head with the sole criterion of making it a welcoming place for everyone. I am sure we got plenty to improve and iterate on, but the groundwork has been laid, and the doors are open.

Are you in?