The landscape of the community-engineered social web, the one based on open technologies, has changed dramatically over the past few months. If you took a year off and just came back, you would probably not recognize it at all.
The movement that started with protocols such as OpenID, OAuth, and Activity Streams, is now mostly gone. All the cool kids got grownup jobs and the market is back again driven by a small number of corporations. In fact, it is so small it can be counted on two fingers. A year ago, a meeting with Chris Messina, David Recordon, Joseph Smarr, Monica Keller, Will Norris, Luke Shepard, and John Panzer represented 7 different organizations or communities – a well-balanced mix of big and small, corporate and independent.
Today it’s just Facebook and Google and that has significant implications. But when examining how these two companies engage in the development of open technologies, the findings are quite surprising. On the product side, Google is famous for their openness while Facebook is notorious for their closed garden. But when it comes to their community engagement, these two giants behave in a rather reverse fashion.
Open technology is slow by definition.
That’s assuming you accept my definition of Open Technology:
Developed in the open with full transparency
Open process for anyone to participate freely
Everyone is free to implement
Decisions are made based on technical merit
Open technology doesn’t have to happen in a standards body or format open source organization, but important work usually does; because if it’s important, there are simply too many people collaborating to be successful without a well-defined process and governance. Successful open technology acts very much like a proprietary one – they both tend to block or delay new innovation. As soon as something becomes successful, it poses high risk to those absent from the process.
OAuth 1.0 created a myth that it is possible to develop open technology fast. But in reality, OAuth 1.0 took a little over a year, and that’s with a tiny, well-aligned group of people almost exclusively from one town, with very little at stake. The numbers are even less impressive if you include getting to revision A as part of the 1.0 lifecycle.
OpenID had a very similar experience where version 1.0 (really 1.1) took little time (mostly created by two people) while version 2.0 took a very long time. Deciding what the next version of OpenID looks like seems to take even longer.
Two years ago I was one of the leaders of this movement. It was the main driving force behind creating the Open Web Foundation (i.e. a lightweight legal framework suited for small, community driven projects). The problem with this approach is that it ignores why and how companies develop open technology, and the real reasons why it takes time.
People new to standards like to accuse the excessive process and legal framework for the time it takes to produce a standard. But this argument is simply not grounded in reality. For example, XRD 1.0 which is quickly becoming one of the building blocks of the social web took over 2 years to mature from XRDS through XRDS-Simple to XRD. During those 2 plus years, the OASIS process (where the work was done) did not slow us down by more than a week or two. What slowed XRD’s development was lack of feedback, review, and editorial time. But even these factors were not the primary reason.
There are two reasons why specifications take time: building consensus and reaching maturity. Consensus time is usually a function of the group size. The bigger the group the longer it takes. Maturity however, is a function of taking intentional breaks during the process to let the technology sink in, and allow time for experimentation. Maturity is what differentiates a useful technology from an essential one.
If I listened to the many calls to get XRDS-Simple done two years ago because people wanted to ship stuff “now“, we would have ended with broken discovery architecture and a complex format that no one really needed. On occasion, these demands for finalizing specifications took the form of subtle threats to develop competing proposals. Getting the right people to read specifications takes time, and it often means waiting for the use cases to catch up with the vision.
If you compare the initial proposal of site-meta with the final RFC for Well-Known URIs, you immediately notice a remarkable transformation which was the result of a yearlong discussion and collaboration across three standards bodies. My discovery work follows the same pattern, from the initial OAuth Discovery work to the current, much simplified host-meta proposal (and the retirement of the LRDD stand-alone specification).
Like anything else, early adopters have to accommodate this reality, and it can often lead to frustration. The Open Web Foundation is one example for how this frustration with standards bodies materialized into an (unsuccessful) attempt to create a whole new framework. While the foundation was very successful in creating a useful legal license, it was not significant enough to get new technology out faster. Ironically, it made it much easier for companies to develop new technologies internally, the open them up when done.
In order to understand the forces that drive the development of open technologies, we must first examine how companies address these issues. There are generally three categories of companies when it comes to interaction with open technology: conservative, agile, and delayed-open. It is important to understand that these categories are specific to how companies interact with open technology, and are not an indication of how the interact in the marketplace.
Traditional companies are rarely first to adopt new technology, wait for final versions before implementing, and actively participates only when a specification takes a wrong turn. They are usually absent from the process or lurk around, but engage with some form of indirect support (such as sponsoring events or supporting the community). Traditional companies tend to focus on building business relationships with other similar companies and establish back-channel alignment, which enables them to let others represent their interests. Yahoo! and Twitter are good examples of traditional companies.
Agile companies live on the cutting edge, deploying early drafts and are not afraid to change quickly. They are usually very hands on, often leading the development efforts, and maintain some level of control over the process, which helps reduce risk by having better understanding of the proposal’s stability. Agile companies are usually very small where being agile provides a necessary edge, or very big where they can afford to take more risk and experiment because of their market dominance. Facebook and (the now defunct) Pownce are good examples of agile companies.
Delayed-open companies are very selective in what they choose to participate in, and are rarely involved in efforts they do not control. The basic strategy of delayed-open companies is to develop as much as possible internally and confidentially. They only share their work when they are ready to ship a product or when they are confident in the stability of the technology. While these technologies end up being open (typically by opening their development to final enhancements, or by contributing them to a standards body), by the time they are opened up, very little can really change. Google and Microsoft are good examples of delayed-open companies.
Companies looking to engage in the development and utilization of open technologies must find the right balance that fits their culture and market needs. Google and Facebook provide important case studies on how companies approach open technology from opposite ends of a given market. When it comes to social web applications, Facebook is the undisputed market leader, while Google is one of the (hardest trying but) least successful players.
Both companies recently made high profile hiring moves, grabbing every free-agent open technologist in the space. Facebook hired David Recordon and Monica Keller (joining Luke Shepard and others), while Google hired Chris Messina, Joseph Smarr, and Willl Noris (joining John Panzer, Brad Fitzpatrick, and others). But their reasons for doing so are fundamentally different, dictated by their involvement category.
Facebook, with time on their side, hired people to help open up their technology and use their market dominance to influence and lead new efforts. Their position allowed them to deploy an early draft of OAuth 2.0 in a highly publicized fashion, and to promise keeping their production code up to date as the specification matures. To accomplish that, their engineers engaged early and intensely, contributing the first draft and running code. The more Facebook invests in open technologies, the longer it takes these technologies to mature (due to their sudden importance and popularity), but with a higher likelihood of maturity.
Google on the other hand, hired top caliber talent for very different reasons. Google is significantly behind Facebook and cannot afford to take years (or even months) for new technologies to mature. They need to build products and ship them quickly, and that requires much more control than is available in open development. By hiring highly respected individuals like Chris Messina and Joseph Smarr, Google is hoping to get away with doing more work delayed-open. Salmon, PubSubHubbub, WRAP, and their OpenID extensions are all examples of past technologies developed successfully using this method.
While Facebook shipped an early draft of OAuth 2.0, Google shipped WRAP. Both promised to ship the final OAuth 2.0 protocol.
It is hard to say which category ends up making the biggest contribution to the development of open technologies. Google is clearly a leader in adopting open technology in general, but the way in which they get comfortable doing so isn’t always the most open or polite. Google’s style helps get more free technology faster, but it also makes is much harder for other companies to participate when the foundation is established. Google is leading an Open war, and in war, there are often innocent casualties.
There is a direct correlation between a company’s ability to control the process and their embrace of the technology. Facebook came in late to the WRAP party, leading them to favor the (at the time) less prominent and less successful OAuth 2.0 effort at the IETF (while Google did their very best to derail the IETF effort). In the same spirit, Google’s bear hug of HTML5 came only after they guaranteed their strong control over the process by hiring the specification editor (for the sole purpose of getting HTML5 done fast).
The manifestation of these two approaches is sometimes ironic. Facebook who is not known for their embrace of open protocols and technology, is the one enabling open communities to take their time and spend a little longer to get the details right. On the other hand, Google who is considered the biggest champion of openness is doing their best to push efforts to a quick conclusion, strongly aligned with their immediate needs.
While this analysis includes a slight bias in favor of the contribution of agile companies, delayed-open companies play an important role in creating alternatives, not possible by purely open means alone. Facebook only turned the page and joined the open community when it felt the open community no longer posed a threat to their dominance, and their behavior depends greatly on the individuals leading the effort. The Facebook influence at the hands of David Recordon (who at this point is by far the most influential voice in advocating open technologies) is an extremely constructive force. However, it is clear that without Google chasing their tail and portraying them as closed, Facebook will be much less motivated to open up.
It is also important to remember the significant contribution of traditional companies. They provide the final seal of approval for new technologies and are the key to mass adoption. They are also critical in getting enough community sponsorship to allow work to continue.
At the end we rely on a few individuals to do the right thing. On the Google side, we rely on people like Chris Messina and Joseph Smarr to know where to draw the line between delayed-open and fully-open. Because delayed-open leave very little for the community at large to influence, they must find the right balance between pragmatic time-to-market and forcing their own ideas on the rest of us. When Chris and Joseph joined Google, I predicted exactly this kind of shift. I have confidence that their work will continue to be innovative and inspiring, but the silence and disengagement coming from the Google team over the past six months is a real cause for concern.