There used to be a big difference between API access and regular human-oriented HTML access: the speed in which requests are made. When a request is made via a browser, there is inherit delay from human interaction, browser response, page rendering, and fetching of images. Most of this is gone once a machine makes the call. However, with recent improvement in browser technology and the wide use of AJAX techniques on the client side, even the human-readable pages can make API calls to render pages.
Scalability plays a central role when designing the ways in which data can be requested from a service, be it via an API call or HTML page request. Both types fetch raw data, process it, and then format it into a presentation format such as HTML, XML, JSON, etc. In the case of a server-rendered HTML page, all the different requests are made internally, hidden from the user, and a single page is returned. If the page uses AJAX scripts, the browser makes multiple API calls to fetch individual data sets, but the server still has to fetch the raw data, process it, and format it. It is the size of the batch that makes the difference.
When discussing microblogging scalability, the conversation includes scaling each individual service, but also scaling the network and relationship between services. Part I discussed the challenges of scaling a single microblogging site with focus on dealing with a large and constantly changing content database. In that post I mentioned that the proposal by some critics to build a distributed or federated microblogging service as a scaling solution will actually make thing worse. This second part will elaborate on that claim.
When discussing a distributed microblogging service, the conversation touches the long debate on the future of social networks and linking communities across individual walled gardens. After all, microblogging is one aspect of the social web, and status updates lives side by side sharing photos, videos, and other personal information and experiences. Being able to choose a social network and make friends from another without having to sign up for multiple accounts is one of the visions being offered. Another is the approach being advocated by the Data Portability group, which focuses on being able to move an entire experience off to another network instead, creating multiple identities.
When it comes to Twitter, everyone’s a critic.
The irony is, the majority of the technical criticism written about Twitter reveals more about the lack of understanding of the author than anything about Twitter. Creating Nouncer – a developer platform for building microblogs and similar services – provides a firsthand understanding of the inner-working and challenges of microblogging sites. This post is not specifically about Twitter but as the leading microblogging service, it is the best working example of the challenges of scaling in the space.
People don’t seem to get what is so hard about scaling the Twitter service. Some think it has to do with Rails, while others look at the hosting company or other operating factors. Then there are those who incorrectly think a distributed or federated service is the solution (which will only make things worse). The idea that building a large scale web application is trivial or a solved problem is simply ridiculous. In a way it is surprising that there are so few companies out there dealing with commoditizing web developing scaling.
Building a business out of a microblogging service is not a trivial task. There has been a lot of chatter regarding Twitter’s eventual monetization, but so far the only people who made money off microblogging are the Jaiku founders when they sold the business to Google. I have written before about the difficulty of monetizing microblogging, but difficulty is something to overcome, not a brick wall. This model is an attempt to figure out the different options of running such a business.
You can try the Google Docs version (if you want to play with it make a copy first). If you know your way around financial models, jump right into the model. Continue reading for some background and points of interest.
Being critical of Twitter is really a compliment. It comes with the territory of being the market leader in a new space. My recent negative rants about the Twitter are really about how it is being used than its qualities. Like many others, I have a vested interest to see Twitter succeed. While Nouncer does not compete with Twitter, it builds upon the usefulness and experience of microblogs users. Most of my points about Twitter apply equally to other microblogs like Jaiku, Pownce and others. And there are many others.
The best features Twitter has to offer are their powerful platform and open API. It is also the reason they are more successful and why others are coming out with their own API almost as fast as their website. I am excited about the soon to be released Pownce API and have been playing around with the Jaiku API. These three sites and the many who try to improve the space (using their lower load as an advantage to build new functionality), all serve an important purpose of getting microblogging into the mainstream. We are still in the imagination phase, trying to figure out what to do with this powerful tool we stumble upon.
Phil Barrett has an interesting post about the usefulness of Twitter and other microblogging sites:
Having said that, some people have said that Twitter has become a hot newswire – industry insight and news is sometimes available through a Twitter feed before it hits the blogs or news pages. I guess that’s important for some industry people…but does anybody else really care if they hear about industry news 3 hours before they read it in their RSS newsreader? One sign that Twitter has become a hit is the fact there are lots of imitations now available globally – like Pownce, Jaiku and Dodgeball. Wikipedia reports that there are over 100 knock-offs of Twitter now. It seems like instant messaging 2.0 has arrived.
First, I would not call Jaiku a Twitter imitation as both appeared around the same time. Dodgeball was up before Twitter and is in a related but different space – Twitter does not have location-based services (which Jaiku does on their mobile application). And (I hope) Barak Obama does not update his Twitter account – a staffer on his campaign is in charge. Spending a year running tech ops for Ehud Barak’s first (successful) campaign for Prime Minister of Israel, I can’t see a Presidential candidate being allowed to post tweets – it is too much of a PR liability, and I’d be surprised if he had the time.