Discovery and Content Negotiation

Content NegotiationAfter following a recent exhausting debate regarding the use of content negotiation (conneg) as a method for obtaining resource descriptors (metadata), I feel compelled to address this issue. When trying to sell link-based discovery, many argue (some religiously) that content negotiation is a better alternative. But first, I need to explain what they mean.

Content negotiation is an HTTP feature that allows the client to state its preferred content type for the requested representation. Assuming web servers can support multiple formats, a client can ask for an HTML representation of a resource over a plain text one, XHTML over HTML, or it can ask for a GIF over JPEG, etc. When applied to discovery, if we define example/metadata as a descriptor format, asking for that content type will produce the metadata while asking for, say text/html, will produce a representation of the resource itself.

The biggest advantage of using content negotiation is that it can be done with a single round trip. The client asks for the descriptor directly, saving bandwidth and improving performance. This of course, is based on the assumption that a descriptor is nothing more than another representation of a resource.

Sometimes this is a valid argument. For example, some image formats allow the inclusion of metadata (such as JPEG) while other don’t. It is reasonable to say that if a JPEG representation includes both the data (image) and metadata (tags), content negotiation can be used to ask for just part of it (just data or just metadata). In this case, the metadata representation is simply a partial view.

One example for a proposed discovery protocol using content negotiation this way is Metalink. Metalink is a descriptor format for file downloads. The view of some in the Metalink community is that a Metalink representation is nothing more than a highly degraded version of the file being downloaded. It doesn’t contain the file itself, just information on where to find it and how to best download it. When a download URI is entered into a download manager, it can simply ask for the Metalink representation (application/metalink+xml) of the file and get it directly without having to first interact with the file itself (via HTTP GET or HEAD).

I strongly disagree with the view that a descriprot is nothing more than another representation of the same resource. But even if we ignore that, using content negotiation in Metalink isn’t a complete solution. For example, using links can help a browser detect that a website supports Metalink in the first place and offer a way to obtain such descriptors without having to use a trial-and-error approach.

If the Metalink format is a valid representation of a file (which I am not convienced it is), then Metalink’s use of content negotiation is a valid form of discovery. But content negotiation isn’t really a good way to perform discovery. In the Metalink use case, content negotiation works (in the technical sense) because Metalink isn’t a very demanding protocol. Any given file only has a single Metalink description, and the Metalink document format doesn’t have much use (at least for now) other than as a descriptor for file downloads. It is a highly specialized content type with a very simply relation structure.

Content negotiation fails multiple tests for being included as part of a uniform discovery framework:

  • Content negotiation overloads content-type with relation type, or worse, an application specific activity. Links are the most basic building block on the web. It all started with hypertext after all. One of the guiding principles is the separation of relations and representations. If we overload a representation format with a relation meaning, we are taking away from the modularity and flexibility of the web. Consider XRD, it is a useful schema for many things. But if we define application/xrd+xml to mean ‘a descriptions of the resource identified by this URI’, we can’t really use it as an independent format anymore.
  • It requires minting new content types that are limited to representing metadata. A quick look at a typical Windows (OS) registry for file types or URI schemes shows just how broken this approach is. Too many highly specialized formats and custom schemes make interoperability impossible to manage and it breaks as soon as someone comes up with a new way of using old building blocks.
  • There is no way to find meta-metadata, that is, a descriptor about a descriptor. Given three resources, C describes B and B describes A, content negotiation fails to offer any practical access to C. We can mint a new content type for a description of a description, but that does not scale. Consider a simple use case: a blog (A) has a descriptor with information about the owner’s address book (B). However, that information is protected and requires authorization. The information about accessing this descriptor is contained in a separate descriptor (C). Content negotiation doesn’t give us easy access to C.
  • It doesn’t allow for a one-to-many relation structure. If a single resource has multiple descriptors using the same content type, there is no easy way to list them or access them. HTTP provides the 300 status code for such cases but it is not widely deployed or understood, nor is it transparent for accessing the resource itself.
  • It does not provide a unique URI for the descriptor which prevent it from being linked and used as any other web resource.

If you are working on a discovery protocol for your use cases, and content negotiation comes up as a proposed solution, please consider the above points when making a decision. The Link-based framework being developed doesn’t mean to exclude content negotiation a tool, but it is designed to offer a much more uniform framework that can be applied to almost any use case.