Introducing Metalink

MetalinkThe majority of the time, downloads just work. Most downloads are relatively small files. But, when files are larger, you are more likely to encounter errors. Errors with large downloads can be frustrating and a waste of time. That’s even more true for areas with unreliable Internet connections, such as developing parts of the world.

Download problems include when a server becomes overloaded, goes down, or if there are errors in transmission. Common download programs like web browsers usually don’t have the capability to recover from most errors, but external download managers do. Download managers are separate programs that your browser can pass downloads to. Download managers usually include features like making use of mirrors for fail-over or download from multiple servers.

Before Metalink, downloading a large file like a Linux distribution (CD or DVD size) could involve copy & pasting various URLs for mirrors into your download manager. Once the download completed, you could manually checksum the file, if you had such a program, to verify that the download was not corrupted. If the download was corrupted, there were a few things power users could try, but most people had to start over from scratch.

With Metalink, we make this process automatic. Mirror lists, P2P sources, checksums, digital signatures, and other metadata about a download are included in an XML file which download programs can read.

Checksums allow errors in a file to be repaired. Other features include enhanced security, adding multiple files to a download queue, and simultaneously downloading from multiple sources, including P2P & FTP/HTTP. Clients are enabled to work their way to a successful download even under adverse circumstances. All this can be done transparently to the human user and the download is much more reliable and efficient. Most download managers support Metalink already, but if this feature is made available in browsers like Firefox and Chrome, it will reach more people.

Metalinks are frequently used for large files or files stored on many mirrors. Because of this, many open source projects like cURL, OpenOffice.org, openSUSE, Fedora, Ubuntu, & others use them to turn their mirrors into a coordinated Content Distribution Network. About 50 applications support Metalinks. Package update programs like Appupdater and yum use them behind the scenes. File hosting services are starting to provide Metalinks for downloads, and search engines are starting to index them.

For example, The following is the Metalink document used with the Ubuntu distribution (ubuntu-9.04-alternate-amd64.iso.meta4):

<?xml version="1.0" encoding="utf-8"?>
<metalink xmlns="urn:ietf:params:xml:ns:metalink">
    <file name="ubuntu-9.04-alternate-amd64.iso">
        <size>732282880</size>
        <publisher name="Ubuntu" url="http://www.ubuntu.com"/>
        <license name="GPL" url="http://www.gnu.org/licenses/gpl.html"/>
        <version>9.04</version>
        <description>Ubuntu CD Image</description>
        <logo>http://www.ubuntu.com/themes/ubuntu07/images/ubuntulogo.png</logo>
        <os>Linux-x64</os>
        <hash type="md5">3b5e9861910463374bb0d4ba9025bbb1</hash>
        <metaurl type="torrent" priority="1">http://releases.ubuntu.com/9.04/ubuntu-9.04-alternate-amd64.iso.torrent</metaurl>
        <url location="jp" priority="1">http://ftp.yz.yamagata-u.ac.jp/pub/linux/ubuntu/releases/9.04/ubuntu-9.04-alternate-amd64.iso</url>
        <url location="de" priority="2">ftp://ftp.rrzn.uni-hannover.de/pub/mirror/linux/ubuntu-releases/9.04/ubuntu-9.04-alternate-amd64.iso</url>
        <url location="gb" priority="2">http://ftp.ticklers.org/releases.ubuntu.org/releases/9.04/ubuntu-9.04-alternate-amd64.iso</url>
        <url location="us" priority="3">http://ubuntu.media.mit.edu/ubuntu-releases/9.04/ubuntu-9.04-alternate-amd64.iso</url>
   </file>
</metalink>

Most of the elements are self explanatory: file name, file size, who published it, what license it is released under, what version, a description, a logo, what operating system, a file hash, and where the file is available (.torrent and FTP/HTTP mirrors). <url> elements contain a geographical location and priority. <metaurl> elements list metadata files, or metainfo files, like torrents, metalinks or others and have a priority attribute, that is used with <url>. In this instance, the download sources should be used in this order: Japan & torrent (if the client supports BitTorrent), Germany & Great Britain, and finally USA.

The Metalink project and the XML format are 4 years old. Over a year ago, we started working on an IETF Internet Draft to better specify Metalink. The new XML format is an evolution of the previous one and offers all the same features. If you’re interested, please take some time to help us improve it.