Erratum

Our early crawls were archived using the ARC (Archive) format, not the WARC (Web ARChive) format.

The ARC format, which predates WARC, was the initial format used for storing web crawl data. It encapsulates multiple resources (web pages, images, etc.) into a single file, with each resource preceded by a header containing metadata such as the URL, MIME type, and length. While effective, the ARC format has limitations, particularly in terms of extensibility and the ability to store additional metadata.

In contrast, the WARC format, which is an extension of ARC, addresses these limitations: it allows for more comprehensive metadata, better handling of content types, and the capability to store additional information such as HTTP headers, which are crucial for a more accurate representation of the archived data.

More information about these formats can be found in our blog post Web Archiving Formats Explained.