Errata

Here you can find comprehensive information about errata that affect our data releases, including crawl data, and web graphs. If you have any problems to report please Contact Us.

Content is truncated

Originally reported by 
.

Some archived content is truncated due to fetch size limits imposed during crawling. This is necessary to handle infinite or exceptionally large data streams (e.g., radio streams). Prior to March 2025 (CC-MAIN-2025-13), the truncation threshold was 1 MiB. From the March 2025 crawl onwards, this limit has been increased to 5 MiB.

Earliest crawl:
 
CC-MAIN-2008-2009
 
Latest crawl:
 
Read More...

No truncation indicator in WARC records

Originally reported by 
Henry Thompson
.

Due to an issue with our crawler, not all truncations were indicated correctly. A workaround to detect length truncation is to be suspicious if the length of the content is exactly 1048576 bytes. Truncations for time or network do not have such a workaround. In the WARC files this indicator is called "WARC-Truncated".

Earliest crawl:
 
CC-MAIN-2008-2009
 
Latest crawl:
 
CC-MAIN-2019-30
Read More...

Missing content_truncated flag in URL indexes

Originally reported by 
.

The flag in our URL indexes (CDX and columnar) that indicates whether or not a WARC record payload was truncated was added in CC-MAIN-2019-47. This indicator is missing in our indexes for all previous crawl releases. In the CDX index this is referred to as "truncated", and the columnar index refers to this as "content_truncated".

Earliest crawl:
 
CC-MAIN-2008-2009
 
Latest crawl:
 
CC-MAIN-2019-43
Read More...

SURT URLs do not properly encode non-UTF-8 percent-encoded characters

Originally reported by 
Tom Morris
.

When constructing SURT (Sort-friendly URI Reordering Transform) URLs, percent-encoded characters that are not valid UTF-8 sequences were not being correctly handled. This could lead to inconsistencies in URL normalization and sorting, potentially causing incorrect deduplication or retrieval issues in datasets that rely on SURT-based indexing.

Earliest crawl:
 
CC-MAIN-2017-04
 
Latest crawl:
 
CC-MAIN-2024-51
Read More...

WAT data: repeated WARC and HTTP headers are not preserved

Originally reported by 
.

Repeated HTTP and WARC headers were not represented in the JSON data in WAT files.

Earliest crawl:
 
CC-MAIN-2013-20
 
Latest crawl:
 
CC-MAIN-2024-46
Read More...

WARC revisit metadata records

Originally reported by 
.

The revisit records in the Common Crawl WARC archives (since Aug 2018) lack the metadata record which is attached to all response records.

Earliest crawl:
 
CC-MAIN-2018-34
 
Latest crawl:
 
CC-MAIN-2024-46
Read More...

Erroneous title field in WAT records

Originally reported by 
Robert Waksmunski
.

The title field in WAT record is extracted from last but not first <title> element in an HTML page

Earliest crawl:
 
CC-MAIN-2013-20
 
Latest crawl:
 
CC-MAIN-2024-33
Read More...

Missing fetch_status fields

Originally reported by 
"NL" via Discord
.

In our columnar index for this crawl, the `content_mime_type` is missing and `fetch_status` is always -1. In the cdx index (columnar: `content_mime_type`), fields "mime" and "status" are missing.

Earliest crawl:
 
CC-MAIN-2015-06
 
Latest crawl:
 
CC-MAIN-2015-11
Read More...

Redundant extra line in response records

Originally reported by 
Greg Lindahl
.

The WARC files of the August 2018 crawl contain a redundant empty line between the HTTP headers and the payload of WARC response records.

Earliest crawl:
 
CC-MAIN-2018-34
 
Latest crawl:
 
CC-MAIN-2018-34
Read More...

Incorrect fetch_time metadata

Originally reported by 
.

The fetch_time metadata for robots.txt might be incorrect in some crawl releases.

Earliest crawl:
 
CC-MAIN-2016-36
 
Latest crawl:
 
CC-MAIN-2019-47
Read More...

Charset Detection Bug in WET Records

Originally reported by 
Javier de la Rosa
.

The charset detection required to properly transform non-UTF-8 HTML pages in WARC files into WET records didn't work before November 2016 (CC-MAIN-2016-50) due to a bug in IIPC Web Archive Commons.

Earliest crawl:
 
CC-MAIN-2008-2009
 
Latest crawl:
 
CC-MAIN-2016-44
Read More...

ARC Format (Legacy) Crawls

Originally reported by 
.

Our early crawls were archived using the ARC (Archive) format, not the WARC (Web ARChive) format.

Earliest crawl:
 
CC-MAIN-2008-2009
 
Latest crawl:
 
CC-MAIN-2012
Read More...

Missing Language Classification

Originally reported by 
.

Starting with crawl CC-MAIN-2018-39 we added a language classification field (‘content-languages’) to the columnar indexes, WAT files, and WARC metadata for all subsequent crawls.

Earliest crawl:
 
CC-MAIN-2008-2009
 
Latest crawl:
 
CC-MAIN-2018-34
Read More...

Some 2–Level CCTLDs Excluded

Originally reported by 
.

A bad configuration was checked into our exclusion list on Sep 22, 2022 and was fixed on Oct 27, 2023. The configuration blocked a number of 2–level domains, meaning they were not included in certain crawls.

Earliest crawl:
 
CC-MAIN-2022-40
 
Latest crawl:
 
CC-MAIN-2023-40
Read More...