Common Crawl

Load it

import { createArchive, providers } from "@agntn/archives";

const latest = createArchive(providers.commoncrawl());
const pinned = createArchive(providers.commoncrawl({ collection: "CC-MAIN-2023-50" }));

What it lists

Common Crawl publishes one index per crawl. Pin collection and the provider asks that crawl's CDX API directly. Leave it out and the calendar decides. index.commoncrawl.org/collinfo.json dates every crawl, and the provider reads it first. No window? The newest crawl. A from/to window gets the newest crawl that ran inside it. So 2019 lands in December 2019, not in the latest crawl, which has never heard of 2019. A window no crawl touched comes back empty, and no index gets asked. Rows carry digest, mime, length and the collection in _meta, plus the WARC filename and offset that locate the body.

A 404 with No Captures found is the index's way of saying the URL was never crawled. The provider turns it into an empty listing, not a failure. Every other 404 keeps meaning failure.

Reading bodies

content() requests the byte range offset..offset+length of the WARC file on data.commoncrawl.org, gunzips the record and parses the HTTP response inside it. Chunked framing comes off first. Then any Content-Encoding the record kept: gzip, deflate (zlib or raw, servers mix those up), brotli or zstd. The result is the original response body, decoded with the charset the record declares.

Common Crawl usually stores the body already decoded and leaves an X-Crawler-Content-Encoding note instead. So that last step is for the odd record. Got one in compress? You get an error that names it, not a page of noise.

The decoders are plain JavaScript from @agntn/compressions. No DecompressionStream needed, and each one loads the first time a record asks for it. A gzip read never drags the brotli dictionary along.

A timestamp picks the crawl the way it picks the capture. The newest crawl that began by then wins. Ask about 2005 and you get the oldest crawl there is. Closest it gets.

Gotchas

• One crawl is a snapshot in time, not a history. A window reads one crawl, the newest inside it. So from: "2019" shows December 2019. A page that crawl skipped comes back empty, even if November had it. For a timeline across years, pin several collections or ask another archive.
• A small maxBytes keeps the download small too. The record comes down in doubling slices until one decodes past the cap. A 4 KB read of a 275 KB record stops long before the end.
• collinfo.json is fetched on every uncached listing when no collection is pinned. Pin one in services that run for long.
• A lookup can take longer than ten seconds, so Common Crawl waits 60 unless you name a timeout. A busy index gives up on its own after about eleven and answers 504 Gateway Time-out. That's the server, not your deadline.
• The index host is occasionally unreachable. When every archive is asked, that shows up as commoncrawl: … fetch failed under _meta.errors, and the other providers still answer.

Where it lives

src/providers/commoncrawl.ts. WARC parsing lives in src/utils/_content.ts.