Common Crawl

factory

providers.commoncrawl()

index

CDX + WARC

reads bodies

yes

provider=all

included

Load it

import { createArchive, providers } from "@agntn/archives";

const latest = createArchive(providers.commoncrawl());
const pinned = createArchive(providers.commoncrawl({ collection: "CC-MAIN-2023-50" }));

What it lists

Common Crawl publishes one index per crawl. Without collection, the provider reads index.commoncrawl.org/collinfo.json and uses the newest crawl it advertises; with collection, it queries that crawl's CDX API directly. Rows carry digest, mime, length and the collection in _meta, plus the WARC filename and offset that locate the body.

A 404 with No Captures found is the index's way of saying the URL was never crawled; the provider turns it into an empty listing rather than a failure. Every other 404 keeps meaning failure.

Reading bodies

content() requests the byte range offset..offset+length of the WARC file on data.commoncrawl.org, then parses the WARC record and the HTTP response inside it, including chunked transfer encoding and gzip. The result is the original response body, decoded with the charset the record declares.

Gotchas

• One crawl is a snapshot in time, not a history. A URL crawled once appears once; for a timeline across years, query several collections or another archive.
• collinfo.json is fetched on every uncached listing when no collection is pinned. Pin one in services that run for long.
• The index host is occasionally unreachable; when every archive is asked, that shows up as commoncrawl: … fetch failed under _meta.errors, and the other providers still answer.

Where it lives

src/providers/commoncrawl.ts. WARC parsing lives in src/utils/_content.ts.