---
title: "Common Crawl"
description: "One CDX index per crawl, and bodies read straight from the WARC byte range."
canonical_url: "https://archives.agntn.dev/providers/commoncrawl"
---
# Common Crawl

> One CDX index per crawl, and bodies read straight from the WARC byte range.

<provider-facts slug="commoncrawl">



</provider-facts>

## Load it

```ts
import { createArchive, providers } from "@agntn/archives";

const latest = createArchive(providers.commoncrawl());
const pinned = createArchive(providers.commoncrawl({ collection: "CC-MAIN-2023-50" }));
```

## What it lists

Common Crawl publishes one index per crawl. Pin `collection` and the provider asks that crawl's CDX API directly. Leave it out and the calendar decides. `index.commoncrawl.org/collinfo.json` dates every crawl, and the provider reads it first. No window? The newest crawl. A `from`/`to` window gets the newest crawl that ran inside it. So 2019 lands in December 2019, not in the latest crawl, which has never heard of 2019. A window no crawl touched comes back empty, and no index gets asked. Rows carry `digest`, `mime`, `length` and the `collection` in `_meta`, plus the WARC `filename` and `offset` that locate the body.

A 404 with `No Captures found` is the index's way of saying the URL was never crawled. The provider turns it into an empty listing, not a failure. Every other 404 keeps meaning failure.

## Reading bodies

`content()` requests the byte range `offset..offset+length` of the WARC file on `data.commoncrawl.org`, gunzips the record and parses the HTTP response inside it. Chunked framing comes off first. Then any `Content-Encoding` the record kept: gzip, deflate (zlib or raw, servers mix those up), brotli or zstd. The result is the original response body, decoded with the charset the record declares.

Common Crawl usually stores the body already decoded and leaves an `X-Crawler-Content-Encoding` note instead. So that last step is for the odd record. Got one in `compress`? You get an error that names it, not a page of noise.

The decoders are plain JavaScript from `@agntn/compressions`. No `DecompressionStream` needed, and each one loads the first time a record asks for it. A gzip read never drags the brotli dictionary along.

A `timestamp` picks the crawl the way it picks the capture. The newest crawl that began by then wins. Ask about 2005 and you get the oldest crawl there is. Closest it gets.

## Gotchas

- One crawl is a snapshot in time, not a history. A window reads one crawl, the newest inside it. So `from: "2019"` shows December 2019. A page that crawl skipped comes back empty, even if November had it. For a timeline across years, pin several collections or ask another archive.
- A small `maxBytes` keeps the download small too. The record comes down in doubling slices until one decodes past the cap. A 4 KB read of a 275 KB record stops long before the end.
- `collinfo.json` is fetched on every uncached listing when no collection is pinned. Pin one in services that run for long.
- A lookup can take longer than ten seconds, so Common Crawl waits 60 unless you name a `timeout`. A busy index gives up on its own after about eleven and answers `504 Gateway Time-out`. That's the server, not your deadline.
- The index host is occasionally unreachable. When every archive is asked, that shows up as `commoncrawl: … fetch failed` under `_meta.errors`, and the other providers still answer.

## Where it lives

`src/providers/commoncrawl.ts`. WARC parsing lives in `src/utils/_content.ts`.


## Sitemap

See the full [sitemap](https://archives.agntn.dev/sitemap.md) for all pages.