Skip to content

Wayback Machine API: CDX Queries for Domain History

How the Wayback Machine API works: the Availability and CDX endpoints, key parameters and ready queries for checking an expired domain's history.

9 min read

The Wayback Machine API is really two public endpoints. The Availability API tells you whether a URL has been archived and returns the capture closest to a date you choose. The CDX Server API returns the raw capture index: every snapshot of a URL, host or whole domain, with its timestamp, status code and content type, and it can filter and collapse the results on the server. The Internet Archive's documentation lists no key for public captures, so both work with a plain HTTP request. For vetting expired domains in bulk, the CDX API is the one you will use most.

This guide covers the parameters that matter for domain history and gives ready queries you can paste into a browser, curl or a script. If you want to read one domain's history by hand, the Wayback Machine domain history guide walks through the calendar and the /* view instead.

The Wayback Machine availability API

The Availability API answers a single question: is there a capture of this URL, and which one is closest to a given time?

https://archive.org/wayback/available?url=example.com
https://archive.org/wayback/available?url=example.com&timestamp=20120101

It takes three parameters, according to the official Wayback API page:

  • url (required): the address to look up.
  • timestamp (optional): the target time in YYYYMMDDhhmmss form, 1 to 14 digits. Without it, you get the most recent capture.
  • callback (optional): wraps the answer for JSONP.

A hit returns an archived_snapshots object with one closest entry. That entry holds available, the Wayback url of the snapshot, its timestamp and its HTTP status. A miss returns an empty object: {"archived_snapshots":{}}.

That is enough to sort a long list of candidates into "has history" and "has none". It is not enough to read history: one snapshot tells you nothing about the years between. An empty answer can also mean the site was excluded from the archive, which is a separate case covered in vetting a domain the Wayback Machine has excluded.

The Wayback Machine CDX API

The CDX server returns the index the Wayback Machine itself uses to look up captures. The base request is:

https://web.archive.org/cdx/search/cdx?url=example.com&output=json&limit=5

Each line describes one capture. The default fields are:

Field What it holds
urlkey The URL in sortable (SURT) form
timestamp When it was captured, YYYYMMDDhhmmss
original The URL as it was crawled
mimetype Content type, such as text/html
statuscode HTTP status the crawler received
digest A hash of the content
length Size of the stored record

With output=json the answer is a JSON array whose first row lists the field names. Use fl to return only the fields you need, for example fl=timestamp,original,statuscode.

The CDX parameters that matter for domain history

The CDX server documentation, linked from the Wayback API page above, lists more options than you need. These are the ones that do the work:

  • matchType: exact (the default), prefix, host or domain. Use host for every URL on example.com, and domain to include subdomains as well. Writing url=example.com/* is the same as prefix.
  • from and to: an inclusive date range, 1 to 14 digits. from=2015&to=2018 limits the answer to those years.
  • filter: filter=field:regex, with ! in front to invert it. You can repeat it. filter=statuscode:200 keeps successful pages; filter=!mimetype:image.* drops images.
  • collapse: removes adjacent duplicates on a field, or on its first N characters. collapse=timestamp:4 keeps the first capture of each year; collapse=urlkey keeps one line per URL; collapse=digest skips captures whose content did not change.
  • limit: limit=N returns the first N results, limit=-N the last N. The server caps a single query at 150,000 results by default.
  • showResumeKey and resumeKey: when more results exist beyond the limit, the server prints a key. Send it back as resumeKey= to continue where you stopped.
  • page, showNumPages and pageSize: an alternative way to walk a very large result set page by page. Ask for showNumPages=true first to see how many there are.

Ready queries for vetting an expired domain

Swap example.com for the domain you are checking.

One capture per year

https://web.archive.org/cdx/search/cdx?url=example.com&output=json&fl=timestamp,statuscode&collapse=timestamp:4

This gives the first capture of each year with its status. Gaps in the years show when the site was offline or not crawled. A long run of redirect codes can mean the domain was pointed somewhere else. The guide to parked periods and archive gaps explains how much each of those matters.

Every archived URL on the domain

https://web.archive.org/cdx/search/cdx?url=example.com&matchType=domain&output=json&fl=original,timestamp&collapse=urlkey&filter=mimetype:text/html

One line per HTML page the archive has ever seen, with its first capture. Read the paths. A recipe blog whose paths turn into product codes, pharmacy terms or numbered spam pages has changed hands, whatever the homepage says.

Spam URL patterns

https://web.archive.org/cdx/search/cdx?url=example.com&matchType=domain&output=json&fl=timestamp,original&collapse=urlkey&filter=original:.*(casino|slot|viagra|cialis|replica|loan).*

Any result here deserves a closer look. The patterns behind expired domain spam history translate well into regex filters. Thousands of new paths in a single year, often in another script, are a common sign of the hack described in Japanese keyword hack history.

Errors and redirects in the final years

https://web.archive.org/cdx/search/cdx?url=example.com&output=json&fl=timestamp,statuscode&from=2022&filter=!statuscode:200

This lists every capture since 2022 that did not return 200. A wall of 3xx near the end usually means the old owner redirected the domain before letting it go; a wall of 4xx or 5xx means the site was simply dying.

A short Python script: archive activity per year

The script below asks for one capture per day (collapse=timestamp:8), then counts how many days per year have at least one successful HTML capture. It is a rough but steady measure of how alive the site was, and it keeps the response small.

import collections
import time
import requests

CDX = "https://web.archive.org/cdx/search/cdx"

def days_per_year(domain):
    params = {
        "url": domain,
        "output": "json",
        "fl": "timestamp",
        "filter": ["statuscode:200", "mimetype:text/html"],
        "collapse": "timestamp:8",
    }
    r = requests.get(CDX, params=params, timeout=60)
    r.raise_for_status()
    rows = r.json()[1:]  # first row holds the field names
    return collections.Counter(row[0][:4] for row in rows)

for domain in ["example.com", "example.org"]:
    print(domain, dict(sorted(days_per_year(domain).items())))
    time.sleep(5)  # be gentle between domains

Say the output for a domain shows steady activity for a decade, then two empty years, then a burst in the final year. That shape tells you exactly which years to open in the browser before you bid.

Rate limits and the 429 error

A 429 status means "Too Many Requests": the server is asking you to slow down. The pages linked above do not publish a fixed request limit, so do not build a script around a number someone quoted on a forum. Build it to be polite instead:

  1. Send one request at a time, with a pause between them.
  2. Make each query narrow with fl, from/to, filter, collapse and limit, so you need fewer and smaller requests.
  3. On a 429 or a 5xx, wait, then retry with a longer pause each time. If the response carries a Retry-After header, respect it.
  4. Cache results. A domain's history up to last year does not change, so there is no reason to fetch it twice.

Firing parallel requests at a free public service is the fastest way to collect more 429s, and it slows the archive down for everyone else who uses it.

What the API cannot tell you

The CDX index tells you when pages were captured, where and with what status. It does not tell you what the pages said. For that you still open the snapshots, or fetch them and read the titles and text. It also inherits the archive's limits: captures are samples, so a short spam period can fall between them, and some sites are excluded entirely.

The API says nothing about backlinks, blacklists, Google's view of the domain or trademarks either. Treat it as one of the six free checks of domain history, not the whole check. If you would rather not build the pipeline yourself, how hunter.domains filters domains describes a scan that reads the archive across a domain's first year, middle years and last two years, and the domain checker runs the same history, spam, blacklist and metrics report on any single domain you enter.

Frequently asked questions

Is the Wayback Machine API free?

Yes. The Availability API and the CDX Server API are public and cost nothing to call. In return, keep your request rate low and cache what you fetch, so you do not burden a shared service.

Does the Wayback Machine API need an API key?

Not for public captures. The CDX server documentation describes a key, sent as a cookie, only for restricted data such as private URLs and certain internal fields. Querying the public archive needs no account and no key.

What is the difference between the Availability API and the CDX API?

The Availability API returns at most one snapshot, the one closest to a date, and is best for a quick "is it archived?" check. The CDX API returns the full list of captures with timestamps, status codes and content types, and can filter and collapse them on the server. For domain history, use CDX.

Why does the Wayback Machine return error 429?

Error 429 means you sent too many requests in a short time. Slow down, send requests one at a time, and retry after a pause that grows with each failure. Narrower queries with collapse, filter and date ranges cut the number of requests you need.

Related articles

Don't miss the next good domain.

Set your rule and leave the rest to us. You hear the moment a matching domain is found.

Get started free