The Wayback Machine API is really two public endpoints. The Availability API tells you whether a URL has been archived and returns the capture closest to a date you choose. The CDX Server API returns the raw capture index: every snapshot of a URL, host or whole domain, with its timestamp, status code and content type, and it can filter and collapse the results on the server. The Internet Archive's documentation lists no key for public captures, so both work with a plain HTTP request. For vetting expired domains in bulk, the CDX API is the one you will use most.
This guide covers the parameters that matter for domain history and gives ready queries you can paste into a browser, curl or a script. If you want to read one domain's history by hand, the Wayback Machine domain history guide walks through the calendar and the /* view instead.
The Wayback Machine availability API
The Availability API answers a single question: is there a capture of this URL, and which one is closest to a given time?
https://archive.org/wayback/available?url=example.com
https://archive.org/wayback/available?url=example.com×tamp=20120101
It takes three parameters, according to the official Wayback API page:
url(required): the address to look up.timestamp(optional): the target time inYYYYMMDDhhmmssform, 1 to 14 digits. Without it, you get the most recent capture.callback(optional): wraps the answer for JSONP.
A hit returns an archived_snapshots object with one closest entry. That entry holds available, the Wayback url of the snapshot, its timestamp and its HTTP status. A miss returns an empty object: {"archived_snapshots":{}}.
That is enough to sort a long list of candidates into "has history" and "has none". It is not enough to read history: one snapshot tells you nothing about the years between. An empty answer can also mean the site was excluded from the archive, which is a separate case covered in vetting a domain the Wayback Machine has excluded.
The Wayback Machine CDX API
The CDX server returns the index the Wayback Machine itself uses to look up captures. The base request is:
https://web.archive.org/cdx/search/cdx?url=example.com&output=json&limit=5
Each line describes one capture. The default fields are:
| Field | What it holds |
|---|---|
urlkey |
The URL in sortable (SURT) form |
timestamp |
When it was captured, YYYYMMDDhhmmss |
original |
The URL as it was crawled |
mimetype |
Content type, such as text/html |
statuscode |
HTTP status the crawler received |
digest |
A hash of the content |
length |
Size of the stored record |
With output=json the answer is a JSON array whose first row lists the field names. Use fl to return only the fields you need, for example fl=timestamp,original,statuscode.
The CDX parameters that matter for domain history
The CDX server documentation, linked from the Wayback API page above, lists more options than you need. These are the ones that do the work:
matchType:exact(the default),prefix,hostordomain. Usehostfor every URL onexample.com, anddomainto include subdomains as well. Writingurl=example.com/*is the same asprefix.fromandto: an inclusive date range, 1 to 14 digits.from=2015&to=2018limits the answer to those years.filter:filter=field:regex, with!in front to invert it. You can repeat it.filter=statuscode:200keeps successful pages;filter=!mimetype:image.*drops images.collapse: removes adjacent duplicates on a field, or on its first N characters.collapse=timestamp:4keeps the first capture of each year;collapse=urlkeykeeps one line per URL;collapse=digestskips captures whose content did not change.limit:limit=Nreturns the first N results,limit=-Nthe last N. The server caps a single query at 150,000 results by default.showResumeKeyandresumeKey: when more results exist beyond the limit, the server prints a key. Send it back asresumeKey=to continue where you stopped.page,showNumPagesandpageSize: an alternative way to walk a very large result set page by page. Ask forshowNumPages=truefirst to see how many there are.
Ready queries for vetting an expired domain
Swap example.com for the domain you are checking.
One capture per year
https://web.archive.org/cdx/search/cdx?url=example.com&output=json&fl=timestamp,statuscode&collapse=timestamp:4
This gives the first capture of each year with its status. Gaps in the years show when the site was offline or not crawled. A long run of redirect codes can mean the domain was pointed somewhere else. The guide to parked periods and archive gaps explains how much each of those matters.
Every archived URL on the domain
https://web.archive.org/cdx/search/cdx?url=example.com&matchType=domain&output=json&fl=original,timestamp&collapse=urlkey&filter=mimetype:text/html
One line per HTML page the archive has ever seen, with its first capture. Read the paths. A recipe blog whose paths turn into product codes, pharmacy terms or numbered spam pages has changed hands, whatever the homepage says.
Spam URL patterns
https://web.archive.org/cdx/search/cdx?url=example.com&matchType=domain&output=json&fl=timestamp,original&collapse=urlkey&filter=original:.*(casino|slot|viagra|cialis|replica|loan).*
Any result here deserves a closer look. The patterns behind expired domain spam history translate well into regex filters. Thousands of new paths in a single year, often in another script, are a common sign of the hack described in Japanese keyword hack history.
Errors and redirects in the final years
https://web.archive.org/cdx/search/cdx?url=example.com&output=json&fl=timestamp,statuscode&from=2022&filter=!statuscode:200
This lists every capture since 2022 that did not return 200. A wall of 3xx near the end usually means the old owner redirected the domain before letting it go; a wall of 4xx or 5xx means the site was simply dying.
A short Python script: archive activity per year
The script below asks for one capture per day (collapse=timestamp:8), then counts how many days per year have at least one successful HTML capture. It is a rough but steady measure of how alive the site was, and it keeps the response small.
import collections
import time
import requests
CDX = "https://web.archive.org/cdx/search/cdx"
def days_per_year(domain):
params = {
"url": domain,
"output": "json",
"fl": "timestamp",
"filter": ["statuscode:200", "mimetype:text/html"],
"collapse": "timestamp:8",
}
r = requests.get(CDX, params=params, timeout=60)
r.raise_for_status()
rows = r.json()[1:] # first row holds the field names
return collections.Counter(row[0][:4] for row in rows)
for domain in ["example.com", "example.org"]:
print(domain, dict(sorted(days_per_year(domain).items())))
time.sleep(5) # be gentle between domains
Say the output for a domain shows steady activity for a decade, then two empty years, then a burst in the final year. That shape tells you exactly which years to open in the browser before you bid.
Rate limits and the 429 error
A 429 status means "Too Many Requests": the server is asking you to slow down. The pages linked above do not publish a fixed request limit, so do not build a script around a number someone quoted on a forum. Build it to be polite instead:
- Send one request at a time, with a pause between them.
- Make each query narrow with
fl,from/to,filter,collapseandlimit, so you need fewer and smaller requests. - On a 429 or a 5xx, wait, then retry with a longer pause each time. If the response carries a
Retry-Afterheader, respect it. - Cache results. A domain's history up to last year does not change, so there is no reason to fetch it twice.
Firing parallel requests at a free public service is the fastest way to collect more 429s, and it slows the archive down for everyone else who uses it.
What the API cannot tell you
The CDX index tells you when pages were captured, where and with what status. It does not tell you what the pages said. For that you still open the snapshots, or fetch them and read the titles and text. It also inherits the archive's limits: captures are samples, so a short spam period can fall between them, and some sites are excluded entirely.
The API says nothing about backlinks, blacklists, Google's view of the domain or trademarks either. Treat it as one of the six free checks of domain history, not the whole check. If you would rather not build the pipeline yourself, how hunter.domains filters domains describes a scan that reads the archive across a domain's first year, middle years and last two years, and the domain checker runs the same history, spam, blacklist and metrics report on any single domain you enter.
Frequently asked questions
Is the Wayback Machine API free?
Yes. The Availability API and the CDX Server API are public and cost nothing to call. In return, keep your request rate low and cache what you fetch, so you do not burden a shared service.
Does the Wayback Machine API need an API key?
Not for public captures. The CDX server documentation describes a key, sent as a cookie, only for restricted data such as private URLs and certain internal fields. Querying the public archive needs no account and no key.
What is the difference between the Availability API and the CDX API?
The Availability API returns at most one snapshot, the one closest to a date, and is best for a quick "is it archived?" check. The CDX API returns the full list of captures with timestamps, status codes and content types, and can filter and collapse them on the server. For domain history, use CDX.
Why does the Wayback Machine return error 429?
Error 429 means you sent too many requests in a short time. Slow down, send requests one at a time, and retry after a pause that grows with each failure. Narrower queries with collapse, filter and date ranges cut the number of requests you need.