When the Wayback Machine says a URL has been excluded, the Internet Archive still may hold captures of the site but has chosen not to show them. A site appears as Wayback Machine excluded for one of two reasons: someone who controlled the site asked the Archive to hide it, or a robots exclusion told the Archive to keep it out of view. For an expired domain you are about to buy, this means the main evidence of its past is gone, and you have to vet it another way.
What "this URL has been excluded from the Wayback Machine" means
The Internet Archive's own help pages say that pages may not be archived "due to robots exclusions" and that "some sites are excluded by direct site owner request" (Wayback Machine general information). The exclusion message is the visible result: instead of a calendar of captures you get a notice that the URL is excluded. Unlike an outage, it affects only that domain.
How a site owner requests an exclusion
The Archive describes the process on its removal help page. An owner emails info@archive.org with:
- the URL or URLs of the material
- the time period they want excluded
- the time period during which they controlled the site
- anything else that explains the request
The Archive reviews each request and says it makes no guarantees about the outcome.
For a buyer, two details matter: a request can name a time period, so an exclusion need not cover the domain's whole life, and it comes from whoever controlled the site at some point. You rarely learn their reason.
Parked domains and robots.txt: how a later owner can hide earlier history
This is the part that catches expired-domain buyers. In a 2017 post the Internet Archive explained that robots.txt files, written mostly for search engine crawlers, did not suit a web archive. It said that when a site became a parked domain and blocked crawlers, that block "has historically also removed the entire domain from view in the Wayback Machine" (Internet Archive blog). In the same post the Archive said it had stopped honoring robots.txt on US government and military sites and was looking to do the same more broadly.
So the person who hid a domain's history may not be the person who built it. Years of clean content can become invisible because of a later parking setup or request. The notice tells you the history is hidden, not who hid it or why.
Excluded, never archived, or just a gap
Three situations look similar at first glance. Tell them apart before you draw conclusions.
| What you see | What it usually means | What it tells you |
|---|---|---|
| Exclusion notice | Captures may exist but are hidden by request or robots exclusion | Someone chose to hide the history |
| A page saying the Wayback Machine has not archived that URL | No captures exist for that exact address | The site may never have been public, or was too small to be crawled |
| A calendar with empty years | Captures exist, with periods missing | A sampling gap or an offline period; see parked periods and archive gaps |
Try the bare domain, the www version, and a few inner URLs from backlink data. An exclusion can be partial, so some pages may still load. Read whatever is visible with the method in the Wayback Machine domain history guide.
What an exclusion tells you, and what it does not
An exclusion is not proof of spam. Owners hide archives for ordinary reasons too: privacy, an old business, or a parking setup that blocked crawlers by default.
But it removes the strongest evidence you have. The archive is where you would spot a casino phase, a script change or a hijacked section. Without it, a clean domain and an abused one can look the same.
Treat a domain with no archive history as an unknown, not as clean and not as dirty. The rest of the vetting has to carry more weight.
Vetting a domain without the archive
Work through these in order. Each one replaces part of what the archive would have shown you.
- Registration record. A WHOIS lookup shows the creation date, registrar, nameservers and status codes. Parking nameservers or a recent registrar change hint at the late-stage history. Since 2018 most owner details are redacted. The guide to WHOIS history covers what older records add.
- Backlinks and anchor texts. The links pointing at the domain describe what it used to be. Anchors about gardening, a local school or a software product tell you the topic. Anchors for casinos, pills or unrelated brands tell you something else. Read the profile with the method in anchor text analysis.
- The referring pages themselves. Open a handful of the pages that link to the domain. They often describe the old site in a sentence ("the official site of…"), and some have their own Wayback captures.
- Google's index. A
site:example.comsearch shows whether pages are indexed. Zero results on an expired domain is common after it has been offline and does not prove a penalty, but leftover indexed URLs with spammy titles are a clear warning. See how to check if a domain is indexed. - Blacklists. Check the Spamhaus DBL and SURBL, and Google Safe Browsing status in Google's Transparency Report. A listing means the domain was used for abuse recently enough to be recorded.
- Current DNS. Parking nameservers or a for-sale page tell you how the domain spent its last period.
The domain checker runs these checks together and reports what it found in the archive, the blacklists, the link profile and the anchor texts, with a verdict. It cannot show you captures the Archive has excluded, and it cannot see Google manual actions; nothing outside Search Console can.
Worked example: discount, skip, or buy
Say you find an expired .org with strong referring domains, and the Wayback Machine returns the exclusion notice for both the bare and www versions.
- WHOIS shows a long registration and parking nameservers on the last record.
- The top anchors are the name of a regional arts festival and phrases about concerts and tickets.
- Several referring pages are local news articles that mention the festival's website.
site:returns nothing, and the domain is not on Spamhaus DBL or SURBL.
Everything that survives points to one topic, and the exclusion probably dates from the parking period. You can buy it, at a lower price than the metrics suggest, because you are accepting a blind spot.
Change one fact: the anchors are mostly casino terms, or the domain is blacklisted. Then skip it. A hidden archive plus a dirty link profile cannot be vetted.
A simple rule:
- Buy at a discount when every other signal is consistent and clean.
- Skip when any other signal shows spam, a topic switch or a blacklist listing.
- Skip or bid low when the other signals are too thin to read, such as very few referring domains and no indexed pages.
If you would rather start from domains whose archive was read across the years, the list of expired domains shows history flags for every listing, including archive gap and parked period, and the clean history list narrows it to domains without them.
Frequently asked questions
Why is a website excluded from the Wayback Machine?
The Internet Archive says sites are excluded either by robots exclusions or by direct request from the site owner. An owner can email the Archive with the URLs, the time period and the period they controlled the site, and the Archive reviews the request. The notice does not say which reason applies.
Can I see an excluded site some other way?
Not through the Wayback Machine itself. The pages that link to the domain, its anchor texts, WHOIS records and any URLs left in Google's index still tell you a lot, and some referring pages describe the old site directly.
Is an excluded expired domain safe to buy?
It can be, but you are buying with less information. If backlinks, anchors, index and blacklists all point to one clean topic, the risk is manageable and the price should reflect the blind spot. If any shows spam, walk away.
Can the new owner of a domain get the exclusion lifted?
That is for the Internet Archive to decide. Its help page says requests are reviewed by its team and the outcome is not guaranteed in advance. Do not price a domain on the assumption that the history will become visible after you buy it.
