What the file can say
A single plain-text file at the root of a host, fetched before almost anything else, and one of the few things on the web that every crawler reads on every visit. Its grammar is small: groups of rules, each addressed to one or more user agents, each listing paths that are disallowed or explicitly allowed, plus a small number of extensions.
The directives
| Directive | Effect |
|---|---|
| User-agent | Opens a group. A crawler obeys the most specific group naming it, and * only if no specific group exists.This is the rule most often got wrong: a crawler with its own group ignores the wildcard group entirely. |
| Disallow | A path prefix the crawler is asked not to fetch. An empty value means nothing is disallowed. |
| Allow | An exception carved out of a Disallow. The longer, more specific rule wins regardless of order. |
| Sitemap | An absolute URL to a sitemap. Global rather than per-group, and read by crawlers that ignore everything else in the file. |
| Crawl-delay | Requested seconds between fetches. Honoured by some crawlers, ignored by the largest. |
Everything in that table is a request. Nothing in the protocol gives a server any means of enforcement, and nothing in it is checked by anybody.
What it cannot do
The file is public, and it is the first place an unfriendly reader looks. A Disallow
line naming a private directory has advertised that directory to everyone while stopping only the
crawlers that were going to behave anyway. Any path listed in robots.txt should be assumed to be
known.
It also cannot express what it is most often used to attempt. It cannot require attribution, forbid commercial reuse, or make a crawler identify itself honestly. A crawler that ignores the file entirely is doing nothing the protocol can detect, and a crawler that presents a false user-agent string is indistinguishable from a browser. Access control is a server function; robots.txt is a notice board.
Disallow is not noindex
This is the single most consequential misunderstanding in the whole subject, so it is worth stating
slowly. Disallow governs fetching. Indexing is a separate decision, and a search engine
may index a URL it has never fetched, on the strength of links pointing at it from elsewhere. The
result is a listing with a URL and no description, which is exactly the outcome the site was trying
to prevent.
Worse, the two mechanisms conflict. A noindex instruction is delivered in the page's
own markup or in an HTTP header, both of which require the crawler to fetch the page. Disallowing the
path prevents that fetch, so the noindex is never seen and the URL stays indexed indefinitely. The
correct sequence for removing something from an index is to allow the fetch, serve noindex, wait for
it to be honoured, and only then consider disallowing.
The rule in one line
Use Disallow to save crawl budget on pages you do not care about. Use
noindex to keep pages out of results. Using the first to attempt the second
guarantees the opposite.
Crawl-delay and real rate limiting
Crawl-delay was never part of the original protocol and support has always been patchy. The largest crawler ignores it outright, managing its own rate adaptively and exposing controls elsewhere. Several other major crawlers do honour it. Some ignore it while honouring their own proprietary equivalent.
Which leaves the server as the only place load is actually controlled. Rate limiting by address or
by user agent, returning 429 with a Retry-After header, and serving
503 under genuine pressure are mechanisms that work on every client because they do not
depend on cooperation. A well-behaved crawler reads those responses and backs off; a badly behaved
one is at least slowed down. This is the same distinction as before: the file requests, the server
enforces.
Where scraping stops being allowed
robots.txt is the least of the constraints on automated collection, and reading it as the whole boundary is a mistake. Four other layers apply and each is independent of the file.
- Terms of service. The operative restriction on nearly every large platform. Bulk automated querying of a search service is prohibited by its terms regardless of what any file says, and always has been.
- Computer-misuse law. Jurisdiction-dependent and unevenly applied to public data, but circumventing a technical access control moves the question from contract into criminal law in many places.
- Copyright and database rights. Facts are generally not protected; a substantial extraction of a structured collection frequently is, particularly in Europe where database right exists independently of copyright.
- Data protection. If the collected material includes personal data, the regulation applies to the collector whether or not the material was publicly visible. Public does not mean unregulated.
Why the aggregators went away
Which brings this section to a close on the honest note it should end on. A large family of web services in the 2000s and early 2010s worked by querying search engines and third-party data services in bulk and republishing the results as pages. Directory sites, rank checkers, "everything about any domain" profilers, keyword aggregators: all of them ran on the same mechanism.
That mechanism no longer exists in usable form. The querying it depended on was always against the terms of the services being queried, and platforms acquired both the technical means and the commercial motive to stop it. The free data sources those services blended have closed: Yahoo Site Explorer in 2011, Compete in 2016, Alexa in 2022, and the reputation services in between; the full list runs to nine. And regulation on the reuse of collected data tightened throughout.
What is left is what was always the more solid ground: the sources a system is obliged to publish. A whois record, an RDAP response and a domain's DNS records are published under obligation, are designed to be queried, and will answer in ten years. That is a considerably smaller amount of information than the aggregators promised, and all of it is still true.
