For a period this catalog kept a page per domain: type in an address and get back a blended
summary of what could be known about it. General registration information at the top, then an
estimated traffic position, a safety and reputation rating, a list of apparent competitors
and similar sites, and a count of inbound links. It looked like one dataset. It was five.
What each block actually came from
Every block on a profile page was somebody else's data, fetched and reformatted. That is
worth spelling out, because it explains precisely how the whole section became impossible to
maintain rather than merely dated.
Why the whole section went rather than most of it
A page assembled from five feeds has the durability of its shortest-lived input, and this one
lost four. The remaining block, registration data, is genuinely useful and is the only part
worth keeping, which makes a profile page the wrong container for it: a page that is
four-fifths empty is worse than a page about the one thing it can still say.
There is a second reason, and it is the more decisive one. The mechanism these pages ran on
was bulk querying of third-party services and republishing the results. That was always
against the terms of the services being queried, and the combination of platform enforcement,
the closure of the free feeds and tightened regulation on data reuse has made it impractical
as well as impermissible. The page on
robots.txt and crawl limits sets out where
that boundary now sits.
Where those questions are answered now
Split up, and better for it. The registration questions are answered directly from the
registry, which is authoritative and obliged to publish. The resolution questions are
answered from DNS. The measurement questions are answered by commercial vendors whose methods
and blind spots can at least be stated. Nothing is blended, so nothing is presented with more
confidence than its source deserves.
That is what the website data section is: eleven pages, one per
question, each one about a single source rather than a summary of five. The three
browser decoders do the part a profile page did usefully,
which is naming the fields in a record you already have, without querying anything.
The general lesson
An aggregate is only as durable as its least durable input, and it hides which input that is.
A source published under obligation, such as a registry record, outlives a source published as
a product. The report on closed measurement
tools is the full version of that argument, with dates.