The Web Started Checking ID, And 119 Of 153 AI Crawlers Cannot Prove Who They Are

I went looking at my own logs and found that the single largest AI crawler on my website arrived about 1,500 times in a day under Common Crawl's name and spent the visit asking for SSH keys. A name in a log tells you what was typed into a header, and nothing about who sent the request.

In the last week of August, checking that name became automatic. Most crawlers have nothing to present to the check.

On August 26, IPScanner's public directory of AI crawlers listed 153 crawler tokens, the names automated programs use to announce themselves when they arrive at a website. Thirty-four of them publish something a website owner could check them against. The other 119 publish nothing at all, and nine are documented as not respecting robots.txt, among them Bytespider, DeepSeekBot and facebookexternalhit.

Two days later Cloudflare announced that when a bot operator applies to be recognized as legitimate, the claim is now verified automatically instead of by a person reading it.

Nobody coordinated those two. They happened days apart, and between them a crawler's name stopped being a label and became a claim you can either back up or not.

A user agent string is a name anyone can type

When an automated program visits your website it sends a line of text called a user agent string. That is where names like GPTBot, ClaudeBot and CCBot come from. The string is text and nothing else. No key, no signature, nothing to check it against. Anyone running any script can send any name they like, and your server writes it into the log exactly as received.

A name means something only if the operator behind it published something you can check it against. There are three ways that happens.

A crawler's operator can publish the IP addresses it crawls from, usually as a file you fetch and match against. An operator can instead commit to a reverse DNS convention, where you take the visitor's IP, ask what hostname it belongs to, confirm that hostname sits on the operator's domain, then look the hostname up again and confirm it returns the same IP. Or an operator can sign every request with a cryptographic key for you to check, which is Web Bot Auth, a draft internet standard that replaces the honor system with math. Those three are the things being checked.

I catalogued which crawlers support which in the AI user agent landscape reference. IPScanner counted something easier than all three: whether an operator publishes IP ranges, a reverse DNS convention, or the network it runs on. No signature required. Thirty-four of 153 tokens clear that lower bar. The other 119 clear none of it, which is 78 percent of every crawler with a name.

IPScanner is run by Minori LLC in Albuquerque, New Mexico, which also sells a paid verification service. That is worth knowing, since the count makes the case for the product. The method is stated and checkable: base facts from the ai.robots.txt community list, then user agent strings, IP range files and reverse DNS conventions checked against the operators' own published sources. The entries can be confirmed against vendor documentation one at a time, which is what makes the total usable rather than a claim you have to take on faith.

Cloudflare automated bot verification after its own queue grew about seven times

Cloudflare's August 28 post describes what happens now when an operator submits a bot: "We fetch your IP list, confirm your reverse DNS, or validate your Web Bot Auth signature automatically, instead of a person doing it by hand." Submissions that pass proceed immediately. Ones that fail go to a human with the problem already identified. Operators can also see where a submission stands, edit it, cancel it, and declare what the bot does, how it uses content, and who runs it.

Cloudflare gives the reason in the same post: the number of new bots submitted each year has grown about seven times in volume since 2023.

That figure is Cloudflare's own, about its own program. It is the company's account, not an independent measurement. Taken at face value it makes the automation arithmetic rather than principle: the queue got long enough that a person reading each application stopped being possible.

Something like seven times as many bot operators want to be recognized as legitimate machine visitors as wanted it three years ago. Recognition acquired value. That is the larger change here, and the workflow update is only what the volume forced out of it.

The 119 unverifiable crawlers are two populations you cannot tell apart

Many of the 119 unverifiable crawlers never published anything because being uncheckable carried no cost until an automated gate existed. A crawler run by three people had no incentive to maintain an IP range file and nobody to maintain it for.

Others benefit from being uncheckable. If a website owner cannot confirm you are who you say you are, they cannot write a rule that treats you differently from anyone else sending the same name. The nine documented robots.txt ignorers are the visible edge of that. No reason to assume the visible edge is all of it.

From outside, the two look identical. Same log line, same name, no field anywhere that separates a small team who never got around to it from an operator who would rather you could not tell. A website owner cannot tell neglect from evasion, and verification arriving does not change that. Automated verification sorts the 34 crawlers that publish something. About the other 119 it tells you nothing you did not already know, except that they are now on the wrong side of a gate.

A crawler's name is not identification, so sort your logs into verified, failed and unidentified

Most analytics and most bot dashboards report crawler traffic by name, as though the name were a fact. It is a claim, and the fix costs nothing. Rebuild the view with three buckets instead of one list.

1. Verified. The visitor sent a name, that operator publishes ranges or a reverse DNS convention, you checked, it matched.
2. Failed. The name was checkable, you checked, it did not match. Somebody is wearing a name that is not theirs.
3. Unidentified. Nothing is published behind the name at all. Not accused of anything, not cleared of anything either.

You do not need to run signature verification yourself. Checking the handful of crawlers you actually care about against published ranges is a scripted afternoon.

The reason to build bucket two is that I have already been in it and could not get out. Common Crawl is one of the operators that makes itself checkable. It runs CCBot on dedicated IP ranges with reverse DNS, publishes those ranges as JSON, and resolves to hostnames ending in crawl.commoncrawl.org, so the test exists and anyone with the right data can settle it in a minute. I could not run it, because per-request IP addresses are not something my plan exposes. Cloudflare's dashboard listed that traffic under Common Crawl's name, next to Googlebot, and counted it toward my AI crawler total.

That is what the gap between 34 and 119 looks like from inside one website, and it is worse than the count suggests. The operator had published the test. The name was checkable. The check was still out of reach.

A signature proves an operator and that is the whole of what it proves. It says nothing about whether you want the visit.

But the direction is now set. The web is acquiring an identity layer whose resting state is refusal. Prove who you are or the door stays shut, decided at the edge, on behalf of website owners who will mostly never open the settings.

More than three in four crawlers cannot prove anything today. That number will move, because a reason to move it finally exists, and your own three buckets are where you will watch it move. The question worth watching is not whether verification wins. It is which of the 119 turn out to have been unable to prove themselves, and which turn out to have preferred it.