231: A Paywall Stopped ChatGPT From Reading The Article. It Confidently Summarised It For Me Anyway.

A Bill Instead of a Page

HTTP 402 Payment Required has existed since 1992. Almost nothing ever used it. Now TollBit sells it to publishers as a way to charge AI companies for accessing journalism. The promise sounds straightforward: crawlers pay, humans read free. But what does the wall actually check?

It reads the user agent string. That's the name a browser or crawler sends to identify itself. Compare that name against a list. If the name matches an AI crawler, send a bill. If the name says Chrome, or says nothing at all, let it through.

What the Wall Actually Checks

I made up a crawler name. It belonged to no company on earth. One second later, the full article loaded. The paywall doesn't detect robots. It detects honesty. Every crawler that identified itself accurately got charged. A fiction walked in free.

TollBit claims partnerships with thousands of websites, but the numbers vary: 6,000 on one page, 8,000 on another, 9,500 on a third. Only one paying customer has ever been named publicly: a news reader app. Zero AI companies appear on the page supposedly aimed at AI companies.

The robots.txt Contradiction

Variety's robots.txt names 25 crawlers. Twenty-four of them are blocked. But three crawlers that the file never mentions get charged anyway. The file says one thing. The server does another. Googlebot walks in free while Penske, Variety's parent company, sues Google in court.

Your robots.txt describes a policy. Your server enforces it. When the two disagree, the server wins. Publishers trusting robots.txt as a defense mechanism should test what their infrastructure actually returns.

Two Assistants, Zero Reading

I asked Claude and ChatGPT to summarize the paywalled article. Both got charged by TollBit. Both answered anyway. Neither had read the page.

They handled it differently, and the difference is the story. Claude said straight away that the page had blocked it, and told me it would search for coverage of the story instead. ChatGPT gave me a confident summary with no mention of a paywall at all. Its source tags credited Google and Metacast. Not one of them credited Variety. Only when I asked whether it had actually read the article did it say "Short answer, yes, but not in the way you might think," and explain that it had reconstructed the story from search results.

Blocking the machine doesn't block the answer. It decides who gets credited when the machine repeats what you wrote. A publisher locked behind a paywall still gets summarized. The summary just attributes the information to whoever else mentioned it.

Being Read or Being Cited

The question isn't whether AI can access your content. Someone, somewhere, has already quoted you, paraphrased you, or mentioned the same facts. The question is whether the AI knows your name when it repeats you.

Ask your chatbot where the facts came from. Press for the actual source. You'll often find a chain of attribution that never reaches the original. That's the real cost of blocking crawlers without ensuring your content circulates with proper credit attached.