231: A Paywall Stopped ChatGPT From Reading The Article. It Confidently Summarised It For Me Anyway.

[00:00:00] A reporter at Variety magazine wrote a story a few months ago, and then Variety published it. Kevin Roose and Casey Newton, the two people who host The New York Times tech podcast Hard Fork, had signed with a Hollywood talent agency on their way out of The Times. Them going independent was already public.

[00:00:21] Roose had posted about that on his X account about a week before Variety published the story. The agency deal was the new part. Variety puts its name on it and then publishes a story, and then a machine goes to read it, and the website doesn't hide the story, doesn't show the story, doesn't ask the machine to log in.

[00:00:46] It just sends the bill. There's not a please subscribe to read we, the humans get. A bill with a number on it that just had been sitting in the web standards since 1992, waiting

[00:01:02] This is NoHacks with me, Slobodan Manić. Nobody asked any of us if we wanted the internet changed, so I check what AI is actually doing to it. The good, the bad, and the pointless

[00:01:15] I only found out any of this happened because a machine told me. I had an assistant reading some things for me, doing some research, and halfway through, unprompted, it mentioned that one website would not let it in.

[00:01:29] I never sent it to Variety website. It just followed the link it found. So I went and looked, this is what I assumed was going on. I assumed a website that charges robots either gets paid or just keeps them out. Those felt like the only two outcomes available.

[00:01:45] You put up a wall and either money comes in or nobody goes through. Not what happened. What I found was stranger than a wall that doesn't work, that's broken. Let me show you what , really happened.

[00:01:59] But first of all, I wanna thank you for listening. And when I'm done covering the story for this episode, just stick around. I wanna share some thoughts about the podcast and then just where the internet is headed. So when I'm done with the Variety story, with the pay me to read the story, stick around

[00:02:19] So the website I said was Variety.

[00:02:23] It's the magazine that writes about Hollywood and that claims to have invented the word sitcom, I think decades ago. The message the assistant got that I sent was word for word, I'm reading this from my screen. "You're not authorized to access this content without a valid TollBit token."

[00:02:39] There was no price. There was nothing. The number that the website gave the assistant was four zero two. Now, every time anything asks a web server for a page, the server answers with a three-digit code. That's how it starts. It can be either, "Okay, you'll get this," or, "It's not here, look elsewhere." That's a three oh one, three oh two redirects.

[00:03:06] The four XX, so the, the codes that start with four means that, uh, they, they mean that it's your fault that you're not getting this, so maybe it's a not found, you're at the wrong address. You've seen those, I'm sure. Five means the server messed up. The number was four zero two That means payment required.

[00:03:25] That's a status code that has been a part of web standards since nineteen ninety-two, and almost nothing has ever used it. Nobody was using it to charge an agent, an AI assistant, or anyone for reading a web page.

[00:03:40] Now somebody is. TollBit is a company that sells websites a paywall for robots. And if you look at their website, it's either on six thousand websites or eight thousand or nine and a half thousand, depending on which page you read. And, uh, some of the websites listed are Forbes, Newsweek, USA Today, the Associated Press.

[00:04:02] The first thing I wanted to know was what this wall is actually looking at. So hear me out. When anything asks a server for a page, it sends a short message that says, "This path, this URL, give me whatever you have for that." Another line of that message that you get sends a user agent. It sends an identity of whoever's asking.

[00:04:30] So Chrome will say, "I'm Chrome." Google's crawler will just say, "I'm Googlebot." That line is called a user agent, and it's a text string. You write it yourself. You can say whatever you want. It's a fake ID. It's a-- it's McLovin from Superbad. That's what it is. That's what user agent, that, that's what this thing is. The reason it works or worked until we got these new crawlers on the internet is exactly the reason McLovin got away with buying booze with fake ID. Nobody bothered to check. Nobody really looked into it, not the cops, not the lady selling him the booze.

[00:05:06] I've seen Superbad way too many times, and I'll see it again. So I tried. I wrote a few things in it. I tried the same page over and over, the page from Variety, and I was just changing that one simple line. Saying I'm a browser, I got the page. Giving it no name at all for user agen-agent, I got the page.

[00:05:29] Saying I'm Googlebot, I got the whole page. Saying I'm OpenAI's bot or Anthropic's, nope, straight to the payment page or toll booth. Now, the McLovin part is funny. This is where I tried to make my own McLovin, my own fake ID, I gave the robot a clearly fake name that belongs to nobody on Earth and asked if I can have the page.

[00:05:56] I got the page. So that's the first thing that I want you to know from this story, and the wall that they put up, it's not checking whether you're a robot or not. It's checking whether the name you gave it is on a list. That's weird. If you say who you are and you're honest about it, you get charged. Make something up and you walk straight into the website, you get the pages.

[00:06:24] And let me be clear, I-- it's not just Variety. I tried the same name at five more magazines, four owned by the same parent company and one random one, and I got pretty much the same results every time. So this is a product. This is not Variety's preference. And that made me want to know what these websites say out loud about who they let in.

[00:06:46] There's a file that almost every website has. It's called robots.txt. It's been a standard since nineteen ninety-four. You go to the root of any domain slash robots.txt, there's a plain text file at the same address on every website where the website writes down which robots it doesn't want to see. It can say, "You, bot, are not allowed on these pages.

[00:07:10] You're not allowed on these pages." Great concept. But it's a note, it's not a lock, and nothing makes a robot obey it. Variety's file names twenty-plus robots, so I checked it line by line against the robots its server actually charged or tried to charge. And if you don't match See, Variety names all three bots OpenAI publishes, and then one of Anthropic's three and one of Perplexity's two, and then the three it missed got charged anyway.

[00:07:47] By Variety's own file, the three bots that were blocked Were supposed to be allowed, but the server tried to build them regardless. Most people who run a website treat robots.txt as their policy. It's not.

[00:08:08] It's the server that sets the policy. And Variety's two things, what robots.txt says and what the server says, they contradict each other. And then there's a visitor, the big G elephant in the room that nobody charges.

[00:08:23] In the box at the top of Google's result, I'm not talking about old school Google results, I'm talking about the new, the AI overview. It's, it's built from what Googlebot fetches And the crazy thing is, while not blocking Google and Googlebot, Variety's owner is suing Google. So Penske Media, the parent company, filed last September, and the claim is that by early twenty twenty-five, the box was appearing on about a fifth of the searches that returned their websites.

[00:08:54] Their organic affiliate revenue was down by more than a third, and the only way out would be leaving Google Search altogether, which would be, in their own words, devastating That's the bad part. That's not my opinion. The judge in this trial of Penske versus Google, uh, is the pers- the same person that ruled, uh, Google an il- illegal monopolist in search a few years ago

[00:09:18] This is what he wrote. "Publishers cannot opt out of Google's use of their content for display in AI overviews." And the only way to try was to stop being crawled at all. And the judge called that, quote, "Not a tenable choice And you can see exactly this in Variety's robots.txt. There's a line in it telling Google Extended to stay out.

[00:09:48] Google Extended is Google's AI opt-out. It has never covered the answers at the top of the search results, and Googlebot is not on that list. So Variety is doing things that look like opting out of Google's AI, but that's not really what's really happening. And that leaves me with a question

[00:10:08] Is anybody in the world paying for this? TollBit, the company, they have a page aimed at AI companies, the ones that are, you know, supposed to be doing the paying. And the whole page is just a list of publishers. It doesn't name a single AI company. In two years, from what I could find, the only paying customer they ever got and they ever named is a news reader app

[00:10:35] There's a company that if you're in tech, you probably heard of, Stack Overflow. They described what happened when they switched the paywall on. Stack Overflow is the programmer's question and answer website. This happened earlier this year. I don't think Stack Overflow is doing really well. That's what AI is doing to the internet, among other things.

[00:10:56] And they've been adding bad crawlers to block list by hand until it got crazy, so they just started answering with a 402 instead. Their website reliability engi-engineer said that some of the traffic from these bots that used to just get a block of 403, not allowed, they just stopped sending traffic our way.

[00:11:19] So they left rather than pay.

[00:11:22] Now, an executive from TollBit, also earlier this year, said that nearly a fifth of its publishers have made money from the paywall That could mean anything. That could mean $10. That could mean a thousand euros. That could mean a billion whatevers. That's the vendor talking about his product. There are no names, there are no totals.

[00:11:49] This is, this is not really reliable information. Just a little bit after that, the, the co-founder from the company, from TollBit, in an interview said it's not going to make their year this year in terms of revenue that's flowing through the pipes. So should I say negligible here? I, I always wanted to use that word in a sentence.

[00:12:13] English is not my first language Let's go back to my story with the two assistants, with, with, with my AI assistants. I, I tried two of them. I asked two assistants, chatbots, LLMs, to read the Variety article and tell me what it's about. This was not the research that caught this. This was me testing how this actually works.

[00:12:35] Both of them got told that they need to pay. Claude told me that it's blocked, and before I asked any details, for any details, it, it-- this is what it put on the page in the response. "The page blocked direct access, so let me search for coverage of the story instead." Two things there. It said it had been refused, and then it said it was going to look around instead.

[00:13:08] You know, like when you wanna watch a movie, but you don't wanna pay for a movie, you look around and you find it And the copy is not great. It's not identical, but, you know, you didn't pay. I knew the answer it was going to build. It, it was of other people's coverage rather than news And then I asked ChatGPT.

[00:13:33] And Sam Altman, God bless his soul, has built a crazy product. I gave ChatGPT the link and asked, "What does this article say?" And I, and I know that's not how people use ChatGPT, but I wanted to see if it can fetch it, and It came back

[00:13:55] With part of the response was, "This is a clear, no-fluff summary of that Variety article." It was confident, didn't say anything about not being able to reach the page or read the page. Nothing, nothing about a paywall. And then I look at the source tags be- in, in that answer, and one said Google, one said Metacast, not a single one said Variety.

[00:14:22] So I asked a follow-up question: "Were you able to read the article I gave you?"

[00:14:30] The answer was this. "Short answer, yes, but not in the way you might think." And then it explained what happened, and then it said something like, "I wasn't able to directly open and read that exact page like a human browsing a paywalled article. Instead, I used web search to retrieve reliable excerpts and summaries of that specific Variety story, then reconstructed the key point, points from those sources."

[00:14:58] I wanted to be fair in this investigation.

[00:15:01] This is an honest representation of what happened. He didn't pretend. He told me when I asked. But what it-- what actually happened there? Variety paid someone to write that, to find out details ab- of that deal, and then write the story. And then he spent money to stop the machines from reading the story.

[00:15:22] And then a few seconds later, after I asked, I have two machines telling me the story anyway, and then they credit a search engine and a podcast directory. The, the paywall, the tollbooth, the whole thing, it's, it's a good concept, but it doesn't stop the story from spreading. It couldn't 'cause a, a deal like this, that's a fact about the world.

[00:15:49] Once it's out, it's out. And then three other outlets ran the same news that day about Kevin Roose and his co-host signing that new deal. I, I'm blanking on the name. Kevin Roose and Casey Newton, of course. So both ChatGPT and Claude got their response el-elsewhere and pretended that they knew what they were talking about

[00:16:15] What the paywall decided was who is going to get named when a machine repeats the story. And look, this is... If you run a website, you should think about this because you're not choosing between being read and being paid. You're choosing between being read and being cited.

[00:16:35] So, you know, blocking, blocking these machines It could lead to a lot of downstream problems. Number one, somebody else is going to give a version of the story, and what if it doesn't agree with what you want to say? Now, there's a good part.

[00:16:52] There's a good part, and I, I think this is real. The, the money is starting to move from charging per visit towards paying for access, just paying for use. Microsoft has a pilot of a marketplace that pays publishers when AI uses their work, their work, not, not just once they read it, but every time they use it.

[00:17:12] It's built with Associated Press, Condé Nast, and their CEO had some very harsh words about what AI search is doing to their business. It's a pilot.

[00:17:21] But

[00:17:22] it's a better direction than just paying for a page at the time you read it, because you read it and then, then you're done. Why pay for it again? You pay five cents to read everything on the internet you care about and, and, like, that's it. Pay every time you use the information. This is what AI should do and AI companies should do.

[00:17:42] So yeah, that reporter at Variety, they wrote a story, Variety published a story, and I have these two, two machines just in seconds repeat that story or, or a version of that story, and neither one of those had ever read the original version. I, I'm not sure I care about whether the bots pay

[00:18:09] But I want them to disclose what happened

[00:18:13] I couldn't get Claude and ChatGPT to get in and read the page, and I couldn't when I was trying to fetch from my computer So the right thing to do in that response is, "Nope, this doesn't work. Let's try this." And the chatbots don't always do that. One of them did when I tried. The other one, well, had to be pushed.

[00:18:34] So ask your chatbots where they're getting the facts. That's what I'm saying. Ask your chatbots and then follow the trail. If you were responsible enough to check the links on Google when you search for something and do your research, that's not over. It's worse. Just ask

[00:18:57] That's the episode for this week. If you want this in writing with every link and every of the checks I ran, I, I also have a newsletter. I send it every Wednesday, not today because I was out all day. Go to nohacks.co/subscribe. This is NoHacks. Thank you for listening

[00:19:17] now let's talk about the future of, of the internet and the podcast. I try not to be too serious, but this is, this is just me riffing at 10:00 PM on a Wednesday

[00:19:33] Uh, I took a month off this summer. I, I talked about this in a podcast. I, I was away from all the madness in, in the dig- digital world. Well, the world in general. And, uh, it felt good Coming back, I was fresh. My sleep scores were 95 plus every single day. I wasn't looking at screens. I, I'm guessing there's a causation.

[00:19:59] Um, and yeah, when I came back, it, it wasn't... A lot of the things I was doing before my break, it wasn't easy to get back to them and, and to be passionate about doing them again, because a lot of the stuff online is pointless. And I spent the first six months of the year just trying to cover every relevant story about the agentic web and all that tech and, and some of you who are listening have pushed on that because they don't agree that agents will change the internet.

[00:20:36] I, I think they will, but not in a way that the AI bros would like us to think. But anyway, the internet is what gave me everything. Like I, I, I grew up offline, but I got online very early in my life. Everything-- Not everything, most things I have in life are because I had an internet connection. I mean, the podcast and, um, my, my, my first online job and all that, like my web dev career, none of that would happen.

[00:21:07] I would not move country two times if it wasn't for internet. I would not be living where I live now. I don't, I don't like what's, what's happening to it. I don't like what AI is doing. AI is like, in these LLM companies, Yeah, the best analogy I have is they're, they're like the aliens from Independence Day

[00:21:26] Big tech, AI, private equity, whatever you want. Like that, that's what they are. I, I think Independence Day, the original one, where Will Smith practices for the Chris Rock slap 30 years before. I think that that's a movie about private equity, about big tech, about AI companies. All-- I, I don't like what they're doing, is what I'm saying.

[00:21:49] However, I do think that technology is genuinely good for some things. Absolutely not for everything. So what I wanna talk about on the podcast in the future, and that I really haven't, is what the fuck are you doing to the internet? Why is this change now necessary? We had the internet for 30 years, 30 plus years.

[00:22:12] It worked fine, and then these companies show up. And now everything has to be different. There was never a more non-consensual tech, and this is not my phrase, I, I heard this on a podcast I just don't like what's happening. I think If you think it's not good, you have to speak up, and I, I don't think I have been speaking up enough in the past

[00:22:36] so yeah, this podcast is going to be more about... I just try to focus on the positive things of this while ignoring-- not ignoring, but refusing to talk about the negatives. I think it's time. I think, I think I need to talk about that. If you stuck around till the end of the episode... This is not even the end.

[00:22:58] This is after the end of the episode. I, I, I appreciate you. I really, really, really do. I wanna hear your feedback. Anything that's bothering you, anything about the internet, anything at all, [email protected]. That's it for this one. Talk to you next week