Google Built Its Library By Scraping The Web, So It's Fair Game Too

Here is a sentence I did not think I would write: I am on the side of a search-scraper that resells data to AI companies. Not because SerpApi is the good guy. There is no good guy here. But a federal judge in California ruled against Google in its fight with them, and the principle underneath the ruling is the right one, even if it showed up wearing the ugliest costume available. If it is on the open web, a machine is allowed to read it. And that has to include the machine reading Google. You do not get to spend more than two decades building the richest library on earth by crawling everyone else's pages, and then act appalled when someone aims a crawler at yours.

So either we actually mean this open web thing, or we stop saying it.

What the court actually said

On July 20, Chief Judge Yvonne Gonzalez Rogers tossed Google's DMCA claims against SerpApi, a company whose whole business is scraping Google's search results and reselling them through an API, more and more of it to AI companies. Google's theory was that SerpApi broke the law by getting around SearchGuard, its anti-bot system. If you have not heard of SearchGuard, join the club: it is the internal machinery Google uses to spot automated traffic and stop it from scraping search results, the bouncer on the door of Google's results. Google's claim was that beating that bouncer counts as illegal circumvention under the DMCA, the same law that makes it illegal to crack the copy protection on a DVD. The judge was not convinced. Her reasoning: SearchGuard protects Google's ad revenue, not a copyrighted work, and DMCA anti-circumvention is about copyright. A wall around your business model is not a lock on a copyrighted file. She threw the claim out with prejudice wherever no copyrighted content was involved, and gave Google 21 days to come back with a narrow version about Knowledge Panel images. Good luck with that.

Let me be honest about the cast. SerpApi scrapes at industrial scale and resells the results, much of it feeding the exact AI companies everyone is nervous about, so no, not a sympathetic plaintiff. Google is a trillion-dollar company that built itself by crawling the open web and now wants copyright law to stop others crawling it, which is not a sympathetic position either. This is two heavyweights fighting over who gets to package the web, and the rest of us are watching from the cheap seats. It is the worst-person-you-know-makes-a-great-point meme, in legal-docket form.

The people aren't in this fight

When the story gets told as SerpApi versus Google, something goes missing: the open web was supposed to be by the people and for the people. Look at this fight and try to find a person in it. The users whose searches and pages and questions make the web worth scraping in the first place are not a party to anything. Two companies brawl over the spoils, a judge draws a line, and everyone else reads about the outcome later.

But the line she drew is the honest one, and I will take an honest line even out of an ugly fight. If it is on the web, it should be reachable by whatever wants to read it. That is a lovely principle when it is someone else's wall coming down. It stings when you remember who owns the biggest crawler on the planet. Google's entire existence is the open web turned into a product. Running that playbook for more than two decades and then declaring your own results the one crawl-proof corner of the internet is not a legal position, it is nerve. Google is fair game too. That is the deal it signed the day it pointed its first crawler at somebody else's website.

And this is the whole agentic web, not a scraping footnote

"On what terms is an automated visitor allowed onto public web content" is the founding question of the agentic web, not some niche scraping spat, and it does not end with SerpApi. A scraper reselling results, an answer engine reading your pages to cite you, a shopping agent turning up to buy on someone's behalf, an assistant pulling your specs to compare you against a competitor: in the eyes of the law those are one thing, an automated visitor on public content. SerpApi is the ugly early test case. Whatever boundary the courts draw around it is the boundary for all of them.

And this is not a someday problem. A real and growing share of what hits your website already is not human. The terms for how much say you get over those visitors are being written right now, one lawsuit at a time, in fights you have no seat in. Which is exactly why the one decision that is yours matters as much as it does.

Where you actually sit

You are in this too, and you are two things at the same time, and they do not get along.

You are one of the people. Your content gets scraped, resold, and poured into models, and nobody sent you a form to sign. The fight is over your web too, and your seat at the table is the same size as the users': none.

You are also a tiny Google. You would like a say over who takes your content and on what terms, and maybe you would like to get paid for it. This ruling trims the tools for that, because the precedent has nothing to do with Google specifically. An anti-bot wall that guards your revenue instead of a copyrighted work is what most websites are running, and the court said that kind of wall does not buy you DMCA protection.

This is the same frontier the Amazon v. Perplexity case is testing from the opposite end. That one runs on the CFAA and asks whether an AI agent counts as an authorized visitor when it acts on your website. This one runs on the DMCA and asks whether your anti-bot wall counts as copyright protection. Different statutes, same question underneath, and the toolkit for keeping machines out keeps coming up shorter than the people counting on it hoped.

Stop waiting for someone else to decide

The takeaway is not a checkbox to go flip. It is where your head should be.

You do not get to feast on the open web for discovery, every scrap of traffic you were ever found, cited, or ranked for, and then clutch your pearls when that same openness lets a machine you do not care for read you too. It is one web, not two. The consistency runs both ways, whether you like the direction or not.

So make the call yourself. Decide what you want open and what you want closed, per crawler, on purpose, using the AI-crawler controls your host or CDN already gives you, knowing the legal ground under "block them" is still moving and might not hold. Do not outsource that decision to a court refereeing a fight you are not in, and do not outsource it to a plugin that flipped a default you never read. Own it.

Fight for the open web or stop pretending. Whichever you pick, actually pick it. Right now Google and a scraper you have never heard of are making that call for you, and taking it back is the one move in this whole fight that is yours.