Whether an AI agent can visit your website should be a business decision, not a legal one

Whether an AI agent can visit your website should be a business decision you make yourself, at the edge, based on what it actually costs or earns you. It should not be a question a courtroom settles for you, and this week, in its own way, a courtroom agreed.

The Ninth Circuit Says Building A Browser Is Not Hacking

On August 4, the Ninth Circuit Court of Appeals vacated the preliminary injunction that banned Perplexity's Comet browser, and its shopping Assistant, from accessing Amazon. Amazon's argument was that an AI agent acting for a user without Amazon's permission is "unauthorized access" under the CFAA, the 1986 anti-hacking law. The appeals court said no. In the opinion's words, the CFAA contemplates access by a person, and "however advanced the Assistant currently is, it is a tool, not a person for statutory purposes." It is the user who accesses Amazon, with the Assistant's help, as EFF's writeup lays out.

The opinion is worth reading for how narrow it makes itself, because the narrowness is the useful part. The ruling turns on Comet's architecture: the Assistant reads the page inside the user's own browser, and Perplexity's servers never directly touch Amazon's, so a differently built agent, one that does reach out from the cloud on its own, could come out differently, and the court says so. It also says, flat out, "we do not establish a new legal regime governing agentic AI." And then there's the footnote that reads like this article's thesis in judicial prose: the outcome "does not impair Amazon's ability to regulate access to Amazon.com via private terms of service." Amazon is free to control who visits, the court is saying, it is simply unlikely to do it by invoking a hacking law. One more detail from the background section worth knowing: part of why Amazon couldn't selectively block the Assistant is that Perplexity chose not to send a user-agent string identifying it, which is the mechanism that would have let Amazon block it technically. Even inside the lawsuit, the real fight was about the technical layer.

This is the preliminary round, not the end of the case, Amazon's suit continues on other grounds. But it is the second time in a month a platform reached for a decades-old statute to block a machine visitor and found the statute does not stretch that far. Google's DMCA claim against SerpApi was dismissed in July on the same shape of reasoning: the law they grabbed was written for a different problem, on a different internet.

When I wrote about this case in May, I ended the article on exactly this appeal, and the ruling went the way I thought it should. What still gets me, though, is the setup itself. Courts, full of people who are not experts in any of this, keep making decisions that shape the web's future this much. No matter how they decide. So the good news here is not that the court got it right, it is that the court declined to be the one holding the switch.

Cloudflare's Default Is Where The Decision Lives Now

Cloudflare, which by most measurements sits in front of about a fifth of the web, has been flipping AI-crawler access from open-by-default to blocked-by-default since July 1, and on September 15 that posture extends further. So while the legal route keeps narrowing, the infrastructure route keeps widening, and that is who holds the switch now.

Honestly, I think that is the better place for the decision to live. Cloudflare's product team giving website owners the option to do what they feel is right beats blanket blocking of AI user agents, and it certainly beats a court deciding for you.

The catch is the word "option." A default most owners never look at is not really a decision, it is blanket blocking with an escape hatch. Defaults are bad unless what the default does is what you would have chosen anyway. And this particular default now holds some power. Cloudflare's own documentation says that from September 15, crawlers that do both search and AI training fall under AI-training blocks, and in Cloudflare's classification Googlebot is exactly that kind of mixed-purpose crawler. One r/TechSEO user reported that enabling "AI Training = Block" got their sitemap serving 403s to Googlebot and Bingbot, and that turning it off restored access. Google's John Mueller acknowledged the report and asked them to send details so he could investigate. That is all anyone knows publicly: one report, unverified, possibly user error. But the documented policy underneath it is real. The same bot that puts you in Google also feeds Google's AI, and the toggle does not seem to split the bot in half.

Your Bot Settings Are A Business Decision Now

There are two layers to this, and the first one is the floor. Open your bot management settings before September 15, look at what you are blocking, and make the inherited default a conscious choice. After touching any AI-blocking toggle, spend five minutes confirming Googlebot still gets through, Search Console's URL inspection with a live fetch will tell you. That keeps you out of that Reddit thread.

But the floor is not the point. The real move is to treat machine access as a business decision with real downstream consequences, and to decide based on what those consequences are for your business: what you gain or lose in AI answers, in search, in training exposure, in which agents can still read and cite you. The courts keep declining to make that decision for you. The edge will happily make it for you by default. The only wrong version is the one where nobody made it at all.