What Courts Say About robots.txt, Terms of Service, and Blocking AI Bots

"Keep the AI companies off our website" sounds like one decision. It is not, because "AI crawlers" is not one kind of visitor. The label covers machines doing entirely different things to you and for you, catalogued one by one in the AI user-agent landscape, and treating them as one thing, with one switch, could make you pay in a specific currency: humans who discover you through AI.

Block everything wearing the label and you have also blocked the fetchers that put your pages in front of people asking assistants where to buy what you sell.

So what to do with each crawler is a business decision and needs to be made per crawler. The question this page answers is the narrower one that comes next: whichever way you decide, what does the law actually give you to enforce it? The honest answer is not much yet. Plan as if that will not change, and keep an eye in case it does. This page is my attempt to keep-an-eye on it.

It catalogs the seven things a website owner can deploy against automated access, and what courts have actually said each one is worth. Every entry names the case, quotes the language, and says how settled the ruling is, because a preliminary injunction and a final holding are not the same thing and the difference decides how much weight you can put on it.

Initial publication: August 20, 2026. Updated as rulings issue in the cases tracked here, and as new access-control mechanisms produce litigation.

Contents

• How to read the "how settled" field
• Control 1: robots.txt disallow
• Control 2: Terms of service
• Control 3: Anti-bot systems
• Control 4: Rate limiting
• Control 5: Login and password gates
• Control 6: Cease-and-desist plus IP blocking
• hiQ v. LinkedIn did not make scraping public data legal
• Control 7: Copyrighted material behind the wall
• What this means for your website

How to read the "how settled" field

Every entry below says how settled the answer is, and it is the field to read first. Courts say things at every stage of a case, and the stage decides how much weight the words carry.

A holding is a court deciding a question. A preliminary injunction ruling says only that one side is likely enough to win that the court will freeze things meanwhile, which is a far lower bar. A dismissal with prejudice ends a claim permanently. A dismissal without leave to amend means the court thinks no rewrite could save it. Pending means nobody has decided anything.

Coverage of this area collapses these constantly, which is how a preliminary-injunction ruling becomes "the court held that scraping is legal" three articles later. When you are deciding what to deploy, this field is what tells you whether you are standing on rock or on a prediction.

Control 1: robots.txt disallow

• What it is: a plain-text file at your root that names crawlers and the paths they should not fetch.
• What a court has said it buys you: nothing directly. No case holds that ignoring a disallow creates liability by itself.
• How settled: not at all. No controlling decision exists.
• What it does not buy you: enforcement. Google's own documentation states that its user-triggered fetchers generally ignore robots.txt, and OpenAI now documents a similar position for ChatGPT-User.
• How to deploy it: as documentation of intent, which can support a contract or trespass claim later, and as the one control that well-behaved crawlers do honor.

The absence of case law here is the finding. robots.txt is the control most website owners reach for first and the only one on this list with no judicial backing at all. It works because most crawler companies choose to honor it, which is a different thing from working because it must be honored.

Control 2: Terms of service

• What it is: the rules a visitor agrees to, whether by clicking or by browsing.
• What a court has said it buys you: a contract claim that federal copyright law does not swallow, but only if it is built on how the visitor got in rather than on what they copied.
• How settled: remand order signed March 28, 2026 and entered March 30, 2026, Reddit, Inc. v. Anthropic PBC, N.D. Cal. No. 3:25-cv-05643-TLT, Document 60, Judge Trina L. Thompson. A ruling about which court hears the case, not about who is right. The federal case is closed and the fight continues in San Francisco Superior Court, Docket CGC-25-625892.
• What it does not buy you: a federal computer-crime claim. In Van Buren v. United States (No. 19-783, decided June 3, 2021, Justice Barrett for a 6-3 Court) the Supreme Court held that a person "exceeds authorized access" only by obtaining information in areas of a computer that are off-limits to them. The Court framed it as a gates-up-or-down question, and rejected the government's reading that using authorized access for an improper purpose violates the statute. Breaking your usage rules is not the same as breaking in.
• How to deploy it: write the automated-access rules around method and purpose of access, not around copying.

Two courts in this district looked at near-identical facts, a scraper against a platform's terms, and came out opposite ways. The difference might be the most useful thing on this page.

In Reddit's suit against Anthropic, Anthropic moved the case to federal court, arguing it was a copyright claim in disguise. Judge Thompson sent it back. Reddit's content "may be covered by the broad subject matter of copyright," but that is only the first half of the test. The second half asks whether the rights asserted are equivalent to copyright rights, and a claim survives only if it carries an "extra element" that "changes the nature of the action." The court found Reddit's claims rest on "Anthropic's methods of access, restricted purposes, and deceptive conduct on Reddit's platform," and concluded that they "allege extra elements that are qualitatively different from the rights protected by the Copyright Act."

X Corp. got the opposite answer on the other framing. In X Corp. v. Bright Data Ltd. (N.D. Cal. No. C 23-03698 WHA, Document 83, filed May 9, 2024, Judge William Alsup), the court held that "X Corp.'s state-law claims against Bright Data based on scraping and selling of data are preempted by the Copyright Act," and dismissed the complaint in full. Alsup opened by quoting the Ninth Circuit's warning that letting platforms decide who may collect data they do not own and otherwise publish "risks the possible creation of information monopolies that would disserve the public interest."

So a terms claim written as they took our content reads as a copyright claim and gets swallowed by the Copyright Act. The same claim written as they lied about who they were, evaded our access rules, and hit our servers in a way we prohibited carries the extra element and survives. That is a drafting decision you make before anyone scrapes you, and it is the one thing on this page that changes an outcome.

The coverage tends to drop one caution. Van Buren left the neighboring question open on purpose. Footnote 8 states that the Court "need not address whether this inquiry turns only on technological (or 'code-based') limitations on access, or instead also looks to limits contained in contracts or policies." Whether your terms page can set the boundary that a federal computer-crime statute enforces is not settled law. It is a question the Supreme Court chose not to answer.

Control 3: Anti-bot systems

• What it is: a challenge, fingerprint check, or detection layer that blocks automated clients.
• What a court has said it buys you: a DMCA anti-circumvention claim, but only when it protects copyrighted content that you are authorized to protect.
• How settled: dismissed, in part with prejudice. Google LLC v. SerpApi, N.D. Cal. No. 4:25-cv-10826-YGR, order at Document 42, filed July 20, 2026, Chief Judge Yvonne Gonzalez Rogers.
• What it does not buy you: protection for content whose copyright you do not hold.
• How to deploy it: as a technical control. Do not assume it carries legal weight unless you own what is behind it.

Google argued that SerpApi broke the law by getting past SearchGuard, the system Google runs to stop automated scraping of its search results. I wrote about the ruling itself in Google Built Its Library By Scraping The Web, So It's Fair Game Too. The order describes SearchGuard's purpose in Google's own framing, to prevent unauthorized third parties from automatically accessing Google's search results, because that scraping imposes a "deadweight loss" on Google.

The claim failed on a gap in Google's own complaint. In the court's words, "Google does not allege that google.com or the Google Search results displayed therein are protected under the Copyright Act."

Google's fallback was the Knowledge Panel, which it said "often" accompanies results and "may contain some copyrighted content that Google licenses from third parties." The hedging is what killed the Knowledge Panel argument. The court noted Google "does not allege that the 'Knowledge Panel' is always included in Google Search results, or that the Knowledge Panel, if included in the Search results, always contains copyrighted content," concluding that "Google's allegations indicate a mix of content, some with copyrighted material and others without."

The holding: "because the DMCA does not apply where the work controlled by a technological measure is not protected under the Copyright Act, Google's claims under 17 U.S.C. § 1201(a)(1)(A) and 17 U.S.C. § 1201(a)(2) are subject to dismissal as a matter of law to the extent that they are premised on instances where SearchGuard controls access to Google Search results that do not contain any copyrighted content." That portion was dismissed "WITHOUT LEAVE TO AMEND because amendment would be futile."

A second portion, covering results that do contain a copyrighted component, was dismissed with leave to amend on a different and more useful ground. The statute protects a measure that requires "the application of information, or a process or a treatment, with the authority of the copyright owner, to gain access to the work," and the Ninth Circuit reads that phrase as requiring a plaintiff "to allege and later prove that the technological measure in question was implemented and functioned with the authority of the copyright owner." Google gave the court no reason to conclude that SearchGuard "was implemented and functions to control access to copyrighted components that may be included in Google Search results 'with the authority of the copyright owner[s].'" Google argued the phrase describes who may lawfully circumvent a measure rather than who may deploy one. The court called that "unavailing," because the cases Google cited interpret a different part of the law.

For anyone who has read the scraping coverage: SerpApi leaned on hiQ for the circumvention standard and the court refused it outright, holding that "hiQ addresses a statute other than the DMCA" and that "the holdings in that case are irrelevant to the analysis here."

An anti-bot wall earns DMCA protection only where it guards copyrighted work and where whoever deployed it had the copyright owner's authority to do so. The largest aggregator of other people's content on the web ran into that wall over its own search results. Any website that displays material it did not create is standing in the same place.

Control 4: Rate limiting

• What it is: a cap on request volume per client.
• What a court has said it buys you: untested as a standalone control. It appears as one of several circumvented measures in a live DMCA case.
• How settled: pending. Reddit, Inc. v. SerpApi LLC, S.D.N.Y. No. 1:25-cv-08736, filed October 22, 2025, a DMCA anti-circumvention case, before Judge Paul Engelmayer. First amended complaint filed February 6, 2026.
• What it does not buy you: nothing established, because nothing is decided.
• How to deploy it: for the operational reason, which is protecting your infrastructure.

That case is worth watching for a reason beyond rate limits. Reddit named four defendants together, SerpApi, Oxylabs, AWMProxy, and Perplexity, which reaches the search-data provider, two proxy networks, and the AI company in one action. It is an attempt to attach liability along a supply chain rather than to a single scraper.

Control 5: Login and password gates

• What it is: authentication in front of content.
• What a court has said it buys you: the firmest footing of any control here, and less than it did a year ago.
• How settled: preliminary injunction thrown out and the case sent back. Amazon.com Services, LLC v. Perplexity AI, Inc., 9th Cir. No. 26-1444, decided August 4, 2026, marked for publication so it counts as precedent.
• What it does not buy you: a bar on your own users sending agents through a login they are entitled to use.
• How to deploy it: for content you genuinely need to restrict, understanding it separates strangers from users rather than humans from machines.

Amazon won a preliminary injunction in the district court against Perplexity's Comet browser, over the agent reaching password-protected account pages. I covered what the case was set to decide in Amazon v. Perplexity: The CFAA Case That Decides Whether AI Agents Can Visit Your Website. The Ninth Circuit threw it out. The panel concluded "it was the user who 'accessed' Amazon's computers, with the help of Perplexity's AI agent, the 'Assistant,' to carry out specific acts on Amazon.com," and that the California state-law claim failed for the same reason.

The opinion records that "at the core of the dispute was Perplexity's decision not to use a 'user-agent string,' a mechanism 'that would communicate that the user has activated an AI agent.' That user-agent string would allow Amazon to block the Assistant's access to the Amazon store." A federal appellate case about agent access turned substantially on whether the agent identified itself in an HTTP header, which is the mechanism catalogued in the AI user-agent landscape.

Control 6: Cease-and-desist plus IP blocking

• What it is: a letter revoking permission, backed by blocking at the network layer.
• What a court has said it buys you: an open question, and specifically not a settled route to a federal claim.
• How settled: preliminary injunction affirmed, on the low serious-questions bar. hiQ Labs, Inc. v. LinkedIn Corporation, 9th Cir. No. 17-16783, decided April 18, 2022, after the Supreme Court sent the case back.
• What it does not buy you: a reliable conversion of continued scraping into unauthorized access.
• How to deploy it: as one input to a negotiation, not as a legal trigger.

LinkedIn sent hiQ a cease-and-desist and blocked it, then argued further scraping was access "without authorization." The Ninth Circuit affirmed an injunction against LinkedIn, concluding hiQ "raised a serious question as to whether the CFAA 'without authorization' concept is inapplicable" on facts like these, where prior authorization is not generally required but a particular person or bot is refused access.

That sentence is doing less work than it is usually asked to do.

hiQ v. LinkedIn did not make scraping public data legal

The case behind this control is also the most miscited case in this whole area. hiQ v. LinkedIn gets cited as establishing that scraping publicly available data does not violate the Computer Fraud and Abuse Act. Even people paid to read carefully do it: SerpApi's own counsel leaned on hiQ in the Google case, where the court refused it outright because "hiQ addresses a statute other than the DMCA."

The court never decided that.

It affirmed a preliminary injunction, and the standard it applied was whether hiQ had raised "serious questions" about how the case might come out. That is a prediction about likely outcomes made to decide whether to freeze the situation while a case proceeds. It is not a decision on the question.

The same panel also concluded that to scrape LinkedIn data, hiQ needed to access LinkedIn servers, and that those servers were "protected computers" under the statute. The reasoning was never that public data sits outside the CFAA.

This matters for a website owner in a practical way. If you have been told that public equals unprotected, you may have concluded that your access controls are pointless and stopped thinking about them. The actual state of the law is less settled and more contested than that, which means your own decisions carry more weight, not less.

Control 7: Copyrighted material behind the wall

• What it is: putting content you own the copyright in behind a technological measure.
• What a court has said it buys you: the one path through the DMCA that survived in the SerpApi order.
• How settled: dismissed with leave to amend, so a path rather than a win.
• What it does not buy you: anything, if the material is licensed from others without authority to protect it, or if the protection is inconsistent.
• How to deploy it: know what you own, and apply the measure to that specifically rather than to your whole website.

Control 7 is the inverse of Control 3, and it is the actionable half. The DMCA route is available to a website owner whose wall protects work they hold the copyright in. It is not available to an aggregator. Whether Google can rewrite its complaint to fit this path is an open question at the time of writing.

What this means for your website

Across all seven controls, the pattern is the same. The legal layer is slower than your traffic, narrower than your threat model, and dependent on facts about your specific setup that no general rule covers. robots.txt has no case behind it. Terms of service stopped supporting federal computer-crime claims in 2021. An anti-bot system protects you only where you own what is behind it. A login gate no longer separates your users from your users' agents.

So stop asking "is this legal" and start asking "does this work." The law is years behind your traffic. Your edge settings work today, and you do not need a court's permission to use them.

The working version of that question sorts everything on this page into signs and locks. robots.txt and your terms of service are signs: they tell honest visitors what you want and they document your intent for whatever legal fight comes later, but they stop nobody. Logins, edge blocks, and copyright on what sits behind a wall are locks. Signs are for the honest. Locks are for everyone. Deploy both, and never confuse which one you are relying on.

Then decide who gets blocked, and that is not a legal question either. It is the oldest business question there is: what am I getting in return? Every AI crawler reading you is either sending something back (citations, visitors, a reason to be read) or consuming and returning nothing, and each one is a different visitor deserving its own answer. That is the argument I made in Whether an AI agent can visit your website should be a business decision. The courts are not going to make it for you, and their rules will arrive years from now shaped by disputes between very large companies about facts that are not yours.

I made that call on this website while writing this page, twice. The newer AI companies (OpenAI, Anthropic, Perplexity) clear the return bar for now: their fetchers put my pages in front of people who ask assistants questions. Meta never did. I documented on August 1 how everyone negotiates with Google while Meta reads the web for free, and for this website the answer to "what am I getting in return" from Meta was absolutely nothing. So on August 20 I blocked everything Meta in Cloudflare's AI Crawl Control. It took two minutes and no lawyer. Checked from outside minutes later: meta-externalagent gets a 403, and the link-preview fetcher still gets a 200 so shared links still look decent.

Then the rule caught my own outdated. My robots.txt had spent months explicitly welcoming Bytespider, ByteDance's training crawler, by name, and under "what am I getting in return," Bytespider fails the same test Meta failed: it trains models whose answers never send anyone here. No reason to block Meta and hold the door for ByteDance, so Bytespider joined the block the same afternoon, verified the same way: 403. That is what this decision actually looks like in practice: not a policy you write once, a review you run per crawler, and sometimes it catches you.

An edge block matches the name a visitor claims, so it stops the honest holder of the name. That is fine. Blocks are for companies honest enough to be blockable. The dishonest ones were never going to honor anything, which is what the logs on this website already showed.

Know which of these controls you have deployed and whether each one is a sign or a lock. Most website owners have at least one control they believe is protecting them and one they have never checked. As for the law: not much yet, plan as if that will not change, and keep an eye in case it does. The keeping-an-eye is what this page is for. It updates as rulings issue in the cases tracked here.