229: Does llms.txt Work? What 137,000 Domains' Server Logs Show
Slobodan "Sani" Manic
Website Optimisation Consultant, No Hacks Founder & Keynote Speaker
CXL-certified conversion specialist and WordPress Core Contributor helping companies optimise websites for both humans and AI agents.
Ahrefs published a study in June looking at server logs across 137,000 domains. Of the 28% that had a valid llms.txt file, 97% got exactly zero requests in May, not low traffic but literally none. And of the 3% that did get fetched, 22% of those readers were SEO audit tools, the same tools that flag you for not having the file in the first place.
I checked my own Cloudflare logs at nohacks.co. Same story. Nobody fetches the file. GPTBot, ClaudeBot, Perplexity, all of them are perfectly happy reading HTML. The systems llms.txt was invented for don't care about it.
This is the pattern with AI search optimization right now: a plausible idea, massive adoption, zero evidence it does anything. The pitch is always a screenshot. The truth lives in server logs. And when people can't see inside the black box that decides whether their website gets mentioned, they pay whoever sounds most confident.
I use one line to sort every pitch that lands in my inbox: does it change what a machine can read and do on your website, or what the machine says about you? The first is architecture, the second is a hack, and hacks die at the next model update while architecture survives.
KEY TAKEAWAYS
- Check your own server logs for llms.txt requests before trusting any vendor claims. It takes five minutes and reveals whether anyone actually fetches the file.
- Ask any AI visibility vendor for evidence at the level of server logs, tests, or documented mechanisms before spending money. Apply the same bar you would for anyone touching revenue.
- Sort every AI optimization pitch with one question: does it change what a machine can read and do on my website, or what the machine says about me? The first survives model updates, the second does not.
SHOW NOTES
The llms.txt Reality Check
Ahrefs analyzed server logs across 137,000 domains in June 2026. Twenty-eight percent had a valid llms.txt file, which sounds like impressive adoption for a file format that appeared out of nowhere. But 97% of those files received zero requests in May, not merely infrequent visits but none at all.
The 3% that did get fetched tell an even stranger story. Around 22% of those readers were SEO audit tools, the exact tools that penalize websites for not having an llms.txt file. The file exists primarily to satisfy the tools that check for the file. Cloudflare logs at nohacks.co confirm the same pattern: GPTBot, ClaudeBot, and PerplexityBot are all perfectly content reading HTML.
Why the Market Keeps Producing Tricks
LLMs represent the blackest of black boxes. Nobody outside a few labs can accurately observe what happens inside. When people cannot see how the system decides whether their website gets mentioned, they pay whoever sounds most confident. That dynamic creates perfect conditions for unproven tactics.
Traffic reality is shifting away from Google sending clicks, so budgets need somewhere to go. Fear plus money plus opacity equals an explosion of products that don't survive contact with server logs. The pitch is always a screenshot of ChatGPT mentioning a brand. The truth requires checking whether anything actually fetched the file.
Visibility Scores and the Prompt Problem
A vendor sells you a number claiming to measure how visible you are to AI. But what does that even mean? The prompt determines everything. If you prompt LLMs specifically about yourself, you will appear highly visible. Most visibility scores are grading exams that vendors write themselves, selecting prompts where clients are easy to mention. That is not AI visibility. That is theater.
The Hack-Architecture Distinction
Every generation of search produced its own hacks: keyword stuffing, link farms, doorway pages. Every generation of hacks died the same boring death. The sellers moved on without explaining what failed.
One question separates durable work from temporary tricks. Does this change what a machine can read and do on my website, or does it try to influence what the machine says about me? Architecture means structure a machine can parse without confusion, facts stated plainly enough to survive being extracted, and real endpoints an agent can call. Hacks are rented. Architecture is owned.
What Actually Survives Model Updates
Google updates punished websites for years, and the complaints were predictable. But the websites doing fundamental work rarely suffered. The same pattern applies to LLMs. If your foundation is semantic HTML, clear factual statements, and genuine capability, model updates will not erase you. If your strategy depends on Reddit bombing or citation farming, the clock is ticking.
The shift to non-human visitors is real and measurable. Agents are barely buying today, but the foundation is being set this year. Panic and confusion are understandable responses. But when preparation is uncertain, fundamentals beat tricks every time.
QUESTIONS ANSWERED
Does llms.txt actually work for AI search optimization?
According to Ahrefs' June 2026 study of 137,000 domains, llms.txt does not work for AI search optimization. Ninety-seven percent of llms.txt files received zero requests in May. The AI crawlers like GPTBot, ClaudeBot, and PerplexityBot read HTML directly and do not rely on llms.txt files for indexing or citation decisions.
What is llms.txt and what was it designed for?
llms.txt is a text file placed on a website, similar in concept to robots.txt, intended to help AI systems learn who you are and where your important pages live. The file uses markdown formatting. While llms.txt has a valid use case for developers pointing coding assistants at documentation, Ahrefs' server log data shows AI crawlers are not actually reading these files for search or citation purposes.
Who actually reads llms.txt files according to server logs?
Ahrefs' study found that SEO audit tools are the primary readers of llms.txt files. Of the 3% of domains where llms.txt was fetched at all, approximately 22% of those requests came from SEO audit tools. These are the same tools that flag websites for not having an llms.txt file, creating a circular dynamic where the file exists mainly to satisfy the tools checking for the file.
How can I tell if an AI SEO tactic is a hack or legitimate?
Ask one question: does the tactic change what a machine can read and do on your website, or does it try to influence what the machine says about you? Architecture changes, such as semantic HTML, structured data, and clear factual statements, are durable and owned. Tactics that manipulate citations or game prompts are hacks that die when models update. Server logs provide the evidence, not screenshots of ChatGPT responses.
Why do AI visibility scores often mislead marketers?
AI visibility scores are often misleading because vendors select the prompts used for measurement. If a vendor chooses prompts where a client is easy to mention, the visibility score will be high regardless of actual discoverability. The prompt determines everything in LLM responses, so a visibility score without transparent prompt methodology is essentially a test where the vendor writes both the questions and the grading criteria.
How do I check if anyone is fetching my llms.txt file?
Check your server logs or CDN logs directly, not analytics tools like Google Analytics. Cloudflare, Vercel, and similar services provide request logs that show every fetch of any file on your domain. Filter for requests to /llms.txt and examine the user agents. This check takes approximately five minutes and reveals whether AI crawlers or any other systems are actually requesting the file.
RELATED ARTICLES
Whether an AI agent can visit your website should be a business decision, not a legal one
The Ninth Circuit overturned the ban on Perplexity's shopping agents accessing Amazon, the second time in a month a platform reached for an old statute to block a machine visitor and lost. The courts keep declining to hold the switch. Cloudflare's defaults will happily hold it for you, and from September 15 that comes with real consequences for search too.
Everyone Is Negotiating With Google While Meta Reads The Web For Free
The web's machine-access fight is aimed at Google: block it, license to it, sue it. Meanwhile Meta's two crawlers became the majority of AI agent traffic, sending almost nothing back, and unless you are News Corp's size, nobody is holding that meeting for you.
AI Search Only Feels New If Your SEO Was Shallow
Training for AI search and training for Google are two different races with almost the same training. The industry is arguing about what to call the new race. That is the wrong argument, and whether any of this feels new to you depends entirely on what your SEO was in the first place.
ENJOYING THIS EPISODE?
Practical strategies for making your website work for AI agents and the humans using it. Read by SEOs, developers, and AI researchers. Exclusive tools, free for subscribers.