229: Does llms.txt Work? What 137,000 Domains' Server Logs Show
Slobodan "Sani" Manic
Website Optimisation Consultant, No Hacks Founder & Keynote Speaker
CXL-certified conversion specialist and WordPress Core Contributor helping companies optimise websites for both humans and AI agents.
Ahrefs published a study in June looking at server logs across 137,000 domains. Of the 28% that had a valid llms.txt file, 97% got exactly zero requests in May, not low traffic but literally none. And of the 3% that did get fetched, 22% of those readers were SEO audit tools, the same tools that flag you for not having the file in the first place.
I checked my own Cloudflare logs at nohacks.co. Same story. Nobody fetches the file. GPTBot, ClaudeBot, Perplexity, all of them are perfectly happy reading HTML. The systems llms.txt was invented for don't care about it.
This is the pattern with AI search optimization right now: a plausible idea, massive adoption, zero evidence it does anything. The pitch is always a screenshot. The truth lives in server logs. And when people can't see inside the black box that decides whether their website gets mentioned, they pay whoever sounds most confident.
I use one line to sort every pitch that lands in my inbox: does it change what a machine can read and do on your website, or what the machine says about you? The first is architecture, the second is a hack, and hacks die at the next model update while architecture survives.
KEY TAKEAWAYS
- Check your own server logs for llms.txt requests before trusting any vendor claims. It takes five minutes and reveals whether anyone actually fetches the file.
- Ask any AI visibility vendor for evidence at the level of server logs, tests, or documented mechanisms before spending money. Apply the same bar you would for anyone touching revenue.
- Sort every AI optimization pitch with one question: does it change what a machine can read and do on my website, or what the machine says about me? The first survives model updates, the second does not.
SHOW NOTES
The llms.txt Reality Check
Ahrefs analyzed server logs across 137,000 domains in June 2026. Twenty-eight percent had a valid llms.txt file, which sounds like impressive adoption for a file format that appeared out of nowhere. But 97% of those files received zero requests in May, not merely infrequent visits but none at all.
The 3% that did get fetched tell an even stranger story. Around 22% of those readers were SEO audit tools, the exact tools that penalize websites for not having an llms.txt file. The file exists primarily to satisfy the tools that check for the file. Cloudflare logs at nohacks.co confirm the same pattern: GPTBot, ClaudeBot, and PerplexityBot are all perfectly content reading HTML.
Why the Market Keeps Producing Tricks
LLMs represent the blackest of black boxes. Nobody outside a few labs can accurately observe what happens inside. When people cannot see how the system decides whether their website gets mentioned, they pay whoever sounds most confident. That dynamic creates perfect conditions for unproven tactics.
Traffic reality is shifting away from Google sending clicks, so budgets need somewhere to go. Fear plus money plus opacity equals an explosion of products that don't survive contact with server logs. The pitch is always a screenshot of ChatGPT mentioning a brand. The truth requires checking whether anything actually fetched the file.
Visibility Scores and the Prompt Problem
A vendor sells you a number claiming to measure how visible you are to AI. But what does that even mean? The prompt determines everything. If you prompt LLMs specifically about yourself, you will appear highly visible. Most visibility scores are grading exams that vendors write themselves, selecting prompts where clients are easy to mention. That is not AI visibility. That is theater.
The Hack-Architecture Distinction
Every generation of search produced its own hacks: keyword stuffing, link farms, doorway pages. Every generation of hacks died the same boring death. The sellers moved on without explaining what failed.
One question separates durable work from temporary tricks. Does this change what a machine can read and do on my website, or does it try to influence what the machine says about me? Architecture means structure a machine can parse without confusion, facts stated plainly enough to survive being extracted, and real endpoints an agent can call. Hacks are rented. Architecture is owned.
What Actually Survives Model Updates
Google updates punished websites for years, and the complaints were predictable. But the websites doing fundamental work rarely suffered. The same pattern applies to LLMs. If your foundation is semantic HTML, clear factual statements, and genuine capability, model updates will not erase you. If your strategy depends on Reddit bombing or citation farming, the clock is ticking.
The shift to non-human visitors is real and measurable. Agents are barely buying today, but the foundation is being set this year. Panic and confusion are understandable responses. But when preparation is uncertain, fundamentals beat tricks every time.
QUESTIONS ANSWERED
Does llms.txt actually work for AI search optimization?
According to Ahrefs' June 2026 study of 137,000 domains, llms.txt does not work for AI search optimization. Ninety-seven percent of llms.txt files received zero requests in May. The AI crawlers like GPTBot, ClaudeBot, and PerplexityBot read HTML directly and do not rely on llms.txt files for indexing or citation decisions.
What is llms.txt and what was it designed for?
llms.txt is a text file placed on a website, similar in concept to robots.txt, intended to help AI systems learn who you are and where your important pages live. The file uses markdown formatting. While llms.txt has a valid use case for developers pointing coding assistants at documentation, Ahrefs' server log data shows AI crawlers are not actually reading these files for search or citation purposes.
Who actually reads llms.txt files according to server logs?
Ahrefs' study found that SEO audit tools are the primary readers of llms.txt files. Of the 3% of domains where llms.txt was fetched at all, approximately 22% of those requests came from SEO audit tools. These are the same tools that flag websites for not having an llms.txt file, creating a circular dynamic where the file exists mainly to satisfy the tools checking for the file.
How can I tell if an AI SEO tactic is a hack or legitimate?
Ask one question: does the tactic change what a machine can read and do on your website, or does it try to influence what the machine says about you? Architecture changes, such as semantic HTML, structured data, and clear factual statements, are durable and owned. Tactics that manipulate citations or game prompts are hacks that die when models update. Server logs provide the evidence, not screenshots of ChatGPT responses.
Why do AI visibility scores often mislead marketers?
AI visibility scores are often misleading because vendors select the prompts used for measurement. If a vendor chooses prompts where a client is easy to mention, the visibility score will be high regardless of actual discoverability. The prompt determines everything in LLM responses, so a visibility score without transparent prompt methodology is essentially a test where the vendor writes both the questions and the grading criteria.
How do I check if anyone is fetching my llms.txt file?
Check your server logs or CDN logs directly, not analytics tools like Google Analytics. Cloudflare, Vercel, and similar services provide request logs that show every fetch of any file on your domain. Filter for requests to /llms.txt and examine the user agents. This check takes approximately five minutes and reveals whether AI crawlers or any other systems are actually requesting the file.
RELATED ARTICLES
Cloudflare Built A Browser That Throws Away The Pixels
Cloudflare launched Kitesurf, a browser for AI agents that treats rendering as optional: "visual perfection, smooth 60-fps scrolling is not" important. Four weeks earlier OpenAI retired Atlas, a browser built so a person could watch an agent work. Anything your website communicates only in painted pixels, the trust badges and the reassurances, is now definitively invisible to the visitor reading you.
The Biggest “AI Crawler” On My Website Was Hunting For Credentials
Cloudflare's CFO told analysts that non-human traffic could be 1,000 times human traffic within five years and that humans will be "a rounding error on the internet." So I looked at my own crawler logs. The single largest AI crawler on nohacks.co arrived under Common Crawl's name and spent the day asking for my SSH keys, my .env files, and my MCP config.
Shopify Gave Every Store An Agent API Before The Agents Arrived
Shopify switched on WebMCP for every Liquid storefront on August 5: catalog, cart, and checkout are now tools an AI agent can call, with nothing to install. I checked three real storefronts: search already returns products and prices in a stock Chrome, checkout dies on an internal error, and no agent is calling any of it. Why this is not another llms.txt, and what to test before the agents arrive.
ENJOYING THIS EPISODE?
Practical strategies for making your website work for AI agents and the humans using AI to find you. Once a week you get the new articles, the latest podcast episode, and a few links worth keeping.