229: Does llms.txt Work? What 137,000 Domains' Server Logs Show
Slobodan "Sani" Manic
Website Optimisation Consultant, No Hacks Founder & Keynote Speaker
CXL-certified conversion specialist and WordPress Core Contributor helping companies optimise websites for both humans and AI agents.
Ahrefs published a study in June looking at server logs across 137,000 domains. Of the 28% that had a valid llms.txt file, 97% got exactly zero requests in May, not low traffic but literally none. And of the 3% that did get fetched, 22% of those readers were SEO audit tools, the same tools that flag you for not having the file in the first place.
I checked my own Cloudflare logs at nohacks.co. Same story. Nobody fetches the file. GPTBot, ClaudeBot, Perplexity, all of them are perfectly happy reading HTML. The systems llms.txt was invented for don't care about it.
This is the pattern with AI search optimization: a plausible idea, massive adoption, zero evidence it does anything. The pitch is always a screenshot. The truth lives in server logs. And when people can't see inside the black box that decides whether their website gets mentioned, they pay whoever sounds most confident.
I use one line to sort every pitch that lands in my inbox: does it change what a machine can read and do on your website, or what the machine says about you? The first is architecture, the second is a hack, and hacks die at the next model update while architecture survives.
KEY TAKEAWAYS
- Check your own server logs for llms.txt requests before trusting any vendor claims. It takes five minutes and reveals whether anyone actually fetches the file.
- Ask any AI visibility vendor for evidence at the level of server logs, tests, or documented mechanisms before spending money. Apply the same bar you would for anyone touching revenue.
- Sort every AI optimization pitch with one question: does it change what a machine can read and do on my website, or what the machine says about me? The first survives model updates, the second does not.
SHOW NOTES
The llms.txt Reality Check
Ahrefs analyzed server logs across 137,000 domains in June 2026. Twenty-eight percent had a valid llms.txt file, which sounds like impressive adoption for a file format that appeared out of nowhere. But 97% of those files received zero requests in May, not merely infrequent visits but none at all.
The 3% that did get fetched tell an even stranger story. Around 22% of those readers were SEO audit tools, the exact tools that penalize websites for not having an llms.txt file. The file exists primarily to satisfy the tools that check for the file. Cloudflare logs at nohacks.co confirm the same pattern: GPTBot, ClaudeBot, and PerplexityBot are all perfectly content reading HTML.
Why the Market Keeps Producing Tricks
LLMs represent the blackest of black boxes. Nobody outside a few labs can accurately observe what happens inside. When people cannot see how the system decides whether their website gets mentioned, they pay whoever sounds most confident. That dynamic creates perfect conditions for unproven tactics.
Traffic reality is shifting away from Google sending clicks, so budgets need somewhere to go. Fear plus money plus opacity equals an explosion of products that don't survive contact with server logs. The pitch is always a screenshot of ChatGPT mentioning a brand. The truth requires checking whether anything actually fetched the file.
Visibility Scores and the Prompt Problem
A vendor sells you a number claiming to measure how visible you are to AI. But what does that even mean? The prompt determines everything. If you prompt LLMs specifically about yourself, you will appear highly visible. Most visibility scores are grading exams that vendors write themselves, selecting prompts where clients are easy to mention. That is not AI visibility. That is theater.
The Hack-Architecture Distinction
Every generation of search produced its own hacks: keyword stuffing, link farms, doorway pages. Every generation of hacks died the same boring death. The sellers moved on without explaining what failed.
One question separates durable work from temporary tricks. Does this change what a machine can read and do on my website, or does it try to influence what the machine says about me? Architecture means structure a machine can parse without confusion, facts stated plainly enough to survive being extracted, and real endpoints an agent can call. Hacks are rented. Architecture is owned.
What Actually Survives Model Updates
Google updates punished websites for years, and the complaints were predictable. But the websites doing fundamental work rarely suffered. The same pattern applies to LLMs. If your foundation is semantic HTML, clear factual statements, and genuine capability, model updates will not erase you. If your strategy depends on Reddit bombing or citation farming, the clock is ticking.
The shift to non-human visitors is real and measurable. Agents are barely buying today, but the foundation is being set this year. Panic and confusion are understandable responses. But when preparation is uncertain, fundamentals beat tricks every time.
FAQ
Does llms.txt actually work for AI search optimization?
According to Ahrefs' June 2026 study of 137,000 domains, llms.txt does not work for AI search optimization. Ninety-seven percent of llms.txt files received zero requests in May. The AI crawlers like GPTBot, ClaudeBot, and PerplexityBot read HTML directly and do not rely on llms.txt files for indexing or citation decisions.
What is llms.txt and what was it designed for?
llms.txt is a text file placed on a website, similar in concept to robots.txt, intended to help AI systems learn who you are and where your important pages live. The file uses markdown formatting. While llms.txt has a valid use case for developers pointing coding assistants at documentation, Ahrefs' server log data shows AI crawlers are not actually reading these files for search or citation purposes.
Who actually reads llms.txt files according to server logs?
Ahrefs' study found that SEO audit tools are the primary readers of llms.txt files. Of the 3% of domains where llms.txt was fetched at all, approximately 22% of those requests came from SEO audit tools. These are the same tools that flag websites for not having an llms.txt file, creating a circular dynamic where the file exists mainly to satisfy the tools checking for the file.
How can I tell if an AI SEO tactic is a hack or legitimate?
Ask one question: does the tactic change what a machine can read and do on your website, or does it try to influence what the machine says about you? Architecture changes, such as semantic HTML, structured data, and clear factual statements, are durable and owned. Tactics that manipulate citations or game prompts are hacks that die when models update. Server logs provide the evidence, not screenshots of ChatGPT responses.
Why do AI visibility scores often mislead marketers?
AI visibility scores are often misleading because vendors select the prompts used for measurement. If a vendor chooses prompts where a client is easy to mention, the visibility score will be high regardless of actual discoverability. The prompt determines everything in LLM responses, so a visibility score without transparent prompt methodology is essentially a test where the vendor writes both the questions and the grading criteria.
How do I check if anyone is fetching my llms.txt file?
Check your server logs or CDN logs directly, not analytics tools like Google Analytics. Cloudflare, Vercel, and similar services provide request logs that show every fetch of any file on your domain. Filter for requests to /llms.txt and examine the user agents. This check takes approximately five minutes and reveals whether AI crawlers or any other systems are actually requesting the file.
TRANSCRIPT
Read the full transcriptCollapse
[00:00:00] We need to talk about hacks. Most of the stuff that's sold as, quote-unquote, "AI search optimization" has never really been tested by most of those people selling it
[00:00:15] You know, the tricks don't survive contact with the reality, contact with the server logs. So today, let's talk about them. Let's talk about hacks. Let's talk about AI hacks, and let's talk about why the market, why digital marketing keeps producing short-term tricks And let's talk about how we can all avoid that.
[00:00:42] This is No Hacks, agentic web, no hype. It's not easy to do, but I keep trying
[00:00:51] So here's where we are. Over the past year or so, there's a whole economy that's grown around one promise. We will get your ass into AI answers. There are special files you're supposed to add. There are visibility scores. There are citation services to get you mentioned in the places where AI systems are supposedly getting the information from.
[00:01:13] There are checklists. There's Reddit bombing. Some of that costs a few euros a month. Some of it is a lot more expensive. But you know what? Almost none of it has evidence. And I'm not saying the evidence is weak. I'm saying there's zero evidence for things that are being sold as AI search optimization service.
[00:01:36] The typical pitch I keep getting, is either a screenshot of, ChatGPT or, , sometimes they're not even a screenshot. It's just a, "We can make this work for you." And I, I get why it sells, 'cause AI answers and, and LLMs and GEO, AEO, it's the blackest of black boxes.
[00:01:57] You cannot see inside the system that decides whether your website exists, whether it should be mentioned. And when people cannot see, they pay someone who sounds certain. Think about the people that in the-- not the early days, but like the early days of SEO growing. There are people, I'm not going to name names, but there are people that come to mind that just sounded confident.
[00:02:22] They were not good at their job. They just sounded confident That's the market right now. The fear of a black box, big budget because spending money on traditional SEO is not as wise as it used to be, and a lot of people pretending they're confident. That's the entire market. So let's do what nobody selling this stuff does.
[00:02:50] Let's actually check one of those. Let's actually check one of the greatest things in the world of GEO. Ask any LLM how to optimize a website for LLMs, and this will be number one or number two or both. I'm talking about llms.txt. If you are lucky enough to never have heard of llms.txt, it's a text file you put on your website, kind of like robots.txt, but for LLMs.
[00:03:20] Not really, but kind of. And the idea is that the AI system will read it to learn who you are, where your important pages live, all that kind of stuff. It's a, it's a very plausible idea. On paper, this sounds good. Now, there's a spec, there's a proposal, there are numerous conference talks, and there are thousands, if not tens or hundreds of thousands of websites adopting this.
[00:03:51] SEO tools are flagging websites that don't have one. Agencies just put this as, you know, in a deliverable. This is a task that an agency doing AI SEO, GEO, AEO, for a client must do because the clients expect it. There's one problem. There's one problem with llms.txt, and it doesn't make a lot of sense today.
[00:04:16] It might in the future, it could be llms.txt, it could be something similar, but today it does not make sense. In June, so two months ago, Ahrefs published a study. They looked at server logs across a hundred and thirty-seven thousand domains. Server logs, not analytics, not Google Analytics or anything like that.
[00:04:37] Server logs.
[00:04:39] What do you think they found?
[00:04:43] So 28% of the domains that was observed in this study had a valid llms.txt file. Sure, and Ahrefs customers are more technical, they're more advanced, so 28% is probably as high as this number could ever be. Still, that's enormous adoption for a file that appeared out of nowhere. Second number, and I want you to pay attention to this Listening?
[00:05:13] 97% of those llms.txt files across 137,000 domains got... You wanna guess how much traffic they got in May? Zero. Zero. So not occasionally visited. Zero for a month. Zero requests. And that's exactly what I'm seeing at, at, at nohax.co. I'm looking at Cloudflare logs. Nobody cares about these files. So all the crawlers, GPTbot, Cloudbot, Perplexity bot, everything else, the systems that this file was invented for, they are happy reading HTML Huh.
[00:06:00] Now, there's a fun detail on the 3% of domains that had something fetch the file
[00:06:10] Of those 3% of domains that had s- anyone look at their llms.txt, 22% of those readers of llms.txt files were... You're not ready for this. SEO audit tools, the tools that tell you that you have a problem if you don't have an llms.txt file. That's the main thing on the internet that reads llms.txt
[00:06:37] So if someone tells you that this will improve your website in any way, they're selling you a hack Now, does this file have a good potential use? Yes, it does. Absolutely. And even Google has it in Lighthouse for the agentic, uh, check, agentic readiness, I don't know what they call it. They will check if you have an llms.txt file, if it's formatted well, all that stuff.
[00:07:08] If you are a developer, you can just point your coding agent, assistant, whatever you use at an llms.txt file on a documentation website. And for that, it kinda works. It's a niche tool. This has nothing to do with AI visibility, with AI search. And look, every client I've had this year has at least asked about llms.txt.
[00:07:33] I don't know how they learn about it. But just like the rest of the AI industry, this thing is so loud and so useless, not rest of AI, most of AI industry, so useless that I just don't get it.
[00:07:49] It's a markdown formatted text file. There's nothing fancy about this. And according to this study, well, nobody's reading the file. So that's one thing. Llms.txt, it's been bothering me for a long time because, you know, you know those Jordan Petersons and Hubermans and all those grifters and scammers, they will tell you something that sounds like it makes sense. And I'm not suggesting that the entire AI search industry, I don't know if that's the name of the industry, but it-- that, that entire category is trying to be Jordan Peterson.
[00:08:31] I hope not. But If someone opens with llms.txt and tells you that they will put you in ChatGPT, run. Run. Run away, don't look back, and never-- Just block their email. So the question really is why does this keep happening?
[00:08:53] There's always some stupid hack that if you think about it for a second, you realize that there's no way this makes sense. Like, stuff your page with keywords years, decades ago. This is so stupid. If it works, it works because the system that's supposed to use this and monitor this is not advanced enough.
[00:09:14] It will stop working, and it'll get punished. Same thing with Reddit bombing. I don't see a lot of that anymore, but the GEO strategy for a while, maybe still is, for a while, was just post all the things about your brand, just listicles or whatever on Reddit, comparisons, and, and, and then ChatGPT will cite that.
[00:09:36] Look, that, that's nice and sweet, but you know it's a hack when you start doing it, and you know it's going to stop working when you start doing it. So the conditions for this are really, really good because this AI thing, LLMs, it- it's a black box for people who are building these things. Nobody outside a few labs can even observe what's happening accurately.
[00:10:00] I, I don't think they can, but they, they, they got a higher level of accuracy than us. So everybody has real fears because the traffic reality is shifting. We're going away from SEO and, and Google sending clicks, and now those budgets need to go somewhere, and you need to show something for them. And it just needs to be spent.
[00:10:23] And that combination produces products and situations that just, just don't make a lot of sense. Look at visibility scores. Like a vendor will sell you a number that just says how visible you are to AI. Not all vendors, but it's a selling point for a lot. I get those pitches daily. There's a new AI visibility tool every day that wants to be featured on the blog or podcast And what does it even mean how visible you are to AI? What, what's the prompt? If I prompt all the LLMs about something that's specifically about me, I'll be very visible Fun fact, if you wanna go on a, on a LLM deep dive rabbit hole, go to free ChatGPT, so you're not logged in.
[00:11:20] You're logged in, but it's free, you're not paying. So it doesn't use, uh, web search by default. And just type in what is, and then your first name, last name, known for. And just send me screenshots. hi@nohackseo. I wanna see what you're getting, 'cause I'm not happy with what I'm getting. It's disturbing. So visibility scores are just, um, a grading exam for the test that you're writing, the way most people are selling, because they'll just pick some prompts that are easy to be mentioned for.
[00:11:59] I'm not saying it's all like visibility, AI visibility, share of voice, all that stuff. That's not fully a scam. But the way it's being sold will get you into ChatGPT. I mean, you ask ChatGPT something, if it doesn't have you in the knowledge base before the model cutoff, training data cutoff, it will just search.
[00:12:21] It will use SERP API to search Google or get Google data, and it will get whatever's ranking on Google or wherever else. Let's talk about citation farming next. The prompt stuffing, all those things that make very little sense to me. They're all just like anything else with AI.
[00:12:43] They're a sexy demo where you can show that something is happening or suggest that something is happening But it's not new. Every generation of search produced the hacks. I mentioned keyword stuffing, there were the link farms, doorway pages, and every generation of hacks just died the same painful, boring death.
[00:13:08] And the ones selling that They're not there to explain what failed. They just move on
[00:13:17] I wanna feel bad for people getting burned because there are some people that are just getting scammed into this. But for people doing the scam, I, I just, uh, no. No, no, no, no. So my position about all this, and like I've been talking about this on the podcast, I've been writing about this on the blog. My, my, my take is that I really believe that your next million visitors to your website will not be human.
[00:13:42] I do believe that the machines that are now reading and citing websites are moving into using the websites, buying from websites. That's what this podcast is about. GEO stops at the door. It just focuses on read and cite. I, I think there's more. I think there's a lot more. So there's one part that there's a lot of nonsense being sold, and there's also my belief, I really think that this is going somewhere.
[00:14:13] I really think that the internet is changing, and it will never go back to how it was. And both of those things can be true at the same time. So I'm not anti-urgency to change things. I'm not anti-AI in general. I'm, I'm just anti-unevidenced, for lack of a better , word. But the shift is real. The shift is measurable.
[00:14:38] But a lot of stuff, the stuff that's being sold simply is not
[00:14:44] By the way, a reminder, I have a weekly newsletter. If you're not subscribed to that, this is where I try to share the most important links, uh, from the previous week. I usually send them on Wednesdays. I might change the day. I will see about that. But if you're not subscribed, please go to nohacks.co/subscribe.
[00:15:04] It's also linked in the episode description.
[00:15:08] Back to the episode, back to hacks. So as I said, just being a purely a skeptic, it, it, it doesn't really do much So if I tell you that all those tricks don't work, and I really believe they don't, I'm not seeing any of that working. I, I need to tell you what I think does work.
[00:15:30] And I'll bore you to death because what works is boring. There's a reason this podcast is called No Hacks. I, I named it in 2021-- 2020 before launch, really. Before any of this existed. We didn't have LLMs back then. It was... Well, it was the pandemic, so it wasn't better times, but, ah, it was easier without the LLMs.
[00:15:54] The name of the podcast, I don't ha- I don't hate the, the HBO show Hacks. I love it, so it's not that. But the name of the podcast is really fundamentals over tricks. So what survives a Google update? What survives model updates? It's simple. It's boring as hell, but it's simple. It's a structure that a machine can read without getting confused
[00:16:20] it's facts that are stated plainly enough to avoid being left on the page and survive being pulled out of the page and used somewhere. It's real capability that an agent can call on the page, an actual endpoint defined in whatever way that answers an actual question for that agent. It's, it's architecture over short-term tactics And if you want my definition, a hack in this space will try to influence what the machine says about you, not what you are and what you do, but try to have that image distorted.
[00:17:03] Architectural changes change, changes what you do and how you do it and how you say it, and then based on that, the way a machine can read and what it can do on your website. The first one is purely rented. It's short-term. It might work, then the model gets updated, and it doesn't work anymore.
[00:17:23] The second one is fully owned And I want you to think about that. Just think about all those Google updates over the years, it was painful for a lot of websites, and there was a lot of complaint.
[00:17:36] And, and, and now when there's a Google update, when there's a YouTube, uh, punishing channels and demonetizing channels, yeah, yeah, dude, you knew you were doing something wrong, and now the time is up and doesn't work anymore. I'm sorry, but I'm really not sorry about that
[00:17:52] If something is a hack, it's designed to stop working. If it's an architectural structural change, it's not going to stop working. And that shift to the non-human visitors, I know it's early, I know agents are barely buying, barely doing stuff. The, the foundation for that is being set this year in, in particular.
[00:18:15] But that shift to non-human visitors is very, very, very real And that causes a lot of panic. That causes a lot of confusion. But the way to make sure you're ready when you don't know what the way to prepare is, just focus on doing all the fundamental stuff the best you can
[00:18:39] So I'm not saying all those products that help you track AI visibility are necessarily bad. I'm saying if your primary focus is being best X for Y in ChatGPT today or Perplexity or Claude or Gemini Maybe ask yourself if you really are the best X or Y. If your product is, if the world and the internet think you are, if they agree on that.
[00:19:08] And maybe that's the problem you need to fix because LLMs are really nothing but a mirror of everything happening online by design. They stole the internet, and that's what we get. Let me know what you think about this. [email protected]. I really wanna hear from you. Thank you for listening. Talk to you next week
RELATED ARTICLES
Generative UI Could Turn Google Into Your Booking.com
Google and OpenAI now both build answers as interfaces, and five months after I/O neither has shown a website's data inside one. I think the direction still holds, and the decision it would force on a business looks like the one hotels made with booking.com.
Eleven Ways Websites Can Get Paid For AI Use, And Who Sets The Price In Each
In 2025 Cloudflare let websites set the price AI crawlers pay. In 2026 it launched a scheme where the AI company sets the price and reports its own usage. Across eleven schemes that pay websites for AI use, the ones where the website names the price have no AI company named as paying, and where AI companies pay, they set the price or count the usage.
Shopify Now Enrols Your Store In Every New AI Shopping Channel
Shopify signs eligible stores up for every new AI shopping channel by default. Leave it on, and read the shopper count in your Shopify admin as clicks from AI apps, not agent purchases.
ENJOYING THIS EPISODE?
Practical strategies for making your website work for AI agents and the humans using AI to find you. Once a week you get the new articles, the latest podcast episode, and a few links worth keeping.