97% of llms.txt Files Get Zero Requests. But the Biggest Platforms Disagree On Why.

Over a single month this spring, across 137,210 domains, 97% of the llms.txt files sitting on the open web drew exactly zero requests. Not one AI crawler, not one human, not one script. That's the finding Ahrefs analysts Louise Linehan and Xibeijia Guan pulled out of a month of raw server logs, and Search Engine Journal reported the same headline number within days. It lands on a tactic the GEO trade has spent a year selling as table stakes for getting cited by AI.

It gets worse the moment you read which bots actually showed up. Slackbot, Slack's link-preview bot, fetched the file more often than PerplexityBot did, for a format supposedly built to feed AI search engines. Anthropic's Claude-Code beat every dedicated AI retrieval bot. SEO audit tools, not AI systems, were the single largest source of the few fetches that happened at all. And the biggest research crawler in the whole dataset identifies itself as "prompt-injection-survey," Linehan wrote on LinkedIn, which means the most active party touching these files is studying how to poison them. Most AI crawlers don't even probe for an llms.txt that isn't already there.

Half the trade will read this as GEO's death certificate, proof that llms.txt is snake oil and optimizing for machines is a mug's game. That read is wrong, and it costs you twice over. What the data killed is one config-file ceremony the machines never requested. The instinct underneath it, making your content legible to the systems now sitting between you and your readers, survives intact.

How a Developer File Became a Tactic

llms.txt never started as a marketing play. Jeremy Howard, who co-founded Answer.AI and fast.ai, proposed it in September 2024 as a plain Markdown file at a site's root, a way to hand a language model clean, link-rich text instead of forcing it to wade through HTML, navigation, ads, and JavaScript. The problem he was solving was small context windows and messy source markup, the kind of thing that trips a coding assistant reading a documentation site. It was never ratified by the IETF or the W3C, and no major lab, not OpenAI, Anthropic, Google, or Perplexity, ever confirmed its crawlers consume the file in any normative way. Somewhere in the two years since, the GEO and AEO trade rebranded a developer convenience as a visibility growth tactic, and about 28% of the domains Ahrefs checked now ship a valid one. The file is everywhere now, and almost nothing reads it.

The Platform-Interest Clash

The same stretch of the calendar sharpened the real argument, because the two biggest platforms staked opposite public positions on whether any of this matters. In May, Google published its first official guide to optimizing for generative-AI features and told site owners to skip the entire GEO checklist, calling llms.txt, content chunking, AI-specific rewriting, and special schema markup unnecessary. "Optimizing for generative AI search is optimizing for the search experience, and thus still SEO," the guide reads. Weeks later, in June, Microsoft did the reverse. It shipped Web IQ, a set of AI-native grounding APIs built to feed evidence to AI agents rather than rank pages for humans, layered on top of the AI Performance dashboard it had already added to Bing Webmaster Tools in February to track how often a site gets cited in AI answers. One platform tells you not to optimize for AI and just do SEO. The other is building a commercial stack to sell you that exact optimization.

Read that split for what it is, a fight over platform interest rather than a settled technical fact. Google steering you off GEO keeps you inside Google's model of the web, where it already writes the rules. Microsoft turning GEO into grounding APIs and a citation dashboard needs the discipline to be real, measurable, and billable. Both positions can be self-serving and one can still be correct, which is why the reading worth trusting comes from the party with no stake in the platform war. Ahrefs doesn't sell a search engine here, and its logs say the file goes unread. Ryan Law, who runs content marketing at Ahrefs, amplified the finding rather than soften it. Even Search Atlas, which sells AEO tooling, ran a smaller correlation study across 347 domains and found that sites with llms.txt were no more likely to be cited by LLMs than sites without it. When the people paid to prove it works come up empty, that's the answer.

Where the Fetches Actually Go

Stop paying rent on a file the engines ignore and move the work to where the fetch record actually points: the pages LLMs retrieve, the brand mentions they repeat back, and the citations you can watch land in your own logs. Read those logs the way Ahrefs read its own and the ground truth is right there. The one honest reason left to ship an llms.txt is a narrow one, that a platform which actually sends you clients turns around and asks for it.

Short of that, a file the crawlers never fetch isn't an AI-visibility strategy, it's a tidy answer to a question no engine ever asked.