The llms.txt file was supposed to be the bridge between your website and AI search engines. A markdown file at your root, a curated list of your best pages, a clean map for models that struggle with bloated HTML. Add it, and ChatGPT, Perplexity, and Google AI Overviews would finally find you.
I wanted to believe it. I built an open-source GEO audit tool that checks for llms.txt among its signals. Then the data came in, and it doesn't say what the checklist articles say.
In May 2026, Google published its official guide on optimizing for generative AI features in Search. The guide has a mythbusting section. This is what it says about llms.txt and similar files:
You don't need to create new machine readable files, AI text files, markup, or Markdown to appear in generative AI search.
Google's John Mueller had been saying it for over a year before it landed in the documentation. In April 2025, answering a question on Reddit, he put it plainly:
"AFAIK none of the AI services have said they're using LLMs.TXT (and you can tell when you look at your server logs that they don't even check for it). To me, it's comparable to the keywords meta tag — this is what a site-owner claims their site is about … (Is the site really like that? well, you can check it. At that point, why not just check the site directly?)"
The keywords meta tag. The canonical example of a signal search engines learned to ignore because every site claimed to be the best one.
Ten days before the Search guide told everyone to stop bothering, Google's Chrome team shipped an llms.txt check inside Lighthouse as part of its experimental Agentic Browsing audits. The documentation, last updated May 5, 2026, says:
Without this file, agents may spend more time crawling the site to understand its high-level structure and primary content.
Two teams inside the same company, pointing in opposite directions. When pressed on the contradiction, Mueller framed it as: llms.txt is "not done for search." He called it a "temporary crutch, perhaps to save some tokens" for AI coding tools parsing developer documentation. Not something non-developer sites need.
Ahrefs published the largest access study to date on June 15, 2026. They looked at every domain in Ahrefs Web Analytics that received traffic in May 2026 and checked each root for an llms.txt file returning HTTP 200.
The numbers: 28% of the 137,210 domains publish an llms.txt file (about 38,000 sites). 97% of those files received zero traffic in May 2026. Nothing fetched them at all. Not AI bots, not generic crawlers, not humans. 96% of the requests that did reach llms.txt files came from bots. 19.5% of those bot fetches came from named AI tools. GPTBot was first, Claude-Code second. Both are training crawlers and coding agents, not answer engines. Zero requests came from AI bots for llms.txt files that didn't exist. AI bots never go looking.
That last point is the one that kills the mental model where a model "checks for" your file before answering a question about you. It doesn't. It fetches whatever URLs its retrieval step already surfaced, which is normally your HTML.
Ahrefs also found something unsettling: the largest single research crawler in the dataset identified itself as prompt-injection-survey/1.0. Someone is systematically studying llms.txt as a prompt injection vector, because AI agents are designed to ingest and trust whatever the file says.
SE Ranking ran a separate analysis of nearly 300,000 domains. They tested whether having an llms.txt file correlates with how often a domain gets cited by AI systems.
They used both statistical correlation (Spearman) and machine learning (XGBoost with SHAP analysis). The result: removing the llms.txt variable from the model improved its prediction accuracy. The file wasn't a weak signal. It was noise.
Adoption was 10.13% across their dataset, roughly 1 in 10 sites. No major AI platform has publicly committed to reading it.
OtterlyAI ran a controlled experiment over 90 days. They placed a correctly implemented llms.txt file at the root of a test website and monitored AI bot traffic. 62,100+ total AI bot visits to the site over 90 days. 84 of those visits targeted /llms.txt. That's 0.1% of AI bot traffic. The site's average content page received ~265 AI bot visits in the same window. The dedicated AI file performed 3x worse than a random page.