I spent a day measuring how AI crawlers actually see websites, and two things surprised me enough to write down. Your analytics almost certainly cannot see AI crawlers at all
Our site's analytics beacon is client-side — a script tag that fires on page load and POSTs to /api/hit. Completely normal setup. It's what most sites do.
So every property we own had recorded exactly zero crawler traffic for its entire life. Not "low" — structurally zero, because the measurement instrument required the one thing crawlers don't do. Every question of the form "is any AI actually reading us?" was unanswerable in either direction, and we'd never noticed, because the dashboard showed a number rather than an error.
If you're on Cloudflare Pages, the fix is about fifteen lines in functions/_middleware.js, which runs on every request including ones that never execute a line of JS:
The kind field is the part that matters, and it's the thing I'd argue about if you only take one idea from this post.
A training crawl and a retrieval fetch are not the same event and must never be summed. GPTBot is building a corpus — there is no link in a training corpus, so that visit can never send you a human. ChatGPT-User and Perplexity-User mean somebody asked an assistant a question just now and it went to read your page to answer them. Only the second kind can ever become a visitor.
The commonly cited figure is that the large majority of AI crawling is training rather than retrieval. I haven't verified that split myself — but the argument doesn't depend on the ratio, only on the fact that the two events mean different things. If you log them as one number, the signal you can act on is mixed into one you can't.
Two more things worth doing while you're in there: skip asset requests (a bot pulling your CSS tells you nothing about whether it read the page), and wrap the whole thing so a logging failure can never break a response. Instrumentation that can take down a page is worse than no instrumentation. The llms.txt advice going around does not survive contact with server logs
There's a lot of confident writing about llms.txt as the cheap win for AI visibility. I believed it enough to weight it heavily in a site auditor we built.
Then I went looking for the evidence that anything actually requests it, and could not find a primary source I was able to read myself. What circulates is a widely-repeated claim, sourced to a large crawler-log analysis, that the overwhelming majority of published llms.txt files are never requested at all.
I am not going to hand you that number as though I verified it, because I didn't. I tried and failed. What I can tell you is the decision we made under that uncertainty, and why I think it's the right-shaped bet: we cut llms.txt from 15 points to 2 in our auditor, on the reasoning that a file nobody has demonstrated is being fetched cannot be the thing deciding whether you get cited. If someone has server logs showing otherwise, I'd genuinely like to see them — that's a measurement I can't take from outside.
The logger from part 1 is now running on our own site partly to answer this firsthand, which is the honest way to settle it.
Here's the part I can stand behind, because it follows from how fetching works rather than from a statistic: what decides whether an assistant can quote you is how much text exists in your HTML before any JavaScript runs.
An assistant fetcher takes the raw response. It does not hydrate your app. If your content arrives via client-side rendering, you are invisible to it no matter how permissive your robots.txt is — being allowed in is worthless if there's nothing to read once you're there.
Compare that to the total byte count. If you're getting a few hundred characters of text out of a few hundred kilobytes of markup, an assistant sees roughly nothing — and no amount of llms.txt changes that.
Our auditor now scores server-rendered text at 33 points out of 110 and llms.txt at 2. It disagrees with most of the advice in its category, and prints the reasoning in its own output so anyone can argue with it rather than having to trust the score.
All of this is about what crawlers fetch, which is upstream of what models cite. Fetching is necessary, not sufficient. I have no citation data, and I haven't seen anyone else's either — which is worth remembering every time someone tells you confidently what gets you into AI answers. Treat "server-render your content" as removing a hard blocker, not as a growth tactic.
And treat this post the same way: one of the two findings I measured directly, and the other I explicitly could not.