A six-step check for ChatGPT and other AI-assistant traffic, built from the numbers founders shared under my last post.
A founder who commented on my last post had a funnel that looked broken. Over seven days their analytics showed 206 sign-in starts and 0 completed signups. Numbers like that send you straight into the auth code.
The code was fine. Of 205 clicks on the Google sign-in button in four days, about 190 came from Germany. Each one was a different visitor. Many reported browser and OS pairs that don't exist in practice, like Safari on Linux. Once the founder excluded that traffic, 7-day visitors dropped from 358 to roughly 85, and sign-in starts dropped from 206 to 1. That single remaining start was their own test.
Nothing in the funnel was broken. The visitors weren't real, and any conversion rate calculated that week would have had a fake denominator.
My last post argued that AI referrals should be judged by what they turn into. Several readers answered with their own data, which was more useful than anything I wrote. Here it is as a method you can set up in an afternoon and leave running for a month.
Do this before you look at any source. If junk is in your totals, every rate you compute inherits it.
What gave it away in the case above: One country suddenly owning one event. 190 of 205 clicks from a single country is a script, not a market. Every session is a new visitor, with no returns. Browser and OS pairs that don't ship. Apple doesn't make Safari for Linux, so "Safari on Linux" in your data is a user-agent string someone typed.
Exclude those, note what you excluded and why, and keep the before and after numbers.
AI crawlers like GPTBot or PerplexityBot are a separate issue. They mostly don't run your analytics script, so count them in server logs as crawl, never as visitors.
Another founder shared a 30-day split from their product. AI-assistant visitors signed up at 33.7%. Google organic visitors signed up at 24.0%. That's a real gap, and the kind of number I hoped readers would find.
The detail I'd pay more attention to: 877 of their 936 ChatGPT visits arrived with an empty referrer. They were only identifiable as ChatGPT because the links carried utmsource=chatgpt.com. Group by referrer alone and ChatGPT shrinks to 59 visits at most, with the other 877 sitting in Direct.
So build your AI segment on both fields. In GA4, Plausible, PostHog or Fathom, save one filter where source, referrer or utmsource contains chatgpt, perplexity, gemini, copilot or claude. Then open five of those sessions and check that they look like people: a landing page, some time on site, an event. A value in the source column is a hint. It isn't proof.
One gap you can't close here. Google counts clicks from AI Overviews and AI Mode inside the normal "Web" search type in Search Console, so they can't be pulled out. Log them as Google.
A third commenter gave me the cleanest test I've seen. Choose one action a visitor can take in their first session, such as an email capture, a preview open or a checkout start. Only one. Then watch it next to your AI referral count.
If AI referrals rise and that action stays flat, your answer is no. Those visitors aren't doing anything yet, however good the traffic chart looks.
The same commenter added a check I now run first: look at which page ChatGPT actually links to. It's often not the page you would pick. If the landing page doesn't make the next step obvious, a flat action count may be telling you about the page rather than the visitor.
One action, not five: with small numbers, five metrics give you five chances to mistake noise for a win.
A conversion rate on its own says very little. 33.7% looks strong next to 24.0%. Next to 45% it would look ordinary.