Most teams trying to understand AI search start by querying ChatGPT. There's a cheaper signal sitting on your servers already: your access logs.
When ChatGPT, Claude or Perplexity browse the web to answer a user's question, they fetch your page with an identifiable user agent. Each of those hits means a live conversation pulled in that URL, which makes it about the closest thing you have to a server-side "citation candidate" log.
This post shows how to pull those hits out of nginx logs, tell the different kinds of AI bot apart, check they're genuine, and turn them into a list of pages AI assistants actually read.
AI companies run several crawlers with different jobs. Lumping them together hides the useful signal.
| Purpose | Examples | What a hit means | |---|---|---| | Training | GPTBot, ClaudeBot, Google-Extended, CCBot | Content may go into a future model. No user is waiting. | | Search indexing | OAI-SearchBot, Claude-SearchBot, PerplexityBot | Your page is being indexed for the engine's search feature. | | User-triggered fetch | ChatGPT-User, Claude-User, Perplexity-User | A person's conversation fetched this page right now. |
\ Google-Extended is a robots.txt control token, not a separate crawler. You won't see it as a user agent in your logs. Google fetches with its normal crawlers.
The third row is the interesting one. A ChatGPT-User hit on /pricing means someone asked ChatGPT a question and it opened your pricing page to answer it.
User agent strings change, so check the vendors' own bot documentation pages (OpenAI, Anthropic and Perplexity each publish one) for the current list.
(Illustrative numbers.) That shape is typical: training crawlers dominate the volume, and user-triggered fetches are a small but far more meaningful slice.
Over a few weeks of logs, this gives you a ranked list of the pages AI assistants open on behalf of users. Compare it with your Google top pages. The two lists are rarely the same. Docs pages, pricing pages and comparison pages are often overrepresented in AI fetches relative to their search traffic.
Any 403 or 429 in that output means a live conversation tried to read your page and you turned it away. That's usually a WAF or rate-limit rule that nobody set up with AI agents in mind.
User agents are trivially spoofed, and scrapers love pretending to be GPTBot. Before you trust the numbers (or allowlist anything), verify the source IPs.
OpenAI and Perplexity publish JSON files of the IP ranges their bots use, linked from their bot documentation. Here's a verifier that loads those ranges and checks log IPs against them:
The ranges files use a prefixes list with ipv4Prefix/ipv6Prefix keys, the same format Google uses for Googlebot. If a vendor doesn't publish ranges, fall back to reverse DNS plus forward-confirm, where the vendor supports it.
grep works for a quick look. For ongoing tracking, tag AI traffic at the edge so it's queryable without regex archaeology:
The if= parameter writes only requests where $ai_bot is non-empty, so ai.log stays small and tab-separated. Ship it to whatever you already use (Loki, BigQuery, a cron job that loads it into SQLite).
On Cloudflare or another CDN, your origin logs may miss requests the edge served from cache or blocked outright. Use the CDN's own logs or AI-bot analytics for the complete picture.
Server logs show that an assistant read a page. They don't show: what the user asked, whether the answer cited you or used a competitor instead, what the assistant said about you, anything from answers built from training data or a search index without a live fetch.