When an AI crawler gets a 403 response, find the layer that refused the request before changing crawler rules. A browser loading the page successfully does not establish that the crawler can fetch it. An allowed robots.txt entry does not explain which security rule handled a later request.
Start with a real request record, its time and the exact URL. Check the edge security event, then the origin response where available. Keep crawler identity separate from a user-agent string someone can copy. The aim is a narrow diagnosis, with the site's intended access policy kept intact.
This guide uses a fictional documentation site to explain the investigation. It does not ask you to disable protections, open private routes or change every bot policy. The examples are teaching scenarios, not tests of HoneyWeRank or reports from Uriel's infrastructure.
“Claude cannot read our docs” is a starting report, not a complete diagnosis. Which Claude product and mode was used? Was web search active? Did a real request reach your edge? Was the failed address the document itself, a redirect target or a resource needed to display it?
For the fictional site, record a public guide URL, a UTC time and an observed 403. Save the request identifier if the service exposes one. Also record the response body and which system produced it, without copying private headers or credentials into a public issue.
A successful human session may include cookies or a previous challenge result. A crawler has a different request context. Compare those contexts instead of treating the human result as proof that the access report must be wrong.
Use a bounded observation period. A request from last month may have encountered a rule that no longer exists. An event recorded after a deploy may reflect a new configuration. Keep the configuration version or change time beside the evidence so the investigation remains reproducible.
If a request was stopped at the edge, an application log may contain no corresponding entry. That absence alone does not prove which edge product stopped it. Look for the matching security event using the recorded time, hostname, path and request identifier.
Cloudflare's AI Crawl Control documentation says unsuccessful crawler requests may result from different rules or response errors. Its table is therefore a starting view, not a complete explanation that the AI blocking action caused every failure.
Inspect the event's action and rule details. Write a short finding such as “The request matched this custom rule and received the configured denial.” That is more useful than “Cloudflare blocks AI,” which turns one observed configuration into a broad claim.
If the edge passed the request onward, continue at the origin. Compare the origin access log and any application response. A route guard, hosting rule or upstream service may be responsible. Keep a category for insufficient evidence rather than assigning the failure to whichever dashboard looks most familiar.
A user-agent string is a claim made in a request. A local request with a crawler's name can be useful for testing how your own matching rule behaves, but it does not prove that the request came from that crawler's operator.
Cloudflare documents that crawler detection on its free plan uses user-agent strings, with more thorough detection available through Bot Management. Do not describe a name match as verified identity when the evidence only supports that simpler classification.
For OpenAI search access, OpenAI's current search guidance points site owners to OAI-SearchBot and published searchbot IP addresses. Check that current provider evidence when investigating an actual OpenAI request. Do not reuse another provider's address list or assume that every agent from one company has the same purpose.
Keep the investigation record explicit: observed user agent, identity evidence checked and any remaining uncertainty. You can establish that your rule rejects a string without establishing that a real search crawler was rejected. Those findings call for different next steps.
Before proposing a fix, ask the owner which public content they intend to make available for search or user-directed retrieval. That decision is separate from training access. It is also separate from whether a particular request was handled correctly.
Suppose the fictional site intends to allow a relevant search agent to read public guides. An event shows that a broad custom rule denied a matching request. The next task is to review that rule against the intended policy, not to turn off all bot controls.
Check other public paths and keep private ones outside the investigation's release scope. A diagnosis about a public guide gives no reason to expose account pages, internal previews or authenticated material. The fact that a crawler is useful for discovery does not make it entitled to every route.