If you have Cloudflare in front of your site, and at some point you wrote a cache rule that matches bots, go and read that rule again. I wrote one to be helpful to AI crawlers. What it actually did was store my error responses at the edge and replay them to every AI crawler for an hour at a time.
I found out because OpenAI's crawler could not read my site. GPTBot was getting HTTP 429. Amazonbot and Bytespider were getting 403. I run taghazout.io alone, so there was nobody to hand the problem to. I did what most people do first: I looked at the response headers.
LiteSpeed is the web server my host runs. I read that header, decided the origin was producing the 429, and opened a ticket asking about rate limiting on their side. It felt like a closed case. The server had signed its name on the error.
The flaw took me too long to see. A response served from a CDN cache is a stored copy of an origin response, headers included. If Cloudflare stored a 429 an hour ago, it hands you that copy later, x-turbo-charged-by: LiteSpeed and all. That header tells you which server generated the body at some point in the past. It tells you nothing about whether this response, the one in my terminal right then, came from the origin or from a cache in another city.
I had used a header as evidence for a claim it cannot make. It was not lying to me on purpose — server-identifying headers get copied along with everything else, and they will point you at an innocent party by accident.
Namecheap support came back and said the domain is proxied through Cloudflare, that a "Cache Everything" style rule can cause 429 or 403 responses to be cached at the edge, and that this looked more like a stale or corrupted cached response at the CDN than something LiteSpeed generated.
They were right and I was wrong. The easy version of this story is "host blames the CDN, developer is vindicated". That is not what happened. I arrived with a bad theory and their support gave me a better one.
If cf-cache-status says HIT, you are holding a stored copy and the status code may be older than the problem you are debugging. If it says BYPASS or MISS, Cloudflare went to the origin and what you see is live. Look the other values up rather than guessing: EXPIRED does not mean what it sounds like — the object was in cache, had gone stale, and was served from the origin anyway. A non-zero age is a second hint; cf-cache-status answers directly.
Use -D - -o /dev/null, not curl -I: -I sends a HEAD request, and HEAD can be routed and cached differently from the GET a real crawler sends. Measure the request the crawler actually makes.
The rule was mine. In Cloudflare's cache rules sat one I had written, ai-crawlers-edge-cache. It matched 13 AI crawler user agents and set Edge TTL to "Ignore cache-control header and use this TTL: 1 hour". My origin sends no-store on HTML. The rule overrode it.
The trap has a simple shape: Your origin returns an error to one bot. A rate limit, a security block, one bad minute. Your edge rule says "ignore the origin's cache-control, use this TTL instead". Cloudflare stores that error the same way it would store a page. Until the TTL expires, every request matching the same cache key gets the error back, including from crawlers that never did anything wrong.
Caching a 200 is the point of the rule. Caching a 429 is the bug. Nothing in the dashboard separates the two for you.
I had built a machine for turning one bad response into an hour of bad responses, and I had built it on purpose, for exactly the bots the rule was written to help.
What changed: ai-crawlers-edge-cache Edge TTL: Ignore cache-control header → Respect origin headers A full cache purge afterwards, because the poisoned entries survive the rule change Verification is now per-crawler, not "AI crawlers are fine"
ClaudeBot, OAI-SearchBot, PerplexityBot and Googlebot all went from 429 or 403 to 200. Amazonbot and Bytespider did not move: both still return 403, and on a cache miss, which means those are coming from the origin too. At the time I read that as leftover noise. It was the second cause, already visible, and I did not look at it.
So I ran an isolation test built to separate three explanations: a cached error, a flat block, and a rate limit. Ten distinct URLs — distinct matters, or you are testing your own cache — six seconds apart.
First request: 200. The next nine: 429 — roughly one successful request per minute, which is what the timestamps are there to show. The same URLs with a normal Chrome user agent returned 200 every time.
Then the same URL again, six user agents back to back. Running them concurrently is tighter, and it is how I would do it now: