My host keeps access logs in a control panel I have to click through, and nowhere a script can read them. So after September's compromise I could not answer the one question that mattered — are the crawlers coming back, and what are they being served? — except by waiting two days for Search Console to round it into a weekly shape.
So I gave the site its own log. About 120 lines of PHP: if the user agent names a known crawler, append one line at shutdown with the time, the crawler, the status code it was actually served, the path and the IP. Humans are never written to it.
By the end of the day: 24 of 26 crawler requests were to URLs a cloaker invented three weeks earlier.
Those paths are not random junk. They are the exact shape the malware generated — a dictionary word, a slash, a second word with digits pushed between its letters — and it handed about 291 fresh ones to every crawler that asked for a page, for six days, before the site was cleaned. The malware has been gone for weeks. The queue it created has not.
Every one of those URLs was returning 404, which is correct in the sense that they are not there. But 404 means not here right now, and crawlers treat it as a reason to come back and check again later. That is why they are still working through a list from three weeks ago.
410 means gone; stop asking. Both Google and Bing drop a 410 faster than a 404, and the difference is exactly the behaviour I want.
The risk in a rule like this is obvious: match too broadly and you 410 something real. So I wrote the shape, then measured it before it went anywhere near the site:
One more precaution, and it is the part I would get wrong if I were rushing. The rule does not run at the start of routing. It runs at the end, inside the 404 handler — the place a request only reaches once it has failed to match anything real:
That ordering is not decoration. My site has a /s/{code} short-link route: two segments, no hyphen, letters and digits — it matches the shape. Put the rule first and every short link a user shares becomes 410 Gone. Put it last, and a live route can never be caught by it, because a live route never reaches the 404 handler at all.
One line said googlebot 200 /. I was pleased for about ten seconds, until I looked at the IP: 39.34.131.166 — my own network. It was me, testing with a spoofed user agent earlier that afternoon. The log had recorded exactly what it was told, which is all a user agent ever is: a claim anybody can type.
So the log now checks the claim against the crawlers' own published address lists — Google's googlebot.json and special-crawlers.json, Bing's bingbot.json, cached for a day — and reports both numbers:
And the alarm that says "Googlebot has not come" now counts verified visits only, because an alarm that can be silenced by anyone typing Googlebot into a user agent string is not an alarm.
If you are running this kind of check yourself: your crawler's status codes are what it believes about your site, so log what was served, not what was requested. And if you end up needing to retire a batch of URLs permanently, an .htaccess rule is the cheaper place to do it than in application code — as long as you have measured the pattern against your real URLs first.
Three weeks after the cleanup, most of my crawl budget was still being spent on the attacker's imagination. I only know that because the site finally started writing things down.