On 28 August 2026 I audited 38 live pages and fixed some ordinary defects. The two things worth writing down both came from my own tooling being broken. One script reported three pages I had just fixed as still broken, because it did not recognise a relative link. One date parser had thrown an exception on every crawl-statistics read since it was written, so the report that answers "is the crawler arriving at all" had never once returned a number.
This is a note about measurement rather than about search engines. The setting is an indexing problem on one of my hosts, but the transferable part is that I spent a day acting on numbers produced by instruments I had never checked, and both were wrong — one in each direction. One made a working fix look broken. One made a working crawler invisible.
Everything below depends on this, and most writing on the subject collapses it into one word.
Crawling is retrieval. A crawler requests a URL and receives bytes. That is all it means. A page can be crawled every day for a year and appear in no search result ever.
Indexing is selection. Having retrieved the page, the engine decides whether to keep it in a form it will serve. That is a judgement, each engine makes it independently, and none of them will tell you the reasoning.
So there are at least three states worth separating: never fetched, fetched and declined, and fetched and kept. "My pages aren't indexed" describes the middle two identically, and their causes have nothing in common.
On 28 August 2026 I fetched every URL in the sitemap — 38 pages on the affected host, 20 on my notebook site — over plain HTTP with no JavaScript executed, and produced a number per page rather than a verdict. That rule exists because an earlier audit returned "all clean" for months while the homepage was handing crawlers a fraction of its navigation in a form they could use.
It found real defects. Five pages had zero followable links: a crawler arriving there could go nowhere. The worst had two navigation controls, both — usable with a mouse, invisible to a crawler. Three more bound their only navigation to a JavaScript click handler, which has the same effect.
| 38 pages, fetched live | Before 28 Aug | After 28 Aug | |---|---|---| | Pages with zero followable links | 5 | 0 | | Navigation a crawler cannot follow | 2 | 0 | | Pages with an incorrect h1 count | 1 | 0 | | Missing robots directive | 1 | 0 | | Canonical problems | 0 | 0 | | Images with no alt attribute (of 199) | 0 | 0 | | Non-200 responses | 0 | 0 |
After deploying, I re-ran the audit against the live site. It reported that three of the pages I had just fixed still had zero followable links.
They did not. I fetched one by hand and the anchor was plainly in the served HTML.
The bug was mine. The script counted links by matching href values that started with / or http. The anchors I had added were relative — href="./" — so they matched nothing and counted as zero.
What makes this worth writing down is the shape of the failure. A measurement that silently under-reports is indistinguishable from the defect it is supposed to detect. Every downstream step would have been reasonable and wrong: conclude the deploy failed, hunt for a caching problem that did not exist, possibly "fix" three pages that were already correct and introduce a real defect doing it. Nothing in the output looked like an error. It looked like bad news.
Bing's webmaster API serialises dates in a Microsoft-era format — /Date(1784190745000)/, epoch milliseconds wrapped in a string. My parser handled that correctly. But one endpoint, and as far as I can tell only one, returns a second variant carrying a UTC offset:
My pattern captured "digits and minus signs". It swallowed the offset into the number, and the integer conversion raised an exception every time.
Because that variant appears in exactly one report, the failure was invisible everywhere else. Every other call worked. And the one report it broke was crawl statistics — the nearest available substitute for origin access logs, and the first thing to check when asking whether a crawler is showing up.
It had never once returned a value. I had been reasoning about crawler behaviour for weeks with the most direct instrument for it throwing an exception on every call. The fix is one line; the test asserts that both offset signs resolve to the same instant, because the value before the offset is already UTC.
| Date | Pages crawled that day (Bing) | Pages held in index (Bing) | |---|---|---| | 2026-08-24 | 15 | 33 | | 2026-08-25 | 27 | 33 | | 2026-08-26 | 36 | 34 | | 2026-08-27 | 23 | 34 |