When I shipped v0.2.1 of PlannerCritic, I thought the hard part was over. The engine had survived a 170-goal field test across 40 domains. The published results looked strong: 73 of 73 balanced goals approved, 96 of 97 strict goals escalated, 8 of 8 adversarial goals blocked, 30 verdict deltas all attributable, and zero new engine issues discovered by the sweep.
Then a stranger on the internet checked it in public and found that the engine was mostly right, but parts of the story I wrote about it were wrong.
After I published the field-test article, a reader offered something unusual. He did not just leave feedback or point out one suspicious sentence. He proposed a protocol: freeze a small set of claims from the release before inspection, verify them only from public artifacts, and record any divergence as a divergence instead of explaining it away.
That is a much stricter process than most OSS releases get. In practice, many of us release from a mix of memory, local state, CI output, and confidence. We know the code. We know roughly what passed. We know what we meant. What we usually do not do is force our public claims to survive independent reconstruction from the outside.
We froze five claims from the v0.2.1 release: the repaired adapter-import test really exercised imports, approvingauthority was test-proven but not reachable from shipped surfaces, 1295 deterministic tests passed, the field test had zero true failures, the 30 verdict deltas were all attributable.
The important detail here is not the number five. The important detail is that the claims were frozen before inspection. That removed the easiest escape hatch: adjusting the claim after seeing the evidence.
The first failure was the easiest to explain and the most annoying to defend. I said 1295 deterministic tests passed. The public CI run on the release commit showed 1294 passed, 1 failed, and 14 skipped. The failed test was a flaky SQLite concurrency case. Operationally, that meant the release was still understandable. But documentation-wise, it meant my claim was cleaner than the public evidence supported. That discrepancy became #263.
The second failure was worse because it was not a flaky detail. It was a contradiction in the release document itself. In one place, my v0.2.1 field-test report said "zero true failures." Two tables down, Scorecard B recorded True Fail = 1. I had written both statements. I had published both statements. I had not noticed that they were incompatible. That became #246.
The third failure was smaller but revealing. The release documentation said planoscillationdetected fired for 3 goals. The actual stored artifacts showed 5. That became #247.
None of those findings meant the engine was broken. What they meant is more uncomfortable: the engine can be right while the release around it is wrong.
The underlying PlannerCritic results from v0.2.1 were still solid. The field test really did find zero new engine issues. The 30 verdict deltas really were attributable. The inherited corpus behavior really did hold. The live boundary evaluator really did return labelfliprate=1.0, evidencedriftrate=1.0, familymigrationrate=0.0, and underclaimapprovals=0.
The problem was not that the system was wrong. The problem was that the release package around the system had drifted out of sync with the evidence. Prose and scorecards diverged. Counts were copied across documents without a final reconciliation pass. My local confidence ran ahead of the public artifact chain.
That is not a glamorous bug class, but it is exactly the kind of bug that undermines trust in serious infrastructure projects. A release note is part of the product surface. If it overstates what the evidence says, even slightly, readers stop trusting the stronger claims too.
The public audit did not just embarrass me. It materially shaped the next release.
v0.2.2 started by fixing the exact class of mistakes the audit exposed. #246 fixed the “zero true failures” contradiction. #247 reconciled the oscillation count. #248 removed 75 committed .py,cover artifacts from src/. #263 reconciled the 1294/1295 test-count discrepancy. #264 added a failure-origin taxonomy so defects could be tied to the layer that should have caught them first.
By the time v0.2.2 closed, the evidence chain was much tighter. The release finished with 183 of 183 goals complete across 43 domains, 1347 tests passed, 15 skipped, 91% coverage, and the inherited top-level contract still intact: 73/73 balanced approved, 96/97 strict escalated, 8/8 inherited adversarial blocked. The release also added 13 new security fixtures, found a real boundary regression on the first run, fixed it, and reran the benchmark until the published result was clean.
The engine improved, but the bigger improvement was procedural: the release got harder to fool.
What stayed with me was not just that a reader found discrepancies. It was that the verification process itself was better than the one I had implicitly been using.