This is a companion to the PlannerCritic series. Article 2 was about a specific critic bug. This one is about the design principle I extracted from fixing it — and the measurement that proved it holds.
I measured my LLM critic on identical input across five trials. It returned a different verdict every single time. labelfliprate = 1.0. It also never let a defective plan through. underclaimapprovals = 0. Both are true. The frozenset is why.
In v0.2.1 I added a test I'd been avoiding: send the same boundary-case plans through the real critic model five times and measure what changes. The live-critic boundary evaluator (#218).
| Metric | Value | What it means | |--------|-------|---------------| | labelfliprate | 1.000 | The critic changes its verdict on every trial of identical input | | evidencedriftrate | 1.000 | The critic invents a different explanation every trial | | familymigrationrate | 0.000 | No seeded defect landed in an advisory family | | underclaimapprovals | 0 | No defective plan got zero blockers |
Read those first two rows carefully. On the same plan, five times in a row, the critic returned a different verdict and a different reasoning every time. If you were betting the safety contract on the critic being consistent, you'd have lost.
I wasn't betting on that. But it was still uncomfortable to see it measured at 1.0. "Mostly consistent" would have felt safer than "maximally inconsistent." Maximally inconsistent is what we got.
The critic is 100% non-deterministic on verdict. It is 0% under-claiming on seeded defects. Both numbers are real. The reason both can be true at once is that they measure different directions, and the architecture assigns them to different owners.
There are two ways an LLM critic can be wrong: Under-claim — a defective plan gets zero blockers. This is the dangerous direction: a bad plan slips through. Over-claim — a sound plan gets a blocker for "not thorough enough." This is the noisy direction: a good plan gets escalated.
The architecture gives each direction to a different authority, and neither authority is the LLM.
The deterministic gates own the under-claim direction. Preconditions, topological ordering, rollback credibility, verification ordering — these parse the plan's AST, not its prose. They cannot be prompt-injected because they don't read natural language (Article 5 covers this). A defective plan that the LLM critic happens to miss on trial 3 still gets caught by the gate that checks whether every precondition is established by an earlier task. That's why underclaimapprovals = 0 despite labelfliprate = 1.0.
A code-enforced allowlist owns the over-claim direction. This is the part I want to dwell on, because it's the part I learned the hard way.
In Article 2 I told the critic to be "an adversarial plan reviewer." It obeyed. It blocked plans for being incomplete — "this plan could also cover edge case X" — not for being unsafe. Every strict goal escalated for the wrong reason.
The fix wasn't more prompt engineering. I tried that first; the critic still escalated completeness concerns to blocker about 30% of the time. The fix was a frozenset:
Even if the LLM returns blocker for a risk or missingsteps finding, the code downgrades it before it enters the findings list. After the fix, zero advisory findings appeared as blockers across 92 post-fix runs.
That's the sentence I kept coming back to. The LLM is allowed to be wrong about severity — and it is, on every trial — because the code doesn't trust the LLM's severity label. It trusts the structural property (which family the finding is in), which the gate derived deterministically. The LLM's label is decorative; the family is load-bearing.
When you put LLM judgment on a critical path, you inherit every vulnerability of LLM judgment — non-determinism, prompt sensitivity, the tendency to be "thorough" when you asked it to be "adversarial." When you keep the critical path deterministic, you get resistance by design — but only to the things your code can check structurally.
The split that worked: Code is authoritative everywhere it can be. Ordering, preconditions, rollback presence, schema. These have a right answer the AST can verify. The LLM is advisory everywhere it could be wrong. Severity calibration, completeness, "is this rollback credible." These are judgment calls where the LLM adds value but can't be trusted to be consistent.
The mistake is putting the LLM in charge of the second category and making its verdict load-bearing. The fix is letting the LLM inform the second category while code decides whether its verdict counts. The frozenset is the mechanism: the LLM proposes a severity; code checks whether the structural property supports it; code wins ties.