I built one engine where an LLM reviews another LLM's plan. I built another where two LLMs debate a pull request.
Each one failed in the way the other was designed to prevent. That's the only reason I trust either of them.
PlannerCritic runs a draft → critique → revise loop: one model writes a plan, another critiques it, and a bounded loop revises until it converges or escalates to a human.
My first instinct was the obvious one: tell the critic to be adversarial. That's a prompt fix for what I thought was a behavior problem.
It backfired. The critic started blocking plans for being "not thorough enough," which is a useless verdict. I had to move the severity contract out of the prompt and into a frozenset in code, where it couldn't drift.
Here's what the field test showed on 170 goals for $0.49: Critic labelfliprate: 1.0 — on identical input, it changed its verdict every single trial. Critic evidencedriftrate: 1.0 — its justification moved too. underclaimapprovals: 0 — it never knowingly let a seeded defect through. familymigration_rate: 0.0. True failures: 0.
A maximally non-deterministic critic was completely safe. Not because it was reliable — because the deterministic gates owned the under-claim direction, and the critic was never allowed to be the safety boundary. The numbers are in the v0.2.1 field test results.
AdversarialDebate does the opposite: two isolated models review the same PR and either converge on a verdict or preserve their disagreement.
The failure here was subtler. I asked the second model to be independent in the prompt. It looked like it worked. Then I read the raw logs: the "debate" was a stored replay, echoing text the first model had already generated.
I tore out the prompt-level independence and enforced it mechanically — isolated contexts, no shared conclusion, evidence required per claim. Theater went from 89% to 0% across 217 debates, and on 411 debates against 70 real public PRs the system matched human review claims 81% of the time. Details in the v0.2.2 field test report.
| System | Structural problem | Prompt attempt | What actually fixed it | |--------|--------------------|----------------|-----------------------| | PlannerCritic | Unreliable severity contract | "Be adversarial" | Severity contract in code (frozenset) + deterministic gates | | AdversarialDebate | Fake independence | "Be independent" | Mechanical isolation invariant |
Both times, the default instinct was to solve a structural problem with a prompt. Both times, the prompt was the wrong layer. The model was never the safety boundary — the architecture was.
This is a sample size of two, and both systems are mine, so take the convergence as suggestive, not proven.
But it's suggestive in a useful direction: if you can't make an LLM reliable, stop trying to. Make it irrelevant to safety. Let the nondeterministic part be as wild as it wants, and put the non-negotiable decisions in deterministic code that no prompt can override.
The architectures aren't complete. PlannerCritic catches structural plan defects but not subtle logic errors. AdversarialDebate still has an unexplained 11.3% miss rate in narrative domains, and I can't fully account for it.
Nor is "two systems converged" proof of a universal law. It's two data points from one engineer. What I can defend is the direction: the failures were structural, and so were the fixes.
Where does the safety boundary live in your agent — in the prompt or in the code? I've been wrong about that in both directions.
Repos and receipts: planner-critic-engine · adversarial-debate · PlannerCritic failure-mode register — all MIT, all public.