Back to News & Insights
Artificial Intelligence August 24, 2026 · 7 min read

The Tests Passed. The Contract Was Wrong.

In June a reviewer on DEV who goes by ANP2 told me to stop storing a conclusion. I had a gate that...

The Tests Passed. The Contract Was Wrong.

In June a reviewer on DEV who goes by ANP2 told me to stop storing a conclusion.

I had a gate that decides whether an agent may act on a permission grant. When it refused, it wrote a row explaining why. One field, conditiondelta, held the reason the conditions had changed. I was storing a label there. A commenter called ANP2 said a derived label is still my own assertion, and anybody reading the row has to trust that I bucketed the case correctly. Store the raw before and after, he said, and a stranger can recompute the verdict without believing me.

That constraint went into the code on 2026-06-04 and it is still on origin/main:

I have quoted that line in public more than once. It is the thing I point at when I say outside review lands in the work rather than in the acknowledgements.

The gate emits an event. Something else reads that event and classifies what kind of evidence it is. That classifier lives in claim24/mandatecell7.py, and until last night it contained this:

The first one asks a structured field. The second one greps an English sentence.

notes is a human-readable string I write for my own benefit. ttlremaininghours is a number on the same event. The classifier ignored the number and searched the sentence. Rename the note and the evidence classification changes. Put the words "ttl expired" into a different kind of block and it changes the other way. Nothing structured moves.

I stored raw values so a stranger could recompute the comparison, and then I decided why I had not compared by string-matching prose.

The fix looked obvious. Add a typed field. sourceconsult, required, one of CONSULTED, UNREACHABLE, SKIPPEDNOGRANT, SKIPPEDTTLEXPIRED, SKIPPEDTIMESTAMPONLY. The classifier dispatches on the enum. It never reads notes again.

We froze that contract first, hashed it, then wrote the code. The freeze is 9f3dda8c. Its third rule says:

R3. classifyevidence on a BLOCK event with sourceconsult == SKIPPEDTTLEXPIRED returns TTLEXPIRED. Any other BLOCK returns BLOCKEDCONTROL. It must not read event.notes for this branch. ttlremaininghours may be used as a corroboration, not as the sole consult-reason.

Implementation matched. Renaming the note no longer moved anything. 366 tests passed.

Then I described the change, in prose, to a seat that could not open a single file.

And then he answered it himself. If the classifier trusts the enum without checking the structured TTL, we have not removed a self-assertion. We have retyped one. He wrote out the row he wanted tried:

The evidence class says the grant expired. The number on the same row says it has seventeen hours left. Those cannot both be authoritative.

And here is the part that matters more than the bug. The implementation was correct. R3 says the enum decides and the raw field may corroborate. The word is "may." The code did exactly what the contract told it to do. The defect was not in the patch. It was in the sentence I wrote before the patch existed.

We froze the failure before touching anything. That record is 9f5fb47d and it holds the file hashes, R3 verbatim, the attack input and output, and the test count sitting beside it. Then a second contract: an evidence class that asserts a fact must agree with the field that represents it. c686518a.

That worked for the attack that killed the first one. +17.4 with SKIPPEDTTLEXPIRED became INVALIDFORCELL7. Genuinely expired grants still classified. Notes still could not move anything.

Want to discuss this further?

Book a free strategy call with our team to see how these insights apply to your specific business goals.

Book a consultation