This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry.
I maintain a small CLI called llm-council. It puts one question to several models, hides the authorship, and has them rank each other's answers. I use it as an adversarial reviewer on my own work — the whole point is to get disagreement from something that has no reason to be polite to me.
It found a prompt-injection hole in its own prompts. That was mildly embarrassing. What actually kept me up was the second finding: I had already written a test for exactly that hole, and the test was green.
When you chain models, the output of one becomes the input of the next. In llm-council, stage 1 collects answers, stage 2 asks a model to rank them, stage 3 asks for a synthesis. Every stage feeds the previous stage's text — text written by an untrusted party — into a new prompt.
That is OWASP LLM01 in its plainest form, and the standard mitigation is fencing: wrap untrusted content in delimiters and tell the reader that anything inside is quoted data, never instructions.
So a hostile voter — or a model that had simply read the repo during training — could write >> in the middle of its own answer. To the model reading downstream, that closes the block. Everything after it stops being quoted data and starts being orchestrator text.
The name claims a security property: a voter cannot forge a boundary. The assertion counts occurrences of a Python string and checks an index ordering. Both of those are true whether or not the attack works — the forged marker is in the text either way, and it sits where the arithmetic expects. The test verifies that string concatenation concatenated. It never asks the only question that matters: can the reader be deceived?
This is the subtle version of "a test that cannot fail." It is not empty and it is not skipped. It runs, it exercises real code, it would catch a genuine refactoring mistake. It simply does not touch the property its name advertises — and the name is what everyone reads when deciding whether an area is covered.
That test had been sitting in a suite at 100% coverage. Coverage is a claim about lines executed. It says nothing about whether the assertions are pointed at anything.
The defence had to move from the shape of the markers to something the attacker has never seen: a per-run random nonce.
secrets, not random — this is a security boundary, and a predictable PRNG would hand back exactly what the nonce was meant to take away.
Then the test was rewritten to assert the property instead of the arithmetic (abridged — the source has the assert ... is not None narrowing that mypy wants, and an assertion message in Italian):
The property is not "no fake markers exist in the text" — an attacker controls its own output and can type anything. The property is that only the markers we emitted carry the real nonce, so a forged one is inert text.
The same review turned up a third gap: in stage 3, the rankings were going in raw while the responses beside them were fenced. One uncovered seam in a defence that exists precisely because a model's output re-enters another model's input.
I verified the fixes by mutation rather than by trusting the green: reverting to a static nonce turns 3 tests red, and unfencing the rankings turns 2 red. The old test is the control in that experiment — it stayed green for the entire time the vulnerability was live, which is the only measurement that ever mattered.
I opened the PR. The SonarCloud quality gate — newly mandatory, this was the first PR it blocked — failed it.
Same expression on both sides. The rule exists because that shape is usually a copy-paste bug, and the scanner could not know I meant it. But the scanner was right anyway, for a better reason than it had: two draws is a terrible test for randomness. It passes with a counter. It passes with a clock.
A nonce collision is a reusable forgery. That is worth a stronger test, not a waiver.