It had three versions. Each one was correct. Each one gave a different answer. Version one said 18. It counted a reply only if it sat directly under theirs. But dev.to sometimes will not show a comment its API still returns, so you reply beside it instead. Nine of ten had been answered that way. Version two said 34. It counted any later comment from me in the thread. So it counted two other people talking to each other. Version three said 2. It used both rules. One was a friendly sign-off from July. The other was Pascal, suggesting there might be an article hiding in these comments.
Nothing in the tool was broken. Each count was right about its own population. The trouble was the question. Every version reported its number as the answer to "who is waiting?", and every version had quietly decided what "answered" means.
The first version was built with care. It encoded a sensible judgement. Then the judgement stopped looking like one.
It started in July, in the comments under Pascal's article about replacing Calendly. It ran for two months, in bursts, with pauses while one of us went off and tested things in production.
Sometimes Pascal had the answer and I found the hole. Sometimes the other way round. Here it is in the order it came up.
The first example is from Pascal's article. Two people try to book the same slot at the same moment. On Postgres that is a real race. On SQLite it cannot happen, because SQLite lets only one writer in at a time.
So a test for that race, run on SQLite, passes forever. The bug ships on Postgres.
The fix is mechanical. After you write a test, break the thing it protects on purpose and run it again.
Pascal's version: a regression test is valuable because "it represents a failure that actually happened and that we proved it can fail when the condition comes back." Most people keep the first half of that sentence and drop the second.
Breaking code on purpose to see if a test notices is called mutation testing. When the test stays green, everyone assumes the test is weak.
My case was the first box. A key binding compared the key press against " ". The runtime reported the space bar as "space". That branch had never matched in the whole life of the file.
I had already told a colleague I had introduced a regression there. There was nothing to regress.
Pascal turned the boxes into an order of checks: Is the path reachable? Does the test observe the behaviour? Does breaking it make the test fail?
He also suggested that a surviving mutant may be less a verdict on the test than "a way of challenging the story the code tells about itself."
That story has several authors. The code says what it does. The tests appear to confirm it. The last person who explained the system repeats it.
They can all be wrong in the same way at once. That is what happened with the space bar.
What breaks the agreement is something that cannot be told the story. Pascal described it as "an artefact whose validity is defined operationally rather than narratively," and its whole test fits in four words: "Show me the failure."
Then he pushed further. Tests only check what somebody thought of. Real users bring what nobody thought of. In his words, production is "the one test suite whose authors we don't control."