Days later I was looking at something completely unrelated when I noticed it: a state update that had been technically correct in exactly the way I'd asked for, had quietly broken an assumption the rest of the system depended on. The update itself was fine. The problem was that another piece of state was supposed to change along with it and that relationship wasn't obvious from the function I was editing. The tests covered the updated state. They didn't cover the relationship between the two Nothing had crashed. No test had failed. No reviewer, including me, had caught it at the time.
The scary part wasn't that AI wrote the code. I'd expected AI-written code to have bugs, the obvious kind, the kind a test catches immediately.
The scary part was that everything, every single signal I normally trust, told me the code was correct.
That's the actual problem with AI-generated code. Bad code is easy to distrust. Code that passes every test you throw at it is much harder to question, and that's exactly backwards from how risk should work.
I know what you're thinking, because I thought it too: if the tests passed, what's actually wrong here?
Tests verify behavior you thought to check. They don't automatically prove your architecture is sound, that a business invariant survived the change, that there's no race condition, that an edge case is handled, or that the abstraction the AI generated actually makes sense for where the codebase is headed.
A passing test tells you that one expectation passed. It doesn't tell you that your understanding of the system is correct.
That gap, between "the thing I checked worked" and "the thing I understand is true," is where my bug was hiding the entire time.
CI: green. Tests: green. Lint: green. Build: green. The AI's explanation of what it changed: confident, clear, reasonable-sounding. The diff itself: sensible, nothing that looked out of place.
And that's exactly why it was dangerous. I wasn't ignoring red flags. There weren't any red flags to ignore. Every signal I'd normally use to decide "does this need a closer look" said no.
The state update genuinely did what I'd asked. It just also did something I hadn't asked about at all, and nothing in my review process was built to catch that category of problem.
Here's the technical core of what actually happened, stripped down to the general shape of it.
Say you think the rule is: "if a condition is true, allow the update." You write a test for the true case. You write a test for the false case. Both pass. Looks solid.
But the real invariant, the one that actually needed to hold, was something neither test captured: that this piece of state couldn't be safely updated independently of another piece of state elsewhere in the system. The AI's change satisfied the specification I gave it completely. The specification itself just didn't include the constraint that mattered.
AI is very good at satisfying the specification you give it. The problem is that developers often give AI a specification without realizing they've left out the most important constraint. The dangerous part is when the specification is incomplete, and you don't find out it was incomplete until after you've already trusted the result.
AI Doesn't Make This Worse Because AI Is Bad. It Makes It Worse Because AI Is Convincing.
I don't think this is an argument against AI-generated code. I think it's an argument about what confidence actually means.
AI-generated code usually comes with clean naming, comments, reasonable-looking abstractions, tests, and a confident explanation of what it did and why. Your brain reads all of that and concludes: this looks professional, this looks like it was thought through.