I took the whole codebase. Every route, the migrations, the config, that utils.ts nobody's touched since 2023. Dumped all of it into a model with a context window big enough to swallow the entire repo in one go, and asked it one question:
Now, everyone's first reaction to that is the security angle. "You pasted your source into a chatbot?" Fair. I'll get to it, and yeah, it matters. But that's not the part that got to me.
What got to me was the answer. It came back as a clean numbered list, and I sat there realizing I couldn't actually tell which items were true and which weren't. They all read the same.
I expected file summaries. What I got was my architecture, described back to me. It knew auth ran through one middleware. It knew two of my services were quietly sharing a table they had no business sharing. It knew the payment webhook and the signup path both wrote to users from totally different places in the code.
Honestly? It mapped the system better than the onboarding doc I'd written for new hires. Faster, too.
If you've never tried this, do it once just for this. A model sitting on your entire repo can answer "where does X happen, and what breaks if I change it" better than half the people who actually work in the thing. That part is real, and it's worth your time.
This is where I got excited. It started surfacing actual problems: An old .env I'd committed back in 2022 and deleted the same week. Still sitting in git history, of course. Forever. I knew that in the abstract. I had completely forgotten it in practice. A race between the webhook handler and the signup path. Both upsert the same row, no lock. Under a retry, one clobbers the other. Nobody had hit it yet, but it was absolutely there. A dead admin endpoint. Unhooked from the UI two years ago, still mounted, still skipping the newer permission check. Reachable. Forgotten.
These weren't lint warnings. These were "how did I ship that" bugs. For a solid ten minutes I thought I was writing a very different post, the one titled "just give ChatGPT your codebase already, it's incredible."
And look, as a finder, it kind of is incredible. It's read more code than I ever will and it doesn't get tired on file number 340. I'm not going to pretend otherwise.
It had everything going for it, though. Plausible name, plausible file, a nice confident description of how an attacker would exploit it, the whole "this could lead to data exfiltration" bit. Every signal your brain uses to go "ok this person knows what they're talking about" was firing. The only thing missing was the code it was describing.
Item 9 flagged a missing auth check on an endpoint that had the check. It was one file over from where it was looking.
Here's what actually rattled me. Items 7 and 9 looked identical to items 1 through 3. Same confidence. Same layout. Same tidy little "here's how to fix it" block. Nothing in the way it wrote the fake ones told me they were fake.
More context hadn't made it more honest. It just gave it more of my real function names to wrap a made-up story around. And a hallucination wearing your own variable names is a lot more convincing than a generic one, trust me.
Buried in the webhook handler was the real landmine. Not the race this time. Something quieter:
The ack goes out before the write is safe. If the process dies, or the DB throws, in that tiny window after the acknowledgement, the provider thinks it's done and never retries. The payment is just gone. Silently. And every log line is green. (If you've read the 2am outage post, you already know how this movie ends.)
This is the single most dangerous thing in the repo. Data loss with no error to point at.
It told me the handler looked correct. Said the try/catch was good practice. Suggested I add a comment.
Same tone it had just used to correctly nuke my dead admin endpoint. Same tone it used to invent a SQL injection out of thin air. On the one bug that can actually lose a customer's money, it gave me a thumbs up and a note about code style.