An LLM asked to document a payments module — about 900 lines, four public functions, a retry wrapper — will typically produce something gorgeous. Structured headings, a parameters table, a "Common Pitfalls" section. It will also document a timeoutseconds parameter that does not exist. It will describe exponential backoff with jitter when the code does a flat time.sleep(2) in a for loop. It will include a usage example that imports a symbol from the wrong module.
Errors like these survive for weeks, because nobody reads documentation that nobody trusts. The first person to try the example files a bug against the library.
That's the core problem with pointing a model at a repo and asking for docs. The model isn't reading the code so much as pattern-matching it to the average of every similar library it has seen. A retry helper looks like tenacity, so it gets tenacity's parameters. A webhook handler looks like Stripe's, so it gets Stripe's idempotency semantics. The output is fluent, structurally correct, and wrong in exactly the places a reader can't check without reading the source — which is the whole reason they came to the docs.
Wrong docs are worse than no docs. No docs make you read the code. Wrong docs make you skip reading the code.
Models are bad at inventing facts about a system and good at reformatting facts handed to them. So stop asking for documentation and start asking for transformations of artefacts that already encode the truth.
A codebase is full of these: Type signatures. def charge(accountid: UUID, amount: Money, , idempotencykey: str) -> ChargeResult already tells you every parameter, whether it's keyword-only, and what comes back. A model can't invent a fifth parameter if it's constrained to the signature. Tests. A passing test is an example that ran. Turning testrefundpartialamount into a documented example is a formatting job, not a reasoning job. OpenAPI specs / JSON Schema. If the spec is generated from code (FastAPI, tsoa, utoipa), the spec is ground truth. Prose around it is the only thing to write. Migrations. ALTER TABLE charges ADD COLUMN settledat timestamptz NULL tells you the column exists, its type, and that it's nullable. That's three facts you don't have to trust anyone for. Git history. git log --follow -p src/billing/retry.py explains why the sleep is flat. The commit message says "revert jitter, it broke replay determinism in tests." No model would guess that. Every model can summarise it once it's pasted in.
The prompt shifts from "document this module" to "here is the signature, here are the three tests that cover it, here is the commit that introduced it — write the docstring, and write UNKNOWN for anything not present in this input."
That last instruction does most of the work. Models will happily emit UNKNOWN when it's an allowed output. They will not volunteer uncertainty when the only available move is prose.
A whole-repo docs pass is a one-time event that produces a large, unreviewable PR. Nobody reads a 4,000-line documentation diff carefully. It gets approved on vibes and rots from day one.
A diff-scoped pass is small enough to actually review, and it lands with the change that motivated it. A script along these lines does the job.
python'; cat "$f"; echo 'diff'; git diff "$BASE...HEAD" -- "$f"; echo 'python'; echo "$tests"; echo '
llm is Simon Willison's CLI (pipx install llm, then llm keys set anthropic). The system prompt is where the constraints live:
Output goes to a scratch directory, not straight into the file. The engineer applies what's right and deletes the rest. The UNKNOWN markers become a to-do list of things only a human knows.
The single highest-value check: every example in the docs must execute in CI. For Python, doctest gets you there almost free.
That last step is the one that catches drift. If someone adds a query parameter and doesn't regenerate the spec, CI fails with a diff showing exactly what changed. For contract-level checking, schemathesis run docs/openapi.json --url http://localhost:8000 will hammer a running service with requests derived from the spec and report where behaviour and documentation disagree.
Generated prose has no compiler. A docstring that describes behaviour removed six months ago will sit there forever. What's needed is a mechanical link between the doc and the thing it documents.
It's blunt — a whitespace change trips it — but a noisy check that forces a five-second re-read beats silent rot. Add python scripts/checkdocstamps.py to CI and require it on protected branches.
The highest-value prompt is rarely "write docs." It's this, run against a doc page plus the code it describes: