Three weeks ago my release pipeline shipped a product that no buyer could install.
Not "a bug we found in review." Not "a broken build on a branch." The artifact was built, versioned, tagged, uploaded to private storage, published to a public mirror, and advertised on a site with a live PayPal Buy button. Every automated check passed. The product was uninstallable by anyone who bought it.
minimatch@10.3.0 does not exist. The npm registry publishes 10.0.0 through 10.2.6. The tarball returns a 404. It was a transitive dependency — typedoc@0.28.20 requires ^10.2.5, and the published 10.2.6 satisfies that — and typedoc is a devDependency, so a plain npm install, which is the first command in the README a buyer follows, died immediately:
| Check | What it actually verified | |---|---| | The test suite | 1,072 tests across three products (357 / 390 / 325), all passing — run on a developer machine against an existing nodemodules. There was no CI. | | A claims linter | Version strings, page counts, links — not dependency resolution | | An artifact verifier | Contents: version string, manifest, absence of .git — never that the artifact runs | | "Verified from a clean copy" | True — of the working tree, not of the shipped ZIP |
Three of those were checks I ran by hand. The fourth — the test suite — was the only automated gate, and it ran locally against a warm dependency tree, so it could not have caught this. Nothing failed. The product was still broken.
Here is the structural gap: coverage measures the code you already have. A 95%-covered suite running against a warm dependency tree tells you nothing about whether a stranger on a clean machine can obtain and run your code. A test suite cannot prove its own dependencies resolve — it never has to, because they're already resolved when it starts. And with no CI, nothing ever re-ran it from scratch.
I had, in effect, never executed the artifact I was selling. I had executed the repository it was built from.
That is the whole gate. It found the defect immediately, and when I fixed the lockfiles it reported 357 / 390 / 325 passing tests across three products — from a genuinely clean install.
I shipped it in a burst of confidence and it immediately reported a different product as broken — 242 failing tests.
It was lying. That kit's README said npm test # 325 tests (needs TESTDATABASEURL; docker-compose's test-db). My gate had not started the database. I fixed the gate's setup, not its assertions.
It failed again. TESTDATABASEURL is not set — because I had skipped step two of the README, cp .env.example .env.
Both times the fix was to make the gate do the documented steps in the documented order. I did not relax an assertion. A gate that reports false failures is as dangerous as one that never fires: it gets ignored, and then it catches nothing. I'd hit this exact failure mode years earlier with a redirect stub that blocked every deploy until someone narrowed it — and the narrowing was only safe because we'd proven the change on real pages first.
A gate is not a script. It's a thing people believe. It has to be right before it can be trusted, and it has to earn that trust in both directions.
npm ci alone would have failed. We never ran it against a fresh artifact, and npm install with an existing lockfile and warm cache will happily resolve around a version that cannot be fetched.
If your build is a ZIP, a Docker image, a binary, or a Python wheel: extract it somewhere empty and install it with nothing cached. Five minutes. It is the only test that answers the question your buyer actually cares about — does this work when I receive it?
I added that gate, and it's the only sense in which I'd now use the word maintained: https://verdantstack.dev