The left-pad incident is the npm outage every JavaScript developer has heard of and few have read the paperwork for. On March 22, 2016, one developer unpublished 273 packages from npm, one of them an eleven-line function called left-pad, and builds of Babel, Atom and "many thousands of projects" started failing at "hundreds of failures per minute", in npm's own words. It took two and a half hours and an unprecedented restore from backup to fix. The mechanism behind it, transitive dependencies resolved live from a registry anyone can delete from, is still how most of us build software.
TL;DR A naming dispute with Kik, the messaging app, over an npm package called kik ended with npm handing the name to Kik. Its author, Azer Koçulu, then unpublished all 273 of his packages. One was left-pad: eleven lines of code, downloaded about 2.5 million times the month before, pulled in by Babel and Atom through a package called line-numbers that pinned exactly version 0.0.3. A stranger republished left-pad within ten minutes, and nothing changed, because nothing asked for his version. npm restored the original 0.0.3 from backup, which its registry normally cannot do. Total disruption: 2.5 hours. npm's postmortem the next day: "Unrestricted un-publishing caused a lot of pain" and "We dropped the ball". A week later it shipped a 24-hour unpublish window, later widened to 72 hours. The lesson for today: commit your lockfile, and treat eleven lines as a paragraph you write yourself.
left-pad pads a string on the left to a given length. This is the whole of index.js in version 0.0.3, verbatim from the registry tarball:
Eleven lines of code, seventeen with the blanks. According to npm, via The Register, it had been downloaded 2,486,696 times in the previous month. Nearly all of those were not people choosing left-pad. They were machines installing something that installed something that needed it.
The fuse was a package name. Kik's head of messenger, Mike Roberts, later published the full email thread, so the sequence is on the record.
On March 11, Kik's patent agent asked Azer to rename his kik package. Azer said no. Sixty-six minutes later came the line that made the story: "our trademark lawyers are going to be banging on your door and taking down your accounts". Azer answered with a price, "$30.000", and Kik emailed npm support the same day. Kik's own note on the thread says "Bob is our patent agent, not a lawyer", and that Kik had "decided to use a different name for an upcoming package … even when we were told we could have the name Kik".
On March 18, npm's CEO Isaac Schlueter ruled for Kik: "most users who would come across a kik package, would reasonably expect it to be related to kik.com". On March 20 Azer wrote: "I want all my modules to be deleted including my account". Two days later he did it himself, and explained why in a post titled I've Just Liberated My Modules: "NPM is someone's private land where corporate is more powerful than the people".
npm stood by the naming call in its postmortem: "It was abrupt unpublishing, not our resolution policy, that led to yesterday's disruptions."
| When | What happened | |---|---| | Mar 11 | Kik's patent agent asks for the kik name; "lawyers banging on your door"; Azer asks $30,000; Kik emails npm | | Mar 18 | npm's CEO transfers the name to Kik | | Mar 20 | Azer asks for all his modules to be deleted | | Mar 22, ~21:30 | 273 packages unpublished; builds start failing, hundreds per minute | | Mar 22, 21:42 | Cameron Westland publishes a functionally identical left-pad 1.0.0 | | Mar 22, 23:03 | npm's CTO Laurie Voss: "we are un-un-publishing it" | | Mar 22, 23:55 | The original 0.0.3 is back, restored from backup | | Mar 23 | npm's postmortem; Kik publishes the emails | | Mar 29 | npm's new unpublish policy |
The failures start "shortly after 2:30 PM Pacific". Within ten minutes Cameron Westland republished the function as 1.0.0 (the registry records 21:42 UTC), and the builds kept failing. Then npm's CTO tweeted:
npm called the restore "unprecedented": "re-publishing isn't otherwise possible". It was done at 4:55 PM Pacific, two and a half hours after the first failures.
The detail most retellings skip is why a republished left-pad didn't fix anything. Babel and Atom did not depend on left-pad. Per npm's postmortem, they pulled it in through line-numbers, and line-numbers asked for exactly 0.0.3. A simplified sketch of the chain:
An exact pin means only that one version satisfies the dependency. Westland's 1.0.0 was the same function under a different number, so the resolver ignored it and kept returning 404 for 0.0.3. The only fix was to bring back the exact bytes, which is why npm restored from backup rather than waiting for a new release.
Kik got caught in the same chain. From Roberts' post: "our builds started failing because we use … JSCS. Through a long chain of dependencies, JSCS relied on left-pad@0.0.3". The company that asked for the name broke its own builds on the result.
Three properties of the ecosystem made this possible, and the video counted them the same way: Any author could delete any version, instantly. npm did not check who depended on a package before removing it. Dependencies are transitive and resolved live. Babel knows line-numbers, not left-pad, and the whole tree is fetched again on every install. No lockfile by default in 2016. npm shrinkwrap existed but was opt-in. package-lock.json arrived by default with npm 5 in May 2017. Until then, every CI run asked the registry again what the tree should be.
npm's postmortem, published the next day, reads like a good postmortem should: "Unrestricted un-publishing caused a lot of pain", "npm needs safeguards", "If these had been in place yesterday, this post-mortem wouldn't be necessary", "We dropped the ball", and "It took us too long to get you this update".
On March 29 npm published the new unpublish policy: You can unpublish a version only if it is less than 24 hours old. Older than that, you contact support, and support checks who depends on it. A package removed completely is replaced by a security placeholder package.
The policy page itself notes it was updated on January 30, 2020, when the window became 72 hours. The principle has not changed: after the grace period, removing something other people build on is a conversation with a human.