This is part five of a series about pointing an append-only audit log at things that count tokens. Parts one through four found accounting defects in other people's trackers and one expensive hole in my own routing. This one is about what happens after the report lands, because this time the audit ran in both directions: the maintainer shipped the fix in three days, and his closing comment contained two findings aimed back at my report. One of them caught a claim I had published without its scope. My re-measurement then caught the mechanism he had guessed for it.
The target is Clawdmeter, a desktop app that shows your live Claude Code usage with a pixel mascot. weltern announced the cross-platform release on r/ClaudeAI; I pointed my harness at its token path the same day and filed #21: every transcript-derived number in the app read about 2.34× high.
Same class as part three, so one paragraph and no re-litigation. Claude Code writes one assistant message as several JSONL records, one per content block, and each record repeats the same message.id with the same usage object. src/transcript.py summed per record in three places: the token counts beside the 5h/7d bars, everything the Stats page prices, and the per-session totals. Driving those three functions over a synthetic corpus with a known-exact manifest: 2.34×, 2.34×, 2.37×. Both record-level paths emitted 1,249 events for 540 messages. The percentage bars were never affected, since those come from rate-limit headers, and the overage figure comes from the OAuth endpoint. The report said so, because a fair report names what is not broken.
The part of the method worth stealing. Nothing in the report reproduces Clawdmeter's logic, models it, or reimplements it. The harness imports the vendor's own transcript.py and calls his functions. The app is a Qt desktop program, and the token path doesn't need Qt, so the whole trick is a twelve-line stub:
Why bother, when reading the code already showed the defect? Because of the dispute it removes. A reimplementation invites "your model of my code is wrong", and that argument can run for days. Numbers that came out of the vendor's own functions leave exactly one question open, whether the input corpus is fair, and a synthetic corpus with a by-construction manifest answers that in the same breath. The report gets to say "your functions produce 2,592,168 where the manifest says 1,108,697" instead of "I believe your code over-counts".
It also makes reproduction nearly free: git clone, one script, about a minute, a fake $HOME, no PySide6 install, nothing real touched.
Filed August 5. Confirmed August 6: "Thank you again for this report, I've confirmed the bug and will be patching it in an upcoming hotfix release." Shipped August 8 as v3.0.1, with a release note that opens the way I suggested it should, because a correct fix that halves every visible number looks exactly like a regression: "Your token and value numbers will drop by roughly 2.5×. That is the fix, not a regression."
One detail I want to be flat about: he never ran my harness. He reproduced independently, on his own corpus, with his own tooling. That is not the fixture failing at its job. The fixture's job is to make the claim cheap to check and expensive to dispute; his choice to verify it his own way instead is a stronger confirmation than an exit code, and the diagnosis still arrived pre-named, which is what made three days possible. A reproduction script is an offer, not a toll booth.
His closing comment is the reason this post exists. It opens with "Your report was accurate and the diagnosis was right", and then does something I had not seen a maintainer do in five repos: it treats the bug report itself as a measurable claim, measures it, and reports back where it breaks. "Two things turned up that are worth passing back, since you audit other trackers."
His corpus came out worse than mine: 44,196 usage-bearing records across 18,585 distinct message ids, 73.0% on more than one record, work tokens 2.575× high. And then the two findings.
First: the repeated usage objects are not always identical. My report said that on my corpus, 100.0% of duplicated messages carried byte-identical usage on every record. On his corpus, 3,799 groups differ, and in every one outputtokens is a running total: early records carry a partial count, the last carries the final figure. His example message reads 5, 5, 5, 328 across four records. That makes the choice of which record to keep load-bearing. A per-bucket max is correct; keep-first would have reported that message as 5 output tokens instead of 328.
Second: collapsing per file is not enough. 1,094 of his message ids appear in more than one transcript file, because resuming a session replays its records verbatim into a new file. Deduping within each file still left 1.095× inflation on his tree. The collapse has to live in the account-wide aggregator, and a replayed copy can carry an all-zero usage block where the original carries the real numbers, so a record-level dedup placed in front of the message collapse can keep the wrong copy. Folding every record into its message under a per-bucket max sidesteps the whole trap.
He also turned my one-line invariant, usage events should equal distinct message ids, into a regression test, added six more, and ran nine mutations of the fix against them. Nine of nine caught. That last number is a stronger claim than any test count, and it is the part I would not have thought to ask for.
Finding one corrects something I wrote, so I owed it a measurement rather than a thank-you. My "100.0% byte-identical" was true and under-scoped: it was true of the main conversation path, and I did not say so. On my corpus, regrouped:
| grouped per (file, message.id) | groups | byte-identical | |---|---|---| | main path | 10,645 | 10,645 (100.00%) | | sidechain (subagent) | 16,051 | 3,342 (20.82%) |
Not one main-path group differs, across 63 days and 1,521 transcripts. Every differing group has a sidechain record in it. The discriminator is isSidechain, not the Claude Code version he suspected — and the reason I can say that with some force is that the version hypothesis makes a testable prediction and the field hypothesis makes a different one. If the shape tracked the writer version, main-path groups would differ on some machines. They never do here. His Linux and macOS corpora "still showing the byte-identical shape" is what a main-path-heavy tree looks like; an agent-heavy tree shows his 3,799.
His running-total observation, meanwhile, holds without exception on my side: 12,709 differing groups, 100% non-decreasing, 100% with the last record carrying the max. He was right about the shape and wrong about the cause. I was right about my corpus and wrong to publish the number without its denominator. Neither error survived a second corpus, which is the whole argument for sending measurements instead of opinions.
One warning went into my reply because it is the check anyone would write next, and it lies. Group repeated records by message.id alone and my corpus reports 560 non-monotonic groups, output counts that go down mid-message. Every one is fake: records from unrelated files spliced into one sequence by the sort. Group per (file, message.id) and all 560 vanish. I know the trap is real because I fell into it on a different tracker and reported the bad number before catching it.