You ship an AI agent. It calls tools, reads results, calls more tools, answers. Most of the time it works. Then a user reports something wrong, you open the trace, and you find it: the charge_card tool returned a 402, and the agent just... kept going and told the customer their order shipped.
That's not a hallucination in the "made up a fact" sense. It's a structural defect in the run — an ignored tool error. And here's the thing about structural defects: you don't need another LLM to find them. They're decidable by looking at the trace.
That's the whole premise of tracelint: a linter for agent runs. It reads the execution trace — what the agent actually did — and flags structural bugs deterministically, with the exact trace lines as evidence and a CI exit code. It runs after the run, on the trace, not on your code. No second model ever judges it.
Because for this class of bug, a judge is the wrong tool. Published trace-error benchmarks show LLM judges have low localization accuracy — they'll tell you "something seems off" without reliably pointing at which step. They're also non-deterministic, cost money per trace, and can't gate CI (would you fail a build on a coin-flip?).
Meanwhile, a whole category of agent bugs is structurally decidable: A tool call whose arguments violate the tool's JSON Schema. That's not an opinion — you run the schema validator. A tool that returned an error, followed by the agent proceeding as if it hadn't. The same tool called 5 times with identical arguments and identical results (a stuck loop). Arguments that don't appear anywhere in what the agent observed (a candidate hallucinated value).
None of these needs a model. They need the trace and a validator. That's what tracelint does.
demo runs a keyless validation suite — one planted instance of every defect, plus clean controls — and writes an HTML report. No API key, no model download.
Exit codes: 0 clean, 2 a structurally-provable defect, 3 an input error. Heuristic findings never fail CI on their own.
Here's the distribution insight. You're probably already instrumenting your agent — with OpenInference (the OpenTelemetry semantic convention for AI), feeding Arize Phoenix, Langfuse, or an OTel collector. tracelint reads that telemetry directly. You don't learn a new trace format; you point it at the spans you've got.
I validated this against real OpenInference exports, not just hand-built fixtures — a real Phoenix trace, an OTel-SDK span export, the Phoenix dataframe shape. On one real Phoenix trace, tracelint deterministically localized a genuine tool failure:
No model in the loop. Just: this TOOL span has an ERROR status, at exactly this step, here's the message.
| Rule | Finding | |------|---------| | R1 | schema violation — args fail the tool's JSON Schema | | R2 | tool returned an error / an errored value reused by a later side-effecting call | | R3 | hallucinated argument — value not derivable from anything observed | | R4 | loop — N identical no-progress calls | | R5 | redundant call — identical call + identical result, no mutation between | | R6 | malformed arguments — the tool-call arguments aren't valid JSON | | R7 | unknown tool — a call to a tool absent from the declared toolset |
Two design decisions I'd defend Candidate, not verdict. Only structurally-provable things (a schema violation, malformed JSON) are hard defects that fail CI. Heuristic signals — loops, redundant calls, suspicious arguments — are shown as candidates with their evidence, for a human to review, never asserted as truth and never failing your build on their own. A retry loop and a stuck loop look similar structurally; tracelint shows you the evidence and lets you decide, instead of pretending it knows. It tells you what it couldn't check. This is the one I care about most. If a trace is missing a field a rule needs — no tool schemas, no result payloads — that rule doesn't silently pass. It suppresses with a stated reason, printed in the report:
A clean report with hidden gaps is worse than no report — it's false confidence. tracelint refuses to give you that.
Honest limitations It catches structural defects, not whether the final answer was correct. It won't tell you the agent gave bad advice; it'll tell you the agent ignored a failed tool call on the way there. Hallucinated-argument, loop, and redundant-call findings are candidates unless structurally proven. Legitimate value transforms and intentional retries can trip them — that's why they're shown with evidence, not asserted. A trace is only as good as its instrumentation. Missing fields mean suppressed rules, not fabricated ones.
It's open source (MIT), dependency-light (jsonschema + stdlib), Python 3.10–3.12, and the whole test suite is offline and deterministic. Repo: https://github.com/AshwinUgale/tracelint
If you're collecting agent traces and want deterministic checks on them, I'd genuinely like to know what breaks on your real exports — that's how the last three real-world shape fixes happened. Issues and traces welcome.