TL;DR. I tested 17 AI models on writing, recalling and fixing OpenTelemetry instrumentation, graded in code against 26 releases of the spec. 16 of 17 models wrote retired names, and 83% of those names had been retired for more than a year. 12 of 13 models wrote names they had correctly replaced when asked directly. 7 of 17 "fixed" a metric by changing its unit and keeping its retired name, which breaks it both ways. One sentence naming the spec version cut retired names by 36%, and made models invent more. Benchmark: Semconv Drift on Kaggle
I asked Claude Sonnet 5 what replaced the OpenTelemetry attribute db.system. It answered correctly: db.system.name, the name the spec has used since January 2025.
In the same benchmark, I asked it to add tracing to a go-redis call. It wrote this:
All three names are retired. The code compiles, the span exports, the backend stores it. Nothing fails. Weeks later, a dashboard that filters on db.system.name shows no Redis traffic, and nobody connects the empty panel to a line of generated code.
This year I have had 14 pull requests merged into OpenTelemetry's Go compile-time instrumentation project, and this is exactly the kind of bug I would wave through in review. The code looks right. It is only wrong against a spec version, and nobody reviews against a spec version. So I built a benchmark to measure how often it happens.
OpenTelemetry's semantic conventions are the shared vocabulary of telemetry: which key holds the HTTP method, what a database span calls its query, which unit a duration metric uses. They are versioned, and many of the names people learned first have been renamed. http.method became http.request.method in 2023. db.statement became db.query.text in 2024. rpc.client.duration, in milliseconds, became rpc.client.call.duration, in seconds, in January 2026.
| Task | Cases | What the model does | | --- | ---: | --- | | Write | 12 | Instrument everyday code: Flask and Express servers, an HTTP client, psycopg, go-redis, kafkajs, gRPC, metrics and resources, in Python, Go and TypeScript | | Know | 22 | Name what replaced a retired name. Includes span-kind traps and two controls whose right answer is "unchanged" | | Fix | 8 | Modernize old snippets, including real code from redis/go-redis, converting values where a unit changed | | Write with version hint | 12 | The Write requests plus one sentence naming the current spec version |
Every Write prompt already says "Follow the current OpenTelemetry semantic conventions." The models were told. The question is what "current" means to them.
Grading is code, not a judge. The grader extracts every semantic convention name from an answer (string keys, metric names, SDK constants in four languages, enum values) and looks each one up in 1,563 names from 26 releases of the spec, v1.21.0 to v1.44.0, plus every name from v1.0.0 to v1.20.0 for the oldest ones. Every retired name carries the release and date that retired it. Reasoning text is stripped first, so a model is judged on the code it wrote. The answer key is tested too: the build fails if any gold answer is not a current name in the ground truth.
The ground truth comes from Attrition, a dataset of every tagged release of open-telemetry/semantic-conventions that I built for DEV's Sanity challenge.
17 models through Kaggle's model proxy, picked to answer three questions: Does a newer generation fix it? Three same-family pairs: Claude Sonnet 4.5 and 5, GPT-5.4 and 5.5, Gemini 2.5 Pro and 3.1 Pro. Does spending more fix it? Frontier models (GPT-6 Astra, Claude Opus 5, GPT-5.5, Gemini 3.1 Pro) next to small, cheap ones (GPT-5.4 mini, Claude Haiku 4.5, Gemini 3.7 and 3.8 Flash). Do open-weight models differ? Qwen 3 235B, DeepSeek-R1, GLM-5, Gemma 4 31B and gpt-oss-120b.
That is 830 graded answers, plus 84 more from repeat runs. Four reasoning models (DeepSeek-R1, GLM-5, Gemma 4 and gpt-oss-120b) have no Know score: on some short factual questions they spent the whole answer budget thinking and never answered. Grok 4.5 and 4.6 returned 404 for my account, so they are not in the lineup.
16 of 17 models wrote at least one retired name in Write, 357 uses in all. If this were about training cutoffs, the retired names would be recent ones the models never saw. They are not: 297 of 357 (83%) had been retired for more than a year when I ran the benchmark. 238 (67%) had been retired for more than two years, which means before October 2024. Every model in the lineup was released after that. The median retired name was retired 2.4 years earlier.
These models were not missing the news. They had read the changelog, and years of tutorials before it, and the tutorials won. db.system is the clearest case: 24 uses across 12 models, nearly two years after the rename.
The only names a training cutoff could excuse are the January 2026 rpc. renames. 16 models wrote them. GPT-6 Astra was the only exception, and it also avoided every older retired name, so a late cutoff alone does not explain its perfect Write score. It told me the new name, then wrote the old one
Because Know and Write ask about the same names, I can check each model against itself. For every retired name a model wrote in Write, did the same model give the correct replacement in Know?
12 of 13 models wrote at least one retired name they had correctly replaced when asked directly, 58 times in all. Claude Haiku 4.5 said http.method became http.request.method, then wrote http.method five times. Sonnet 5 did it with db.system, as in the opening. Only GPT-6 Astra never did it.
This is not a knowledge gap. Asked a question, a model retrieves the changelog. Asked for code, it retrieves the most common code, and the most common code is old. The half fix