This article reports a Kaggle benchmark for a step every agent performs and few people test, counting what a tool returns, and ends with a step by step guide to reproducing it.
An agent's search, database or API tool usually hands back a list of records. When the user asks how many, the model does the counting. That step passes every quick test with a handful of rows, and nothing flags it when it goes wrong: the model sends the right query and quotes a confident number.
This benchmark asks ten models the same 68 counting questions with two versions of one tool: countids returns the exact count, listids returns the matching ids.
With the count, every model but one answered every question about 330 ids correctly. With the ids, the models split in two. The ones that spent 2,700 to 8,200 output tokens per question counted 15 to 21 of 21 lists correctly. The ones that answered in under 600 tokens counted 0 to 10.
The surprise was who landed where: Claude Opus 5, the most expensive model in the lineup, answered in 579 tokens and counted 9 of 21. Gemma 4 26B, a 26B open-weight model, spent 8,153 and counted 20.
How many are 10 or more? The answer is 8, and every model here gets it. Eleven ids say nothing about three hundred, and eleven is about the size of the lists most quick tests use.
Language models' trouble with counting is familiar from examples like counting the letters in a word. This benchmark measures it where agents meet it: inside tool use, with every miss traced to the query or the count, and every answer priced in tokens.
| Task | What the model gets | Who counts | |---|---|---| | count-engine | countids(where), which returns the exact count, minimum and maximum | The tool | | count-rows-tool | listids(where), which returns the matching ids | The model, from the returned list | | count-python-tool | Every id in the prompt, plus runpython with ids already defined | The model's code, if it writes any |
In the first two tasks the ids never appear in the prompt, so the only difference is whether the tool returns a number or a list. Both tasks check every filter the model sends by the ids it selects, so id > 9 counts as right for "10 or more", and every wrong answer is traced to either a wrong query or a wrong count.
Each task asks 68 questions: lists of 11, 110 and 330 ids, seven phrasings of the threshold ("10 or more", "no less than", "under", "between 5 and 9 inclusive" and so on), three seeds each, and the original eleven ids five times. Every threshold is an id in the list, so > and >= always give different answers.
| Vendor | Models | |---|---| | Google | Gemini 2.5 Flash, Gemini 3.7 Flash, Gemini 3.8 Flash, Gemma 4 26B A4B | | Anthropic | Claude Haiku 4.5, Claude Sonnet 5, Claude Opus 5 | | OpenAI | GPT-5.4 nano, GPT-5.4 mini, gpt-oss-20b |
The lineup takes a small, a mid-sized and a large model from each vendor, plus the open-weight models from Google and OpenAI, so the comparison covers price, size and what anyone can run. Each model runs with the settings Kaggle's model proxy serves it with. Some of those reason before answering and some answer straight away, and that setting tracks the result more closely than size or price.
GPT-6 Astra is refused function tools by the proxy (Function tools with reasoningeffort are not supported for gpt-6-astra in /v1/chat/completions). Gemini 3.5 Flash-Lite and both Qwen 3 Next 80B models returned 429 or 503 on most calls.
| Model | Rows tool correct | Output tokens per question, rows tool | Engine correct | Rows-tool cost vs engine | |---|---|---|---|---| | Gemma 4 26B A4B | 20/21 | 8,153 | 21/21 | 18.6x | | Gemini 3.7 Flash | 20/21 | 2,785 | 21/21 | 13.5x | | Gemini 3.8 Flash | 21/21 | 2,742 | 21/21 | 14.3x | | gpt-oss-20b | 15/21 | 2,684 | 19/21 | 5.2x | | Gemini 2.5 Flash | 0/21 | 580 | 21/21 | 2.6x | | Claude Opus 5 | 9/21 | 579 | 21/21 | 2.4x | | Claude Sonnet 5 | 10/21 | 401 | 21/21 | 1.9x | | Claude Haiku 4.5 | 5/21 | 143 | 21/21 | 1.5x | | GPT-5.4 mini | 5/21 | 39 | 21/21 | 1.9x | | GPT-5.4 nano | 0/21 | 35 | 21/21 | 1.7x |
The Claude rows-tool figures come from their 2026-09-25 run, which asked the same 68 questions with the same tool; their 2026-09-28 rows-tool run stopped on the daily quota. Every other figure comes from the 2026-09-28 runs. With the Count in the Tool, Every Model Is Right at a Flat Cost
Every model but one answered all 21 engine questions about 330 ids correctly, at 35 to 451 output tokens per question, and each model's tokens stayed about the same from 11 ids to 330. Turning "no less than 244" into id >= 244 and quoting the number back is something every model here does reliably. The exception, gpt-oss-20b, answered 5 of 68 engine questions with a number other than the count it was given, all of them 0, 1 or 2, and answered once without calling the tool. With the Rows, Quick Tests Pass and Larger Lists Fail
Eight of the ten models counted all 26 questions about 11 ids correctly. At 330 ids, five of those eight counted 10 or fewer of 21 correctly. None of them returned an error or a hedge: each answer was a single confident number. The Models That Count Right Spend the Tokens
The models split into two groups by how many tokens they spend, and model size and price do not predict the split. That was the surprise. The models that count correctly spend more tokens as the list grows: Gemma 4 26B went from 394 output tokens per question at 11 ids to 8,153 at 330. The models that miscount spend about the same at every size: GPT-5.4 nano spent 35 tokens per question at 11, 110 and 330 ids, which leaves no room to count anything. Claude Opus 5, the most expensive model here, spent 579 output tokens per question at 330 ids and counted 9 of 21 correctly; Gemma 4 26B, an open-weight model, spent 8,153 and counted 20.