Those who read my articles know that a lot of them are actually benchmarks of something. Mostly .NET related, but still, someone could say that this challenge should be pretty close to what I usually do.
Building a code benchmark and benchmarking AI models are two different worlds, so when I first saw this challenge, I had no idea what exactly I should benchmark. Comparing models on coding tasks felt too generic, and I didn't want to create a benchmark just for the sake of having one.
Then I looked at the topics of my last couple of articles. A lot of them were about APIs, resilience, failures, and how systems behave when something goes wrong. And that gave me an experiment idea: What if I benchmark AI models on one very simple question: To Retry or Not to Retry?
A 503 Service Unavailable does not automatically mean that retrying is safe. A POST request may already have been processed. An idempotency key can completely change the answer. A timeout may happen before the server receives anything or after it has already changed some state.
So the HTTP status code alone is often not enough. The model has to understand the whole situation.
The idea is simple. I prepared several API failure scenarios containing information about the request, the response, and some additional context. The model has to return two things: a decision: YES, NO, or YESAFTERDELAY a one-sentence explanation of why
The important part is not only the 503. The context tells us that an idempotency key was supplied and repeated requests with the same key will not process the payment twice. But will an AI model notice the same thing? And more importantly, what happens when the scenario is less obvious?
For this small experiment, I prepared scenarios where the answer depends on details such as HTTP method, status code, idempotency, rate limiting etc.
For scoring, I only use the Decision. This keeps the benchmark simple and deterministic. The Reason does not affect the score. I collect it because it can show whether the model actually understood the scenario or simply arrived at the correct answer for the wrong reason. That also makes the benchmark easy to compare between models. Each scenario has an expected decision, so the final result can simply be calculated as the percentage of retry decisions the model got right.
Imagine that you are integrating an external service and want to make the call more resilient. In today's world of AI-assisted coding, there is a very good chance that a coding agent will do the work. The AI can easily generate a retry policy. The more interesting question is: Will it retry the right requests? Because retrying something that should not be retried can be much worse than not retrying at all.
I prepared 14 scenarios, each based on an actual task used in the Kaggle benchmark. Originally, all of them were part of the article, but that made it too long for a small experiment. So I decided to move the full scenarios into a standalone app, where you can browse every task together with its expected answer and explanation. And if you want to try them yourself first, there is also a test for humans. After all, why not benchmark some humans too? :grin:
Here are the scenarios used in the experiment: Safe Payment Retry - A payment request fails with 503 Service Unavailable, but an idempotency key guarantees that retrying cannot create a duplicate payment. Unsafe Payment Retry - A payment request returns 503 Service Unavailable with Retry-After, but there is no idempotency key, so retrying could create a duplicate charge. Idempotent PUT - A PUT request is sent successfully, but the connection is lost before the response arrives. Rate-Limited Order - An order request is rate-limited before processing and returns 429 Too Many Requests with Retry-After. Payment Details Conflict - A multi-step payment flow reaches /payments/details, which returns 409 Conflict with transient-error: false. Non-Idempotent PATCH - A PATCH request increases inventory, but the connection is lost before the client knows whether the change was already applied. Idempotent DELETE - A session deletion request is sent, but the connection is lost before the response arrives. Service Unavailable - A GET request receives 503 Service Unavailable together with a Retry-After header. Stale Resource Version - A resource is updated by another client, so a PATCH using an old ETag fails with 412 Precondition Failed. Rate Limit Without Retry-After - A safe GET request repeatedly receives 429 Too Many Requests, but the server does not provide a Retry-After header. I'm a Teapot - A coffee request receives the legendary 418 I'm a Teapot response. Eventual Consistency - A newly created resource immediately returns 404 Not Found when another service tries to use it. Expired Filter Workflow - A temporary filter times out several times and eventually returns 404 Not Found, so the original filter can no longer be used. Cached Cart Options - A request for cart options fails with 504 Gateway Timeout, but a recently expired cached response is still available through stale-if-error.
Picking the models was pretty straightforward. First, I picked models I use a lot while working: GPT-5.6 Luna and GPT-5.6 Sol. From my experience, Luna is very efficient. It makes more mistakes than Sol, but it also uses significantly fewer tokens.
Then I added Gemini 3.7 Flash and Gemini 3.1 Pro Preview. I’ve used both quite a lot for text generation and CSS styling.
I also included Claude Sonnet 5 and Claude Haiku 4.5. I used Claude a lot for coding before, but these days I’ve mostly switched to GPT-5.6 because it feels more efficient for my workflow.
Lastly, I added GLM-5 and DeepSeek-R1 mostly out of curiosity to see how well they would perform.
I also tried to include Grok, but I immediately hit 404 Not Found errors with both Grok 4.5 and Grok 4.6, so they are not included in the results.
The goal wasn’t to test every available model, but to compare a mix of models I already use with a few I was curious about.