Pick any two of ChatGPT, Claude and Gemini and there is a benchmark where each one wins. That tells you almost nothing, because the benchmark isn't your codebase, your prompt, your latency budget, or your legal team's stance on data retention.
What does predict the outcome: how each product behaves when it hits the edge of what it knows, whether it can follow the seventh item in a ten-item instruction list, and whether a compliance review will approve the vendor at all. Those are things measurable in an afternoon with prompts already sitting in Jira.
Treat everything in this section as a hypothesis to test, not a fact. Model behaviour shifts with every release, and the three vendors ship constantly. But these are the patterns developers describe over and over, and each one is checkable.
Long, multi-part instructions. Claude has a reputation for grinding through numbered constraints and often restating them before answering — helpful when you want to see what it thinks the job is, annoying when three lines of code were the goal. ChatGPT tends toward brevity and, under a long constraint list, is more often reported to quietly drop a later item. Gemini sits somewhere in between and sometimes compresses a multi-part request into a summary answer. Test: take a real ticket with eight acceptance criteria and count which criteria appear in the output.
Behaviour when it doesn't know. This is the single most expensive difference. ChatGPT is frequently described as producing a confident, plausible, wrong API call. Claude is more likely to hedge in prose — sometimes so much that the caveats have to be stripped. Gemini with search grounding enabled behaves differently from Gemini without it, which is worth knowing before any comparison. Test: ask all three to use a function that does not exist in your library and see who invents a signature for it.
Refusals and safety. The failure shapes differ operationally, not just tonally. ChatGPT typically returns a short refusal as normal content. Claude tends to explain and offer a partial answer. Gemini can block at the API layer, returning a response with no text and a finish reason indicating safety — which will throw an AttributeError in a parser that assumed response.text always exists. For pipelines processing user-generated content, that difference is a production incident waiting to happen.
Editing files versus writing them. Generating a new module from scratch is the easy case. The hard case is "change these three lines in this 600-line file and leave everything else alone." Some models return the whole file with silent unrelated edits. Some return a diff that doesn't apply. Tools like Aider expose different edit formats (whole-file, unified diff, search/replace) precisely because models differ here. Test with git diff --no-index on the before and after.
Agentic loops and tool use. Each vendor now ships a first-party coding agent: Claude Code, OpenAI's Codex, Gemini CLI. They differ in how many tool calls they'll chain before checking in, how they recover from a failed shell command, and how aggressively they read files before editing. The model and the harness are entangled — Claude inside Cursor is not Claude Code — so evaluate the combination that will actually ship.
Very long inputs. Every vendor advertises a large context window. Advertised capacity and usable capacity are not the same thing. Check the current documented limits directly, because they change, and then check the more important thing: paste an actual repo dump in and ask a question whose answer lives in the middle. That's where degradation shows up.
Characteristic failures. Reported patterns: ChatGPT invents plausible library APIs. Claude adds defensive code and explanatory comments nobody asked for. Gemini wraps JSON in markdown fences after being told not to. All three are fixable with prompting. Which one is cheapest to fix depends on the pipeline.
Set the model env vars from each vendor's current model list — don't hardcode IDs into a script that'll outlive them.
cases.jsonl comes from the backlog, not from a public eval set. Ten to twenty lines like:
That second case is deliberate — ReconcileInvoiceBatch doesn't exist. It measures hallucination, not correctness.
Score blind, one case at a time, five criteria, 0–2 each: Instruction adherence — every stated constraint honoured. Honesty — flags what it can't know instead of inventing it. Edit fidelity — no unrequested changes; patch applies. Format compliance — parseable on the first try, no stray fences. Signal-to-noise — shippable without deleting paragraphs.
For cost, don't guess: the usage dicts captured during the run give real token counts for real prompts. Multiply by each vendor's currently published rates. Do that in a spreadsheet that gets refreshed, because all three change pricing and tiering, and the cheap-tier models — Gemini Flash, the smaller ChatGPT and Claude tiers — often change the answer entirely.
Data retention and training. All three offer API terms that differ from their consumer chat terms, and the consumer ChatGPT, Claude and Gemini apps have their own opt-out settings that are not the same as the API defaults. Zero-retention arrangements exist but usually require asking. Read the current DPA for the specific product and tier being bought — not a blog post, not this one.
Region and deployment. Claude runs on Anthropic's API, AWS Bedrock and Google Vertex AI. ChatGPT models run on OpenAI's API and Azure OpenAI. Gemini runs on Google AI Studio's API and Vertex AI. Where EU-only processing is required, the answer is usually the cloud-hosted variant, and available regions per model are documented per platform. This constraint eliminates options faster than any capability test.
Rate limits. All three tier limits by account maturity and spend. A model that's fast in a notebook can throttle hard on launch day. Log the 429 responses and retry-after headers during the eval to see what's coming, and request increases before launch, not after.