Have you ever asked AI to do something very specific and it did almost everything right?
Instead of asking “Which AI model is smartest?” I wanted to test something much more specific:
Did it solve the problem? Did it use the requested language? Did it avoid the prohibited method? Did it follow the required output format? Did it change anything that wasn't requested?
For every task, I: Give the exact same prompt to each model. Collect their responses. Check whether the main problem was solved. Check each instruction separately. Record which instructions were followed or violated. Compare the results across models.
This keeps the test focused on the model's behavior rather than changing the task for each model.
A response can solve the coding problem while still breaking one of the instructions.
So I measure two things separately: Task Correctness Did the model actually solve the underlying coding problem? Instruction Compliance Did it follow the individual constraints in the prompt?
Across the benchmark these scores can show whether a model is not only capable of solving the problem but also capable of doing exactly what was asked.
We often focus on whether a model can generate working code but real developer tasks usually come with constraints:
Those tiny instructions can completely change whether the response is actually useful.
Multi-turn instructions Longer prompts More constraints in a single task Conflicting instructions Ambiguous instructions More complex code editing tasks Self correction after an instruction following mistake
That could reveal whether instruction following remains reliable as tasks become more realistic.
What's the smallest instruction you've given an AI that it completely ignored? 😭