I generate images locally in batches, and a vision model scores each one before anything ships. Theme, humour, wit, background, one composite number. Anything under the bar gets rebuilt.
That ran for weeks. Then I dumped the raw scores instead of the pass/fail summary and actually looked at them.
Humour came back as 7 forty times in a row. The best image on that slate scored 8.82 and the worst scored 8.38, which is a range of 0.44 on a ten point scale. Nothing was ever going to fail that gate.
They sat in a sensible range. They had decimals. The composite even wobbled between images, because bg had some genuine variance and dragged the average around by tenths, so no two images ever printed exactly the same final score. That wobble is what sold it. It looked like a measurement.
If your judge throws an exception you fix it that afternoon. If it hands back 8.6 for everything, you skim it, think "yeah, slate's decent", and run on it for a month.
So: go and take the standard deviation of each axis over your last run. Five lines. If an axis is flat it isn't measuring anything, and no individual score will ever tell you that.
First thing I assumed was that my rubric was too woolly, so I rewrote it with hard numeric anchors and told the thing to be a harsh critic.
Same model, same 40 images. Humour loosened up and started using 2 through 5. Wit then collapsed instead, 7 on 38 of 40, standard deviation 0.22.
All I'd done was move the flat axis. That's when I gave up on prompt fixes and went looking for a different model.
Same 40 images, same anchored prompt, three models. I'm quoting standard deviation rather than min to max, because a single outlier makes a rubber stamp look like it's doing something.
| model | humour sd | wit sd | distinct wit scores | text defects found | speed | |---|---|---|---|---|---| | qwen2.5vl:7b | 0.99 | 0.22 | 2 | 0 of 40 | 1.0 s | | qwen3-vl:30b-a3b-instruct | 0.82 | 1.12 | 6 | 0 of 40 | 2.6 s | | qwen3-vl:32b-thinking | 2.54 | 2.33 | 10 | 4 of 40 | 164.7 s |
The 30b deserves better than I first gave it, though. It isn't a rubber stamp. It reliably drops the images that ignored the brief to a 2 while everything else gets a 5, so there's real pass/fail signal in there. It just can't rank anything that passed, which was the bit I actually wanted.
The other thing I want from a judge is catching mangled text, because generated images produce garbled shop signs constantly and it's the fastest tell there is.
The 32b flagged 4 of 40. I opened all four. A garbled banner strung over a street scene, a shopfront reading "FRAME D", another row of nonsense signage, and a sports jersey with a melted logo on the chest. All four are real.
The other two models flagged nothing, on the same images, with the same instruction in the prompt.
Worth saying I only checked the four it flagged, not the thirty six it didn't, so that's precision and I have no idea about recall.
Ollama takes a format: "json" parameter that constrains the output to valid JSON, which is the obvious thing to set when you want a score sheet back. On qwen3-vl:32b you get a zero length response. No error, no exception, nothing in the logs, just an empty string where your JSON should be.
I'd originally filed this under "thinking models are weird about structured output", figuring the constraint was fighting the reasoning block. Wrong. The dense non-thinking 32b does exactly the same thing. And qwen3-vl:30b-a3b-instruct handles format: "json" without complaint, so it isn't an ollama-wide thing either. It follows the model.