{"slug": "my-ai-quality-gate-scored-40-images-humor-7-forty-times", "title": "My AI quality gate scored 40 images. Humor: 7, forty times.", "summary": "A developer found that their AI quality gate for locally generated images was effectively a rubber stamp, with a vision model returning identical scores across 40 images. After testing multiple models and prompt variations, they discovered that only a larger thinking model used the full scoring range and caught text defects, while a JSON output parameter in Ollama silently failed on that model.", "body_md": "I generate images locally in batches, and a vision model scores each one before anything ships. Theme, humour, wit, background, one composite number. Anything under the bar gets rebuilt.\n\nThat ran for weeks. Then I dumped the raw scores instead of the pass/fail summary and actually looked at them.\n\n```\n40 images, one slate, qwen2.5vl:7b\n\ntheme    9 on all 40\nhumor    7 on all 40\nwit      8 on 37 of 40\nscore    8.38 to 8.82\n```\n\nHumour came back as 7 forty times in a row. The best image on that slate scored 8.82 and the worst scored 8.38, which is a range of 0.44 on a ten point scale. Nothing was ever going to fail that gate.\n\nBecause the numbers looked fine, which is the whole problem.\n\nThey sat in a sensible range. They had decimals. The composite even wobbled between images, because `bg`\n\nhad some genuine variance and dragged the average around by tenths, so no two images ever printed exactly the same final score. That wobble is what sold it. It looked like a measurement.\n\nIf your judge throws an exception you fix it that afternoon. If it hands back 8.6 for everything, you skim it, think \"yeah, slate's decent\", and run on it for a month.\n\nSo: go and take the standard deviation of each axis over your last run. Five lines. If an axis is flat it isn't measuring anything, and no individual score will ever tell you that.\n\nFirst thing I assumed was that my rubric was too woolly, so I rewrote it with hard numeric anchors and told the thing to be a harsh critic.\n\n```\nHUMOR anchors: 0-2 nothing amusing; 3-5 mildly charming; 6-7 a clear visual joke;\n8-10 genuinely funny, a character is REACTING.\nWIT anchors: 0-2 theme applied as texture; 3-5 sensible pairing; 6-7 real idea;\n8-10 clever twist.\n```\n\nSame model, same 40 images. Humour loosened up and started using 2 through 5. Wit then collapsed instead, 7 on 38 of 40, standard deviation 0.22.\n\nAll I'd done was move the flat axis. That's when I gave up on prompt fixes and went looking for a different model.\n\nSame 40 images, same anchored prompt, three models. I'm quoting standard deviation rather than min to max, because a single outlier makes a rubber stamp look like it's doing something.\n\n| model | humour sd | wit sd | distinct wit scores | text defects found | speed |\n|---|---|---|---|---|---|\n| qwen2.5vl:7b | 0.99 | 0.22 |\n2 | 0 of 40 | 1.0 s |\n| qwen3-vl:30b-a3b-instruct | 0.82 | 1.12 | 6 | 0 of 40 | 2.6 s |\n| qwen3-vl:32b-thinking | 2.54 |\n2.33 |\n10 | 4 of 40 |\n164.7 s |\n\nThe histograms make it more obvious than the summary stats do.\n\n```\nqwen2.5vl:7b           wit      7 on 38 of 40\nqwen3-vl:30b-a3b       humour   5 on 32 of 40\nqwen3-vl:32b-thinking  humour   nine different values, 0 through 8\n```\n\nThe 30b deserves better than I first gave it, though. It isn't a rubber stamp. It reliably drops the images that ignored the brief to a 2 while everything else gets a 5, so there's real pass/fail signal in there. It just can't rank anything that passed, which was the bit I actually wanted.\n\nOnly the 32b-thinking used the range it was handed.\n\nThe other thing I want from a judge is catching mangled text, because generated images produce garbled shop signs constantly and it's the fastest tell there is.\n\nThe 32b flagged 4 of 40. I opened all four. A garbled banner strung over a street scene, a shopfront reading \"FRAME D\", another row of nonsense signage, and a sports jersey with a melted logo on the chest. All four are real.\n\nThe other two models flagged nothing, on the same images, with the same instruction in the prompt.\n\nWorth saying I only checked the four it flagged, not the thirty six it didn't, so that's precision and I have no idea about recall.\n\nThis one cost me an afternoon and I can't find it documented anywhere.\n\nOllama takes a `format: \"json\"`\n\nparameter that constrains the output to valid JSON, which is the obvious thing to set when you want a score sheet back. On `qwen3-vl:32b`\n\nyou get a zero length response. No error, no exception, nothing in the logs, just an empty string where your JSON should be.\n\nFour runs, same image:\n\n``` php\nqwen3-vl:32b           format:json=True   -> len=0      22.9s\nqwen3-vl:32b           format:json=False  -> len=103   121.8s   {\"humor\":8,\"wit\":9,...}\nqwen3-vl:32b-thinking  format:json=True   -> len=0       6.2s\nqwen3-vl:32b-thinking  format:json=False  -> len=106   146.6s   {\"humor\":6,\"wit\":8,...}\n```\n\nDrop the format hint and both give you clean JSON first try.\n\nI'd originally filed this under \"thinking models are weird about structured output\", figuring the constraint was fighting the reasoning block. Wrong. The dense non-thinking 32b does exactly the same thing. And `qwen3-vl:30b-a3b-instruct`\n\nhandles `format: \"json\"`\n\nwithout complaint, so it isn't an ollama-wide thing either. It follows the model.\n\nTwo other bits if you go down this road. Parse the last `{...}`\n\nin the response, not the first, because a thinking model will happily emit braces while it reasons. And give it a real `num_predict`\n\n, around 3072, or it burns the whole budget thinking and never gets to the answer.\n\n`qwen3-vl:30b-a3b-instruct`\n\nsits directly next to `qwen3-vl:32b`\n\nin the library, and the names suggest two roughly comparable models.\n\nThe `a3b`\n\nis doing a lot of quiet work in that tag. It's a mixture of experts with about 3B parameters active per token. The dense 32b runs all 32B. For \"is this actually funny\", which needs the model to hold a whole image against a rubric and then commit to an opinion, that difference is not subtle.\n\nIf you're picking local models off a list by the number in the name, check whether it's total or active first.\n\nI measured spread, not correctness. All this tells you is whether a model can tell your images apart. It says nothing about whether it's right. I have no human labelled ground truth for this set, so a model could hand me a lovely wide spread of confidently wrong numbers and my method would call it a winner. Discrimination is necessary, not sufficient.\n\nOne rubric, one prompt, one domain, my hardware.\n\nThe rubber stamp finding is the solid one. That's 40 rows of real output and the flat axes stay flat regardless of what the model comparison says.\n\nThe good judge is slow. 164.7 seconds an image is nearly two hours for a 40 image slate, which isn't a gate, it's a second render pass. So it runs as a cascade now. The cheap model screens the lot and throws out the obvious failures, which is the one thing it's reliably good at, and only the survivors go to the expensive one. Puts a slate at roughly 25 minutes.\n\nThe smaller change is the one I'd actually recommend. The gate prints the standard deviation of every axis next to the scores now, so if an axis comes back flat I find out on the first run instead of the fortieth.\n\n*I use Claude for a lot of this work, including a hand with the writing. The measurements are mine and they're reproducible, every number above came off my own machine.*", "url": "https://wpnews.pro/news/my-ai-quality-gate-scored-40-images-humor-7-forty-times", "canonical_source": "https://dev.to/booyaka101/my-ai-quality-gate-scored-40-images-humor-7-forty-times-2kn8", "published_at": "2026-08-19 01:08:17+00:00", "updated_at": "2026-08-19 01:42:10.743150+00:00", "lang": "en", "topics": ["artificial-intelligence", "computer-vision", "large-language-models", "ai-tools", "developer-tools"], "entities": ["Ollama", "qwen2.5vl:7b", "qwen3-vl:30b-a3b-instruct", "qwen3-vl:32b-thinking"], "alternates": {"html": "https://wpnews.pro/news/my-ai-quality-gate-scored-40-images-humor-7-forty-times", "markdown": "https://wpnews.pro/news/my-ai-quality-gate-scored-40-images-humor-7-forty-times.md", "text": "https://wpnews.pro/news/my-ai-quality-gate-scored-40-images-humor-7-forty-times.txt", "jsonld": "https://wpnews.pro/news/my-ai-quality-gate-scored-40-images-humor-7-forty-times.jsonld"}}