The results show a massive performance cliff based on model scale.
Experimental Setup #
I tested 20 scenarios across four categories: Explicit Directional Failures (DF), Subtle DF, Valid controls, and Garbage controls. Each scenario was run multiple times to check for consistency.
Models used:
qwen3:0.5b: Local (Ollama)
gemma3:latest: Local (Ollama)
deepseek-v4-flash: API
The judge prompt was kept strictly minimal with temperature 0 to avoid rubric bias:
Judge whether this output satisfies the task.
Output: {output}
Task: {task}
Respond in JSON: {"passes": bool, "confidence": 0-1}
Performance Breakdown #
The gap between small local models and larger frontier/distilled models is jarring.
Global Accuracy: qwen3:0.5b (61.5%) vs. gemma3/deepseek-v4-flash (both 92.0%)Explicit DF: qwen3:0.5b missed about 36% of direct contradictions, while the larger models hit 100%.Subtle DF: This is where it gets messy. qwen3:0.5b failed nearly half the time (44% miss rate). gemma3 was better but still missed ~10%. DeepSeek-v4-flash was the most robust here with only a 2% miss rate.Controls: Valid and garbage controls were handled well across the board, though DeepSeek oddly missed 10% of valid controls.
The Verdict #
If you're building an AI workflow that relies on an LLM agent to grade other LLMs, do not use anything under 4B parameters for the judge.
Models below 1B are essentially guessing when the failure is subtle. Once you hit the 4B+ range, explicit contradictions are caught perfectly, and subtle semantic reversals become much easier to detect. For high-stakes deployment, a larger API-based model like DeepSeek-v4-flash is significantly more reliable for catching those "plausible but wrong" outputs.
Next EveAgents: Open-Source AI Agent Deployment Guide →