{"slug": "evaluating-deepseek-models-broke-three-assumptions-my-rubric-was-built-on", "title": "Evaluating DeepSeek models broke three assumptions my rubric was built on", "summary": "A developer evaluating DeepSeek's R1 and V3 models found that a standard answer-only eval suite missed key failure modes of open-weight models, including correct reasoning traces paired with wrong final answers (occurring on 8-15% of hard reasoning traces), quantization gaps between FP16 reference and Q4 production builds, and uneven English/Chinese quality on long-tail tasks. The engineer added a trace-and-answer judge, a quantization delta axis, language-stratified rubrics, and safety gates covering jailbreaks, prompt injection, refusal calibration, and system-prompt leakage, documenting the four-gate method in a DeepSeek evaluation playbook.", "body_md": "I started evaluating DeepSeek models the lazy way. The API speaks the same `/v1/chat/completions` shape as OpenAI, so I swapped the `base_url`, pointed my existing eval suite at R1, and waited for a score. I got one, and it told me almost nothing. That suite was written for a closed model. It scored the final answer only, in one language, against one build of the weights.\n\nDeepSeek is cheap and the reasoning is strong. Neither fact tells you whether it can replace the model you run today, and three gaps in my rubric showed up before anything else did.\n\nR1 returns a `reasoning_content` field next to the final `content`. You can read it, store it, and score it. My rubric ignored it.\n\nThe failure I care about lives between those two fields: the trace is correct and the answer is wrong. On hard reasoning sets that happens on eight to fifteen percent of R1 traces. An answer-only check sees a wrong answer and has no idea the reasoning was fine. So I added a separate judge that gets the trace and the answer together and scores one thing, whether the answer follows from the trace.\n\nOpen weights mean most teams self-host, and self-hosting usually means Q4, Q5, or Q8 on vLLM, SGLang, or TGI. My benchmark numbers came from the FP16 reference. Production was going to run Q4.\n\nThe two builds agree on ninety-plus percent of cases. The ten percent where they disagree are the cases that matter. I now run the same prompt set through both and score the delta as its own axis.\n\nDeepSeek is trained bilingually, and its English and Chinese quality split unevenly on long-tail tasks. An English-only golden set misses Chinese regressions that bilingual users notice on day one. A safety-block rate measured only in English hides the false positives Western-trained classifiers throw on Chinese text.\n\nI stratify every rubric by language now. For Chinese output I pair LlamaGuard with a Qwen3Guard model in a majority vote, since Qwen3Guard throws fewer false positives on Chinese content.\n\nJailbreaks succeed more often on R1 and V3 than on Claude or GPT-5. The same models over-refuse medical-adjacent, legal-adjacent, and security-research questions the incumbent answers cleanly, and no accuracy benchmark will show you that. I run prompt injection, jailbreaks, refusal calibration, and system-prompt leakage against both models on identical payloads. Any regression on refusal or injection blocks the swap.\n\nI wrote up the full four-gate method, with the winrate floors and the canary rollout, in [my DeepSeek evaluation playbook](https://futureagi.com/blog/evaluating-deepseek-models-2026/?utm_source=dailydev&utm_medium=organic&utm_campaign=blogdistribution).\n\nWhich of these would have caught you first on an open-weight swap? For me it was the trace: a correct chain of reasoning sitting on top of a wrong answer, and a rubric that never looked at it.", "url": "https://wpnews.pro/news/evaluating-deepseek-models-broke-three-assumptions-my-rubric-was-built-on", "canonical_source": "https://dev.to/kartik-nvjk/evaluating-deepseek-models-broke-three-assumptions-my-rubric-was-built-on-333a", "published_at": "2026-10-15 16:09:37+00:00", "updated_at": "2026-10-15 16:21:29.884376+00:00", "lang": "en", "topics": ["large-language-models", "ai-research", "ai-safety", "mlops", "ai-tools"], "entities": ["DeepSeek", "R1", "V3", "vLLM", "SGLang", "TGI", "LlamaGuard", "Qwen3Guard"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/evaluating-deepseek-models-broke-three-assumptions-my-rubric-was-built-on", "markdown": "https://wpnews.pro/news/evaluating-deepseek-models-broke-three-assumptions-my-rubric-was-built-on.md", "text": "https://wpnews.pro/news/evaluating-deepseek-models-broke-three-assumptions-my-rubric-was-built-on.txt", "jsonld": "https://wpnews.pro/news/evaluating-deepseek-models-broke-three-assumptions-my-rubric-was-built-on.jsonld"}}