I started evaluating DeepSeek models the lazy way. The API speaks the same /v1/chat/completions shape as OpenAI, so I swapped the base_url, pointed my existing eval suite at R1, and waited for a score. I got one, and it told me almost nothing. That suite was written for a closed model. It scored the final answer only, in one language, against one build of the weights.
DeepSeek is cheap and the reasoning is strong. Neither fact tells you whether it can replace the model you run today, and three gaps in my rubric showed up before anything else did.
R1 returns a reasoning_content field next to the final content. You can read it, store it, and score it. My rubric ignored it.
The failure I care about lives between those two fields: the trace is correct and the answer is wrong. On hard reasoning sets that happens on eight to fifteen percent of R1 traces. An answer-only check sees a wrong answer and has no idea the reasoning was fine. So I added a separate judge that gets the trace and the answer together and scores one thing, whether the answer follows from the trace.
Open weights mean most teams self-host, and self-hosting usually means Q4, Q5, or Q8 on vLLM, SGLang, or TGI. My benchmark numbers came from the FP16 reference. Production was going to run Q4.
The two builds agree on ninety-plus percent of cases. The ten percent where they disagree are the cases that matter. I now run the same prompt set through both and score the delta as its own axis.
DeepSeek is trained bilingually, and its English and Chinese quality split unevenly on long-tail tasks. An English-only golden set misses Chinese regressions that bilingual users notice on day one. A safety-block rate measured only in English hides the false positives Western-trained classifiers throw on Chinese text.
I stratify every rubric by language now. For Chinese output I pair LlamaGuard with a Qwen3Guard model in a majority vote, since Qwen3Guard throws fewer false positives on Chinese content.
Jailbreaks succeed more often on R1 and V3 than on Claude or GPT-5. The same models over-refuse medical-adjacent, legal-adjacent, and security-research questions the incumbent answers cleanly, and no accuracy benchmark will show you that. I run prompt injection, jailbreaks, refusal calibration, and system-prompt leakage against both models on identical payloads. Any regression on refusal or injection blocks the swap.
I wrote up the full four-gate method, with the winrate floors and the canary rollout, in my DeepSeek evaluation playbook.
Which of these would have caught you first on an open-weight swap? For me it was the trace: a correct chain of reasoning sitting on top of a wrong answer, and a rubric that never looked at it.