The model card is not the artifact you're serving

A developer argues that teams benchmarking and deploying open-weight models must pin the exact artifact — repo commit hash and file digest — rather than a model name or tag, because the community GGUF quant most people actually serve is not the vendor's reference build. The writeup recommends recording the sha256 of the .gguf alongside eval scores, diffing the prompt template embedded in GGUF metadata against the vendor's tokenizer_config.json, pinning the full serving envelope (context size, KV cache type, rope scaling, batch size, offload, sampler, seed), and measuring latency separately from quality. The developer notes aggressive quantization degrades unevenly, with long structured output drifting before plain reasoning.

I keep a baselines.md in every agent repo. It has one rule at the top that I wrote after a phantom regression ate a week of my evenings: a baseline is a file, not a model name. This week's reminder is that the DeepSeek V4.1 build most people are actually pulling down isn't DeepSeek's. Search the model name on any hub and the thing that surfaces first, the thing that gets mirrored into every tutorial and every starter compose file, is a third-party mixed-precision GGUF. The vendor's official card is sitting right there and almost nobody is serving it. That's not a scandal, it's just how open weights work — the quant lands, the community makes it run on consumer hardware, and the runtime gets adopted before the reference implementation does. The problem is what that does to your numbers. If you benchmarked on the vendor's card in bf16 on an A100 and then shipped the community 4-bit GGUF on a 4090, your regression suite is comparing two different artifacts and calling the difference "quality." It isn't. It's a quantization delta, a runtime delta, a template delta, and a hardware delta all stacked on top of each other, and you have no way to attribute any of it. I've watched a team spend a week tuning prompts because their agent's tool-calling accuracy fell off a cliff after "upgrading the model." They'd upgraded the quant type. Same weights, different file. The prompts were fine. So here's the tip, and it's boring on purpose. Tags move. latest moves. The filename ...-Q4 K M.gguf can be re-uploaded under the same name with different tensors if the maintainer re-runs the quant. If your deployment config says "pull model X, 4-bit," you do not have a reproducible system, you have a wish. Pin the commit hash on the repo and the digest of the file. In llama.cpp, -hf user/repo:quant accepts a revision — use it. In vLLM, --revision <sha on the HF repo. In Ollama, pin by digest, not by tag, and check what you actually have with ollama show --modelfile rather than trusting the name you typed. Then write the sha256 of the .gguf into your eval output, next to the score. Not in a wiki. In the eval output, so it's impossible to read a number without also reading what produced it. If you can't reproduce the exact bytes six weeks from now, the number means nothing. This is the one that keeps biting people. A GGUF carries its prompt template in metadata. If the quant repo's template drifted from the vendor's tokenizer config.json — a missing BOS, a different system-prompt wrapper, tool-call tokens in a slightly different order — the weights are identical and the model still behaves like a different model. Your tool-calling agent will look dumber and you'll blame the quant. Dump the template from the artifact, diff it against what you expected, and put that diff in the PR. It takes minutes. Quantization changes memory footprint, which changes what you can afford to keep in KV cache, which changes your context length, which changes your agent's behavior on long tasks. So the eval config needs the whole serving envelope pinned, not just the model: context size, KV cache type, rope scaling, batch size, offload settings, sampler, seed. I run evals greedy with temperature 0 and a fixed seed, because I want my failure to be my fault and not a dice roll. And measure latency separately from quality. A 4-bit quant that scores the same as bf16 but blows up your time-to-first-token under batch is not a win, it's a relocation of the cost. Two tables, always. The honest way to know what the quant cost you is to have both files on disk and route the same prompt set through both in the same harness on the same day. Shadow traffic if you can, a fixed offline set of real prompts if you can't. What I look for isn't the average score — averages hide everything — it's which categories of task get worse. In my experience the damage from aggressive quantization is uneven: plain reasoning holds up, long structured output drifts first, and multi-turn tool calling is where things quietly fall apart, because errors compound across turns and one malformed argument kills the whole trajectory. I haven't pushed V4.1 deep into a long-context agent loop yet, so treat that as the thing I'd check first rather than a finding. Mixed-precision quants keep certain tensors at higher precision, and which ones get protected varies by recipe, so "4-bit" from one repo is not comparable to "4-bit" from another. That's another reason the digest matters more than the label. Fair warning, because I've been on both sides of this. Pinning means you don't get silent upstream fixes. So schedule a re-test — monthly is fine — where you deliberately unpin, run the suite, and decide whether to move the pin. That's a config change with a diff and an eval attached, which is exactly what you wanted anyway. The vendor card is the reference implementation. The community quant is the production artifact. Most of us are serving the second one and reporting the first one's numbers. Write down which file you ran, and the argument you have next quarter will be about the code instead of about the download.