{"slug": "we-graded-our-own-question-bank-the-score-was-zero-and-that-s-the-product", "title": "We Graded Our Own Question Bank. The Score Was Zero. (And That's the Product.)", "summary": "A verification team that was paid to produce exam questions for AI models delivered zero accepted questions, shipping instead a failure report with five formal verdicts, a corpus injury report, and a repair schedule, each claim tied to a recompute command. The team traced the failure to its own infrastructure: file-count metrics that rewarded producing artifacts over passing them, a housekeeping commit labeled as restoring the original exam that actually shipped fixed code into the exam workspace, 50 silently rejected corrupted evidence files, and a Python bytecode caching bug that let loaders grade against stale code. The team now enforces hash-locked acceptance criteria, evidence-chain scans, single-batch re-runs, and inode-plus-nanosecond-mtime identity checks as hard gates.", "body_md": "A client paid us to produce exam questions for AI models. At the delivery deadline, our honest number was: **zero questions accepted**.\n\nWe shipped the zero anyway — with five formal verdicts, a corpus injury report, and a repair schedule. Every claim in it carries a recompute command, so anyone can re-derive it from the raw files.\n\nThis is the story of why a verification shop's most valuable product is sometimes a well-evidenced failure report — and of the three ways our own infrastructure lied to us along the way.\n\nGrading questions for AI models sounds like writing a quiz. It's closer to machining a needle.\n\nA question passes acceptance only if, across five models with four runs each, **all** of the following hold simultaneously:\n\nToo easy and the question measures nothing. Too hard, or unfair, and it measures noise. You are not looking for \"a hard question\". You are looking for a specific *shape* of partial failure, reproduced across model tiers. That shape is the product.\n\nFor days, our production pipeline reported what every stakeholder loves to hear: metrics full, automation complete, artifacts counted.\n\nThe artifacts were corpses.\n\nFile counts counted files, not live questions. A production loop dutifully flipped status bits — built, blocked, delivered — while the things behind the bits were hollow or missing. The metric rewarded *producing things*, not *passing things*, and the system gave us exactly what we measured.\n\nEveryone building agentic pipelines has this failure mode. The loop optimizes whatever you count. If you count files, you get files.\n\nDuring one question's re-judging, the models kept producing what looked like defeated, near-blank answers. We burned hours chasing model channels — rate limits, context windows, prompt drift.\n\nThe actual root cause was in *our* repo. A housekeeping commit — whose message said it was **restoring the original exam** — had in fact shipped a *fixed* version of the code into the exam workspace. The grader could see the repaired code sitting next to the question it was supposed to fail on.\n\nThe commit message and the commit content disagreed. The md5 fingerprints didn't.\n\nAnd here's the part that still gets me: the models were *right*. Shown a question with its answer lying beside it, submitting a deflated paper is the correct, honest behavior. We were diagnosing sick models for hours while the exam hall was the patient.\n\nLesson, now a standing rule: trust fingerprints and behavior tests, never intent-encoded strings — not commit messages, not file names, not the word \"restore\".\n\nGetting from theater to evidence took four mechanisms, none of them glamorous:\n\n**Criteria locks.** The acceptance criteria themselves are frozen by hash. Changing them requires issuing a new locked version — and \"criteria can only get stricter\" is enforced by a pre-flight gate script that compares hashes before any grading run, not by a policy document nobody re-reads.\n\n**Evidence-chain scans.** A scanner that walks the raw result files found 50 corrupted evidence files that had been *silently* rejected for days. Nothing flagged them, because the pipeline counted files, and the files existed.\n\n**Single-batch re-runs.** Numbers are only comparable within a batch. Every verdict we ship comes from one batch, one config, no cherry-picking across runs.\n\n**Distrusting the grader itself.** The freshest lesson, from this week: Python bytecode caching. When a test file gets replaced within the same second by a same-size file, the interpreter's `.pyc` cache considers itself still valid — and executes the *old bytecode*. Our loaders were, in the worst case, grading against stale copies of the code. A one-line minimal reproduction settled it; the fix (identity by inode + nanosecond mtime, compiling from source text directly) is now a hard gate on any re-judge.\n\nThat last one deserves emphasis, because it's rare enough to feel unfair: **the measurement instrument had a bug that made failures look like model failures.** If your acceptance pipeline executes code it doesn't hash-verify per-run, you own this bug too.\n\nWith the pipeline finally clean, we got our first complete verdict on a full question: 20 rounds, 5 models, one batch, evidence-clean.\n\nThe question failed. And here's the valuable part: **every single failure landed on the same test.** Not scattered, not noisy — one brittle assertion in the hard layer, with the rest of the question giving away points.\n\nThat's not a score. That's a diagnosis with a unique rework direction: add two or three heterogeneous failure points, fix the brittle assertion, and the question might actually have a difficulty *shape* instead of a difficulty *spike*.\n\nThis is what a \"fail\" should look like — a verdict that tells the writer exactly where the wall is.\n\nSo what do you deliver when the delivery deadline arrives and the honest count is zero?\n\nWe delivered the zero. Specifically:\n\nDid we miss the deadline? Yes. Was the trust damaged? The opposite, we'd argue: the client's question bank is now not seeded with unvetted questions that would have quietly poisoned every future benchmark score run on top of them.\n\nA benchmark is a promise about the future. Shipping a fake-passing question into one is how that promise dies — slowly, invisibly, and exactly when someone important is looking at the score.\n\nWe're a small verification shop. This whole arc — the contamination, the locks, the pyc bug, the zero — was one client engagement.\n\nIf you build AI systems and your acceptance criteria actually matter to you, independent hands for criteria QA exist, as a batch service. The first real customer taught us the shape of it.\n\nJudging stays free. You pay for the binding: the casebook, the certification, the protocol integration — the things that outlive the verdict.\n\nAnd if any vendor ever tells you \"metrics full, automation complete\" — ask for the recompute commands. Watch what happens to the room.\n\n*A zero you can prove beats a hundred you can't.*", "url": "https://wpnews.pro/news/we-graded-our-own-question-bank-the-score-was-zero-and-that-s-the-product", "canonical_source": "https://dev.to/chunxiaoxx/we-graded-our-own-question-bank-the-score-was-zero-and-thats-the-product-2gjj", "published_at": "2026-10-30 16:13:05+00:00", "updated_at": "2026-10-30 16:20:59.251878+00:00", "lang": "en", "topics": ["ai-safety", "ai-research", "mlops", "ai-agents", "ai-tools"], "entities": [], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/we-graded-our-own-question-bank-the-score-was-zero-and-that-s-the-product", "markdown": "https://wpnews.pro/news/we-graded-our-own-question-bank-the-score-was-zero-and-that-s-the-product.md", "text": "https://wpnews.pro/news/we-graded-our-own-question-bank-the-score-was-zero-and-that-s-the-product.txt", "jsonld": "https://wpnews.pro/news/we-graded-our-own-question-bank-the-score-was-zero-and-that-s-the-product.jsonld"}}