Grade Your LLM Pass/Fail and You Will Ship a Disaster

An engineer known as ramses203 detailed a severity-based grading system for LLM order-reading tests, arguing that counting correct answers is misleading and that a single irreversible error should block deployment. The approach assigns grades from FATAL to HARMLESS based on reversibility, and the engineer shared lessons from grading bugs, including the importance of saving model answers and writing results to disk incrementally.

I gave my LLM a 29-question order-reading exam. Last time https://dev.to/ramses203/a-good-llm-exam-is-90-traps-4faj was how to build the exam. Today: grading. Grading gets its own post for a reason. Build the grading wrong, and the score lies to you. No idea. Because "which 5" is missing. If it missed 5 typo-riddled questions, ship it. But if one of those 5 was reading "please cancel my order" as a NEW order? Then even with everything else perfect, you can't ship. That program sends goods to a customer who just cancelled. So don't grade by count. Grade by severity. My grader has 4 grades. One criterion — is it reversible? In this program, the irreversible moment is when the wrong goods get loaded onto a truck. FATAL Wrong goods on the truck. Cannot be undone RISKY Confirmed something ambiguous without asking. Right this time — fatal next time MISSED Dropped an order. The customer calls. Fixable HARMLESS Over-asked "please confirm." Just slower One principle falls out of this: A wrong confirmation is worse than no confirmation. Sounds obvious. In production you'll be tempted to flip it. Someone complains "it asks for confirmation too often," so you lower the confidence bar. The screen gets cleaner. And the accidents start happening off-screen. FATAL 0 · MISSED 1 → Ship it. Humans catch what it drops FATAL 1 · everything else perfect → Don't ship. You don't know when that 1 comes back Same score. Opposite fates. The grader is code I wrote. Like all code I write, it had bugs. Accident one — zero points over formatting. A model answer was perfect in content, but the JSON wrapper arrived with the tail cut off. The grader ruled "broken format = fatal." A 100-point answer, zeroed over one missing brace. The fix is simple: count the open brackets and close what's missing ignoring brackets inside strings . The actual code is in parse json in the repo https://github.com/ramses203/llm-test-harness . Accident two — penalizing a good answer. For "250 boxes, 5 units" the model answered: Verdict: needs confirmation Likely candidate: shipping box 250 Reason: if "units" means boxes it's 5 boxes; if sheets, 0.1 box — cannot confirm A considerate answer — asks for confirmation AND offers a hint. But my grader saw the candidate field filled in and ruled "aha, you confirmed " Wrong answer. I fixed it to read the verdict field first. Lesson: when the grader is wrong, you end up "fixing" a healthy model. And every fix makes it worse. Save every model answer to a file. Never throw them away. In this project, what kept changing wasn't the model's answers — it was the grading side. I fixed the answer key three times and the grader twice. Each fix means re-grading all 29 questions. With saved answers, re-grading takes seconds. Without them, one re-grade means calling the model 29 times again. It's the difference between re-marking stored answer sheets and calling every student back to retake the exam. I built this as a --rescore flag. Write results to disk after every single case. In another experiment I collected 5,578 items with a save-at-the-end design. The last request failed and took all 5,000 with it. Paid API — every lost item was money. Learned that one the hard way. When the results come in, the question is not "how many did it get right?" It's "among the failures, is anything irreversible?" That's why I only really read one line of the grader's output. Fatal = 0: ship. Fatal = 1: don't — even if everything else is perfect. P.S. Next up: the model that costs 3x more won by exactly one question. All code and the 29 questions are public → github.com/ramses203/llm-test-harness