{"slug": "a-good-llm-exam-is-90-traps", "title": "A Good LLM Exam Is 90% Traps", "summary": "A developer known as ramses203 detailed the design of a 29-question exam for an LLM-based order-reading program, emphasizing that effective tests focus on edge cases and traps rather than normal scenarios. The exam includes questions about non-orders, changes, cancellations, ambiguities, typos, and learning scenarios, with the goal of preventing accidental shipments. The developer also released the exam and catalog publicly on GitHub.", "body_md": "[Last time](https://dev.to/ramses203/i-gave-my-llm-an-exam-the-exam-author-lost-5-times-12b0) I gave my LLM an order-reading exam and lost 5 times as the exam author.\n\nToday: how that exam was built. Conclusion first — nice questions are a waste of paper.\n\nAsk anyone to write a test and they start with the case that works. \"5 boxes of the 250 shipping boxes please\" → shipping box 250, 5 boxes. It passes. Feels good. Reassuring.\n\nBut that's wasted points. **Models rarely fail the normal cases. What fails is everything that isn't normal.**\n\nMy 29 questions broke down like this:\n\n```\nNormal orders               4\nThings that aren't orders   6   ← the biggest group\nChanges & cancellations     4\nAmbiguous ones              5\nTypos & extreme shorthand   3\nAfter learning kicks in     7\n```\n\nNormal is the smallest group. On purpose.\n\nThe worst accident for this program is **shipping something nobody ordered.** So the exam should aim at that accident more than anything else.\n\nWhat are the dimensions of the 250 shipping box?\n\nProduct name: present. Number: present. But it's not an order. It's a question.\n\nA program that treats \"product name spotted\" as \"order detected\" calls the truck right here. So I planted six of these: price inquiries, stock inquiries, delivery questions, greetings, a tax-invoice request.\n\nChanges and cancellations are nastier.\n\nI ordered 5 boxes of the 250 — please send only 3\n\nTwo numbers. Read only the first half and it's a perfect order. Treat it as a new order and the goods ship twice.\n\nIt's not just about hard questions. **Make the data itself messy.**\n\nOne reason: **real data already looks like this.** A real product catalog always has near-twins.\n\nRun the exam on a clean catalog and here's what happens — everything passes. Then you plug in production data and it collapses. If the exam passed but production has accidents, that's not the model's fault. That's the exam's fault.\n\nThis program learns. When a human picks a match once, it remembers. \"250\" → shipping box 250. Automatic from then on.\n\nConfession: my first 22 questions had zero learning scenarios. **\"Fix it once and it's automatic afterwards\" is this program's reason to exist, and I hadn't tested that path even once.** I added 7 questions late.\n\nAnd those 7 questions showed me something scary. Learning isn't just a convenience feature — it can be **an accident-generating feature.**\n\n**Trap one.** The program has learned \"tape = 48mm.\" Then this arrives:\n\ntape 60, 2 boxes\n\nApply the learned match as-is and 48mm ships. The customer said 60. **An explicit spec must beat the learned match.**\n\n**Trap two.** The program has learned \"250 = shipping box.\" Then:\n\nWhat are the dimensions of the 250?\n\nThe more it learns, the more confident the program gets. Use that confidence to read a question as an order and it's over. **Learned or not, a question is a question.**\n\nIt passed both. Good. But if I had never written these questions? It would have gone to production with nobody knowing whether it passes them.\n\nIf the exam is nice, the exam passes — and production has the accidents.\n\n*P.S. All 29 questions and the trap-laden catalog are public → github.com/ramses203/llm-test-harness*\n\n*Next up: grading — why pass/fail grading will wreck you.*", "url": "https://wpnews.pro/news/a-good-llm-exam-is-90-traps", "canonical_source": "https://dev.to/ramses203/a-good-llm-exam-is-90-traps-4faj", "published_at": "2026-08-20 00:04:15+00:00", "updated_at": "2026-08-20 00:43:50.882666+00:00", "lang": "en", "topics": ["large-language-models", "ai-products", "ai-safety", "developer-tools"], "entities": ["ramses203", "GitHub", "llm-test-harness"], "alternates": {"html": "https://wpnews.pro/news/a-good-llm-exam-is-90-traps", "markdown": "https://wpnews.pro/news/a-good-llm-exam-is-90-traps.md", "text": "https://wpnews.pro/news/a-good-llm-exam-is-90-traps.txt", "jsonld": "https://wpnews.pro/news/a-good-llm-exam-is-90-traps.jsonld"}}