{"slug": "steal-this-exam-here-s-how-to-port-it-to-your-own-pipeline", "title": "Steal This Exam. Here's How to Port It to Your Own Pipeline.", "summary": "A developer shares a method for porting an LLM evaluation exam to any pipeline, emphasizing the need to identify the worst irreversible accident and design test questions around it. The approach includes four trap types and a grading rubric based on human undoability, with the warning that answer keys will be wrong.", "body_md": "So far this series has been about [giving my order-reading LLM an exam](https://dev.to/ramses203/i-gave-my-llm-an-exam-the-exam-author-lost-5-times-12b0).\n\nSome of you have been reading it thinking: \"Mine isn't orders, it's meeting-minutes summarization.\" \"I'm using it for email triage.\"\n\nGood news — the exam is built to be stolen. Only two things need swapping: **the reference data you match against** (mine: a product catalog) and **the worst accident** (mine: wrong goods loaded onto a truck). Let's go in order.\n\nBefore writing a single test question, write this:\n\n**When this AI is wrong, which of the consequences cannot be undone?**\n\nFor my order program it was \"the wrong goods get loaded onto a truck.\" What is it for yours? A few examples —\n\n```\nauto-reply email AI    a wrong answer goes out to the customer — you can't unsend it\nmeeting-minutes AI     writes \"decided\" on something undecided — once shared, work proceeds on it\nresearch AI            an invented number enters the report — reports travel upward\ndata-cleanup AI        overwrites the original — no backup, no recovery\n```\n\nYou can see the pattern. **Sent, deleted, escalated — these don't come back.** That's your truck.\n\nOnce the worst is written down, the grades fall out on their own. One criterion — can a human undo it?\n\n```\nFATAL      cannot be undone              sent, deleted, reported, charged\nRISKY      confirmed something ambiguous  right this time, fatal next time\nMISSED     dropped something              a human can still catch it\nHARMLESS   over-asks \"please confirm\"     just slower\n```\n\nThe only line you have to define yourself is the first one. The other three read the same in any project.\n\nAnd take the principle with you as-is: **a wrong confirmation is worse than no confirmation.**\n\nExam questions come from traps, not from normal cases. There are four kinds, and every domain has all four.\n\n**Confusable pairs** — mine was clear tape 48mm vs 60mm. In meeting minutes: two attendees named Kim, one an associate, one a manager. In email: reply vs forward. Find the pairs in your data that are similar enough to confuse, and deliberately put both in.\n\n**Plausible non-targets** — \"What are the specs on the 250 shipping box?\" contains a product name but is not an order. In minutes: \"let's decide that next time\" (not a decision). In email: a promotional mail (not something to answer). Put in **things that only look like what the AI is supposed to catch.**\n\n**Mid-message reversals** — \"5 boxes please — no wait, make it 3.\" In minutes: \"let's go with A → actually B is better.\" Questions that are wrong if you only read the first half.\n\n**Accidents after learning** — mandatory if your system has memory. A situation where fresh information must beat the remembered value (\"tape 60\"), and a situation where memory must not change the verdict (a question is still a question).\n\nHow many questions? The metric isn't a count — it's your accident list from Step 1. **At least one question per accident type, built to cause exactly that accident.** From my list:\n\n```\ninquiry mistaken for an order → goods nobody ordered ship   → \"What are the specs on the 250 shipping box?\"\nchange/cancel mistaken for a new order → ships twice        → \"I ordered 5 boxes — please send only 3\"\nunit misread → quantity ships at 50x                        → \"250 5\"\n```\n\nFive accidents on your list means five questions minimum. Mine grew to 29 because I kept adding variants per accident; 10 is plenty to start.\n\nAdd a few normal, well-behaved cases at the end — only a few. Models rarely fumble the easy ones, so normal cases mostly catch nothing. The exam's job is not to watch the model succeed. It's to find where the accidents are.\n\nFor each question, write the expected answer. And here comes the lesson of [the first post](https://dev.to/ramses203/i-gave-my-llm-an-exam-the-exam-author-lost-5-times-12b0):\n\n**Your answer key will be wrong.** I was wrong three times in the key and twice in the grader.\n\nSo keep three rules.\n\nThe third rule by example — \"tape 60, 2 boxes\": the catalog has a 60mm, so the data pins the answer. If the model answers differently here, the model is wrong; the key stands. \"Clear tape, 2 boxes\": no width given, the data cannot pin it. The correct answer was \"needs confirmation\" from the start — if I wrote anything else in the key, the key is what's broken.\n\nA reader of the first post talked me into this flag. Bonus effect: recording it forces you to open the reference data for every question while you're still writing the exam — which catches answer-key mistakes at authoring time.\n\nCount results by grade, not by score. And save the model's raw answer sheets to files — grading criteria keep changing, and with the raw sheets you can re-grade without calling the model again.\n\nEverything passed? Then write the three-line ledger. It's you being honest with yourself about what the exam guarantees and what it doesn't.\n\n```\nVERIFIED       one question per accident type, and FATAL held at zero\nNOT VERIFIED   production data has never been through it — I invented every sentence myself\nGUARDED        on anything uncertain it must not confirm — it hands off to a human\n```\n\nWhat justifies shipping is not line one. It's **lines one and three combined.** The exam score covers only what I could imagine (line two is that confession), and the sentences I couldn't imagine land on line three's guard. Without line three, an exam score is just a number — the first unimagined sentence causes the accident.\n\nThe exam you may steal. The answer key you have to write yourself.\n\nAnd that answer key will be wrong. Catching it is what the exam is for.\n\n*P.S. The test runner and grader that run exactly this structure are public → github.com/ramses203/llm-test-harness*", "url": "https://wpnews.pro/news/steal-this-exam-here-s-how-to-port-it-to-your-own-pipeline", "canonical_source": "https://dev.to/ramses203/steal-this-exam-heres-how-to-port-it-to-your-own-pipeline-36oo", "published_at": "2026-08-25 00:27:34+00:00", "updated_at": "2026-08-25 01:14:08.773327+00:00", "lang": "en", "topics": ["large-language-models", "ai-safety", "ai-products", "developer-tools"], "entities": [], "alternates": {"html": "https://wpnews.pro/news/steal-this-exam-here-s-how-to-port-it-to-your-own-pipeline", "markdown": "https://wpnews.pro/news/steal-this-exam-here-s-how-to-port-it-to-your-own-pipeline.md", "text": "https://wpnews.pro/news/steal-this-exam-here-s-how-to-port-it-to-your-own-pipeline.txt", "jsonld": "https://wpnews.pro/news/steal-this-exam-here-s-how-to-port-it-to-your-own-pipeline.jsonld"}}