# Steal This Exam. Here's How to Port It to Your Own Pipeline. > Source: > Published: 2026-08-25 00:27:34+00:00 So far this series has been about [giving my order-reading LLM an exam](https://dev.to/ramses203/i-gave-my-llm-an-exam-the-exam-author-lost-5-times-12b0). Some of you have been reading it thinking: "Mine isn't orders, it's meeting-minutes summarization." "I'm using it for email triage." Good news — the exam is built to be stolen. Only two things need swapping: **the reference data you match against** (mine: a product catalog) and **the worst accident** (mine: wrong goods loaded onto a truck). Let's go in order. Before writing a single test question, write this: **When this AI is wrong, which of the consequences cannot be undone?** For my order program it was "the wrong goods get loaded onto a truck." What is it for yours? A few examples — ``` auto-reply email AI a wrong answer goes out to the customer — you can't unsend it meeting-minutes AI writes "decided" on something undecided — once shared, work proceeds on it research AI an invented number enters the report — reports travel upward data-cleanup AI overwrites the original — no backup, no recovery ``` You can see the pattern. **Sent, deleted, escalated — these don't come back.** That's your truck. Once the worst is written down, the grades fall out on their own. One criterion — can a human undo it? ``` FATAL cannot be undone sent, deleted, reported, charged RISKY confirmed something ambiguous right this time, fatal next time MISSED dropped something a human can still catch it HARMLESS over-asks "please confirm" just slower ``` The only line you have to define yourself is the first one. The other three read the same in any project. And take the principle with you as-is: **a wrong confirmation is worse than no confirmation.** Exam questions come from traps, not from normal cases. There are four kinds, and every domain has all four. **Confusable pairs** — mine was clear tape 48mm vs 60mm. In meeting minutes: two attendees named Kim, one an associate, one a manager. In email: reply vs forward. Find the pairs in your data that are similar enough to confuse, and deliberately put both in. **Plausible non-targets** — "What are the specs on the 250 shipping box?" contains a product name but is not an order. In minutes: "let's decide that next time" (not a decision). In email: a promotional mail (not something to answer). Put in **things that only look like what the AI is supposed to catch.** **Mid-message reversals** — "5 boxes please — no wait, make it 3." In minutes: "let's go with A → actually B is better." Questions that are wrong if you only read the first half. **Accidents after learning** — mandatory if your system has memory. A situation where fresh information must beat the remembered value ("tape 60"), and a situation where memory must not change the verdict (a question is still a question). How many questions? The metric isn't a count — it's your accident list from Step 1. **At least one question per accident type, built to cause exactly that accident.** From my list: ``` inquiry mistaken for an order → goods nobody ordered ship → "What are the specs on the 250 shipping box?" change/cancel mistaken for a new order → ships twice → "I ordered 5 boxes — please send only 3" unit misread → quantity ships at 50x → "250 5" ``` Five accidents on your list means five questions minimum. Mine grew to 29 because I kept adding variants per accident; 10 is plenty to start. Add a few normal, well-behaved cases at the end — only a few. Models rarely fumble the easy ones, so normal cases mostly catch nothing. The exam's job is not to watch the model succeed. It's to find where the accidents are. For each question, write the expected answer. And here comes the lesson of [the first post](https://dev.to/ramses203/i-gave-my-llm-an-exam-the-exam-author-lost-5-times-12b0): **Your answer key will be wrong.** I was wrong three times in the key and twice in the grader. So keep three rules. The third rule by example — "tape 60, 2 boxes": the catalog has a 60mm, so the data pins the answer. If the model answers differently here, the model is wrong; the key stands. "Clear tape, 2 boxes": no width given, the data cannot pin it. The correct answer was "needs confirmation" from the start — if I wrote anything else in the key, the key is what's broken. A reader of the first post talked me into this flag. Bonus effect: recording it forces you to open the reference data for every question while you're still writing the exam — which catches answer-key mistakes at authoring time. Count results by grade, not by score. And save the model's raw answer sheets to files — grading criteria keep changing, and with the raw sheets you can re-grade without calling the model again. Everything passed? Then write the three-line ledger. It's you being honest with yourself about what the exam guarantees and what it doesn't. ``` VERIFIED one question per accident type, and FATAL held at zero NOT VERIFIED production data has never been through it — I invented every sentence myself GUARDED on anything uncertain it must not confirm — it hands off to a human ``` What justifies shipping is not line one. It's **lines one and three combined.** The exam score covers only what I could imagine (line two is that confession), and the sentences I couldn't imagine land on line three's guard. Without line three, an exam score is just a number — the first unimagined sentence causes the accident. The exam you may steal. The answer key you have to write yourself. And that answer key will be wrong. Catching it is what the exam is for. *P.S. The test runner and grader that run exactly this structure are public → github.com/ramses203/llm-test-harness*