In the first post of this series I wrote about making a 29-question exam and getting it wrong five times myself — three times in the answer key, twice in the grader. Ever since, one question kept nagging me: if an AI wrote the exam, how many times would it get it wrong?
So I counted.
Hamel Husain's evals essay — the closest thing this field has to a textbook — describes how to scale up test questions: split the feature into scenarios, then mass-generate the input sentences with an AI. After putting an LLM in the grading seat, this was the next piece to test for real.
I gave the author AI (Sonnet 5) three things:
Five questions per request, ten requests, fifty questions.
Review came in three layers:
layer 1 code check nonexistent product codes? flag contradicting the key?
layer 2 AI reviewer per question: "is this answer justified by the data?"
layer 3 human (me) full re-read of all 50
Layer 3 has a story. The original design stopped at two layers — and only after running it did I notice: the author was Sonnet, and the reviewer was Sonnet. I had written "never let a model grade its own answers" in the judge post, then made exactly that mistake. So I sat down and re-read all fifty myself.
structural errors (fake codes, flag contradictions) 0
answer-key errors (all three review layers) 0
I, the human author, got the answer key wrong three times in 29 questions. The AI author got it wrong zero times in 50. Every fully-specified order had the right code and quantity. Every undecidable one went to "needs confirmation" with the correct candidates attached. All fifty flags matched their answers.
The questions weren't bad, either. Take this one —
Boss, please send 3 boxes of the post-office boxes~ same size as last time!
It planted the fake hint "same as last time" on its own. There's no order history in the data, so nothing can pin the size down — and its answer key says exactly that: needs confirmation, three post-office box candidates. Correct. It reinvented a trap from my original exam ("the usual, 3 boxes") without being told to.
If the story ended here, the conclusion would be "let the AI write the exams."
Nothing in the 50 questions is wrong. But lay them side by side and something else shows.
First, the composition is a photocopy of my instructions. I had told it: "per 5 questions, mix 1 normal order, 1 non-order, 1 change/cancel, 2 traps." Ten requests later, the counts came out exactly 10 / 10 / 10 / 20. The 20 traps split into a tidy 10 missing-spec and 10 mid-message-reversal. A human author would have drifted — "shouldn't something like this go in too?" — and produced a few questions that escape the plan. There isn't one.
Second, the materials repeat. Counting the products in the confirmed answers, one item — shipping box A-type 300 — appears 12 times. I handed it fifty products; it kept reusing its favorites.
Third, the types I didn't order never appear. My original 29 questions had ultra-abbreviations ("250 5"), typo-ridden messages, after-learning questions, custom-order inquiries. The 50 new questions contain none of these. The reason is simple: I didn't put them in the prompt.
Put the three together: the AI author is never wrong. It only writes what you ordered.
Now look back at human me. I got the exam wrong five times while making it. But that same process was where the traps got invented — putting two tape widths in on purpose, making up a sentence like "250 5", imagining the accidents that only happen after the system has learned. The five mistakes were the tuition I paid while doing the inventing.
The porting guide says the accident list decides the question count. This experiment showed the other side of that sentence:
inventing accident types human work. it comes from living in the domain
stamping questions per type AI work. more accurate than me, never tires
When I made 29 questions alone, those two jobs were one lump. Separated, the real bottleneck of exam-building isn't the question count — it's the length of your accident list. Discover one new type, and the AI stamps ten questions of it in five minutes.
P.S. The question-generation script (gen_cases.py) is in the repo with everything else → github.com/ramses203/llm-test-harness