Last time I gave my LLM an order-reading exam and lost 5 times as the exam author.
Today: how that exam was built. Conclusion first — nice questions are a waste of paper.
Ask anyone to write a test and they start with the case that works. "5 boxes of the 250 shipping boxes please" → shipping box 250, 5 boxes. It passes. Feels good. Reassuring.
But that's wasted points. Models rarely fail the normal cases. What fails is everything that isn't normal.
My 29 questions broke down like this:
Normal orders 4
Things that aren't orders 6 ← the biggest group
Changes & cancellations 4
Ambiguous ones 5
Typos & extreme shorthand 3
After learning kicks in 7
Normal is the smallest group. On purpose.
The worst accident for this program is shipping something nobody ordered. So the exam should aim at that accident more than anything else.
What are the dimensions of the 250 shipping box?
Product name: present. Number: present. But it's not an order. It's a question.
A program that treats "product name spotted" as "order detected" calls the truck right here. So I planted six of these: price inquiries, stock inquiries, delivery questions, greetings, a tax-invoice request.
Changes and cancellations are nastier.
I ordered 5 boxes of the 250 — please send only 3
Two numbers. Read only the first half and it's a perfect order. Treat it as a new order and the goods ship twice.
It's not just about hard questions. Make the data itself messy.
One reason: real data already looks like this. A real product catalog always has near-twins.
Run the exam on a clean catalog and here's what happens — everything passes. Then you plug in production data and it collapses. If the exam passed but production has accidents, that's not the model's fault. That's the exam's fault.
This program learns. When a human picks a match once, it remembers. "250" → shipping box 250. Automatic from then on.
Confession: my first 22 questions had zero learning scenarios. "Fix it once and it's automatic afterwards" is this program's reason to exist, and I hadn't tested that path even once. I added 7 questions late.
And those 7 questions showed me something scary. Learning isn't just a convenience feature — it can be an accident-generating feature.
Trap one. The program has learned "tape = 48mm." Then this arrives:
tape 60, 2 boxes
Apply the learned match as-is and 48mm ships. The customer said 60. An explicit spec must beat the learned match.
Trap two. The program has learned "250 = shipping box." Then:
What are the dimensions of the 250?
The more it learns, the more confident the program gets. Use that confidence to read a question as an order and it's over. Learned or not, a question is a question.
It passed both. Good. But if I had never written these questions? It would have gone to production with nobody knowing whether it passes them.
If the exam is nice, the exam passes — and production has the accidents.
P.S. All 29 questions and the trap-laden catalog are public → github.com/ramses203/llm-test-harness
Next up: grading — why pass/fail grading will wreck you.