In the porting guide I wrote that the exam is built to be stolen — follow five steps and it moves to any job.
So I tried being the other person. Following only what the guide says, start to finish.
For the second job I picked YouTube comment classification: scraping 20,000 comments and sorting each one into "a need," "chatter," or "a signal someone would pay." Every number in the 20,000-comments post came out of this classifier.
Which makes this a double-edged experiment. It tests whether the exam ports — and at the same time it tests whether the tool that produced my own published numbers can pass an exam.
Before writing a single question, I opened the classifier's code to understand what I was about to test. The thing that sorted 20,000 comments was not an AI. It was a regex — word matching: "if the comment contains this keyword, it's this category."
The second line of the actual data file was already an accident.
My grad-school senior bet that nobody would bother replacing humanities majors because they don't pay. He was right.
Social commentary. Not a need, and certainly not about errors. The classifier had filed it as a need in the "errors & debugging" category — because the Korean phrase for "doesn't pay" contains the same two characters as the error keyword "doesn't work." With 10,000 likes, it sat near the top of the ranking.
The accident showed up before the exam even existed.
Step 1 — write down the worst. These classifications feed decisions about what to build and what to sell. So the worst accident is "promoting chatter into a need and manufacturing fake demand." A product decision built on fake demand burns weeks.
Step 2 — the grade table. Four grades: fatal, risky, missed, harmless. In the guide I had written "only the first line, FATAL, is redefined per project; the other three read the same everywhere." Porting it, one line didn't hold. In the order domain, MISSED (failing to catch something) was a mild grade — miss an order and the customer calls, so a human finds out. But a comment that gets dropped is never looked at again by anyone. The same MISSED is effectively fatal here.
Filling in the table also exposed something about the regex tool itself: it has no way to say "I'm not sure, a human should look." Every comment gets forced into a category. Under our first principle — a wrong confirmation is worse than no confirmation — that's a dangerous property before the exam has even started.
Step 3 — plant the traps. I rewrote the guide's four trap types (confusable pairs, plausible non-targets, mid-message reversals, after-learning) into comment form. While planting them, a fifth type showed up that isn't on the list: accidental keyword hits — chatter like "Can't do this anymore, going to bed lol" classified as a need because "can't" matches an error keyword. It only exists for tools that match characters, so the guide never had it.
Step 4 — the answer key. Fifteen exam comments, each with an expected answer and the flag "can the data alone decide this?" A reply like "Me too 😭" with no parent comment is undecidable — its correct answer is "needs confirmation."
Step 5 — grading. Another place the guide didn't hold. The order exam's grader is bound to order-specific fields — product codes, quantities — and not one line of it could be reused. What ported was the frame of thinking, the grade table and the principles; the 40-line grader was written from scratch.
my comment classifier (regex): 15 cases · 8 clean (53%)
🔴 FATAL 3 🟠 RISKY 5 🟡 MISSED 1 🟢 HARMLESS 0
All three fatals were accidental keyword hits. Social commentary, chatter, and a personal anecdote — each promoted to a need.
Our shipping rule has been one line from the start: FATAL 0 ships, anything else doesn't. This tool is at FATAL 3. And I had already published the numbers it produced. The core of that post — the "I can't trust it" comments with 598 likes — is quoted verbatim, so it stands. But the percentages are only as trustworthy as this regex, and I'm writing that down honestly.
It ports. It took one hour. But my body learned that the manual has four defects.
① missing step "understand the thing you're testing" isn't there — in
practice it ate half the hour
② grade table oversold "the other three lines are the same everywhere" — MISSED
turned fatal on this project. Re-argue the table every port
③ grader is domain-bound what ports is the frame, not the code. Rewrite the grader
④ traps incomplete four types are a start. Each tool type (regex, LLM) has
traps of its own
That defect list is the real harvest of this experiment. The next person who steals this exam, knowing these four in advance, finishes in thirty minutes instead of an hour.
P.S. The 15-question comment exam and its grading results are public → github.com/ramses203/llm-test-harness, comment_exam.py
Next up: I gave the same exam to an LLM. The regex-vs-LLM decision was made by that scorecard.