Can we prove this: the code submitted with this research paper actually produced the numbers reported?

A practitioner's account of replacing ad-hoc output checking with a four-part validation setup for LLM-based classifiers: schema-first JSON extraction, validation-feedback retries, a pytest eval harness that runs recorded cases without API calls, and temporal validation that holds out whole sessions. The author reports that random row splits instead of session-level holds overstate accuracy by 20-30 points because adjacent frames leak between train and test. The stated principle is that code accompanying research must be re-runnable to reproduce its reported numbers, or the numbers do not exist.

This is exactly the problem I hit fine-tuning my first classifier: you look at a few outputs, they seem right, and you ship. Then production fails. What replaced vibe-checking for me: - Schema-first extraction: every LLM call returns JSON against a fixed schema. Drift fails fast instead of silently producing garbage. - Validation feedback retries: schema check fails, error goes back to model, one retry. Most malformed outputs fix themselves. - CI-safe eval harness: recorded case files run in pytest, no API calls during the test run. A prompt or model change either passes the full suite or it does not merge. - Temporal validation: hold out whole sessions, not random rows. Adjacent frames leak; random splits overstate accuracy by 20-30 points. The code that produced the numbers should be the source of truth. If you can not re-run it and get the same result, the numbers do not exist.