# Can we prove this: the code submitted with this research paper actually produced the numbers reported?

> Source: <https://discuss.huggingface.co/t/can-we-prove-this-the-code-submitted-with-this-research-paper-actually-produced-the-numbers-reported/191017#post_19>
> Published: 2026-11-03 18:15:49+00:00

This is exactly the problem I hit fine-tuning my first classifier: you look at a few outputs, they seem right, and you ship. Then production fails.

What replaced vibe-checking for me:

- Schema-first extraction: every LLM call returns JSON against a fixed schema. Drift fails fast instead of silently producing garbage.
- Validation feedback retries: schema check fails, error goes back to model, one retry. Most malformed outputs fix themselves.
- CI-safe eval harness: recorded case files run in pytest, no API calls during the test run. A prompt or model change either passes the full suite or it does not merge.
- Temporal validation: hold out whole sessions, not random rows. Adjacent frames leak; random splits overstate accuracy by 20-30 points.

The code that produced the numbers should be the source of truth. If you can not re-run it and get the same result, the numbers do not exist.
