{"slug": "can-we-prove-this-the-code-submitted-with-this-research-paper-actually-produced", "title": "Can we prove this: the code submitted with this research paper actually produced the numbers reported?", "summary": "A practitioner's account of replacing ad-hoc output checking with a four-part validation setup for LLM-based classifiers: schema-first JSON extraction, validation-feedback retries, a pytest eval harness that runs recorded cases without API calls, and temporal validation that holds out whole sessions. The author reports that random row splits instead of session-level holds overstate accuracy by 20-30 points because adjacent frames leak between train and test. The stated principle is that code accompanying research must be re-runnable to reproduce its reported numbers, or the numbers do not exist.", "body_md": "This is exactly the problem I hit fine-tuning my first classifier: you look at a few outputs, they seem right, and you ship. Then production fails.\n\nWhat replaced vibe-checking for me:\n\n- Schema-first extraction: every LLM call returns JSON against a fixed schema. Drift fails fast instead of silently producing garbage.\n- Validation feedback retries: schema check fails, error goes back to model, one retry. Most malformed outputs fix themselves.\n- CI-safe eval harness: recorded case files run in pytest, no API calls during the test run. A prompt or model change either passes the full suite or it does not merge.\n- Temporal validation: hold out whole sessions, not random rows. Adjacent frames leak; random splits overstate accuracy by 20-30 points.\n\nThe code that produced the numbers should be the source of truth. If you can not re-run it and get the same result, the numbers do not exist.", "url": "https://wpnews.pro/news/can-we-prove-this-the-code-submitted-with-this-research-paper-actually-produced", "canonical_source": "https://discuss.huggingface.co/t/can-we-prove-this-the-code-submitted-with-this-research-paper-actually-produced-the-numbers-reported/191017#post_19", "published_at": "2026-11-03 18:15:49+00:00", "updated_at": "2026-11-03 18:19:34.118333+00:00", "lang": "en", "topics": ["large-language-models", "ai-tools", "mlops", "ai-research"], "entities": [], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/can-we-prove-this-the-code-submitted-with-this-research-paper-actually-produced", "markdown": "https://wpnews.pro/news/can-we-prove-this-the-code-submitted-with-this-research-paper-actually-produced.md", "text": "https://wpnews.pro/news/can-we-prove-this-the-code-submitted-with-this-research-paper-actually-produced.txt", "jsonld": "https://wpnews.pro/news/can-we-prove-this-the-code-submitted-with-this-research-paper-actually-produced.jsonld"}}