{"slug": "i-built-code-repair-training-data-and-shipped-the-eval-so-you-can-rerun-it", "title": "I built code-repair training data and shipped the eval so you can rerun it", "summary": "A developer built TrueSET, a code-repair training dataset of 924 issue-diagnosis-fix examples verified by execution, and fine-tuned Qwen2.5-7B-Instruct to achieve a 40.0% fix rate on hard multi-file faults—more than double the base model's 17.1%—with statistical significance (p=0.039). The dataset and evaluation harness are publicly released for under $0.50 to reproduce results, challenging the reliability of LLM-judged benchmarks.", "body_md": "[ Datasets:](/datasets)\n\n# Your 7B fixes 17% of our hardest faults. Trained on TrueSET: 40% — more than double. Check it yourself for under $0.50.\n\n*(Hard tier = multi-file faults base models genuinely fail, and every one of its verification scripts survived adversarial attack-testing — wrong-but-plausible code cannot pass them. The honest table below shows every tier, including the ones training didn't move.)*\n\n**Training data with receipts.** 924 issue → diagnosis → fix examples, every one proven by\nexecution — the broken code demonstrably fails its hidden checker, the fixed code\ndemonstrably passes, run and re-run before anything ships. No LLM judge decided\nwhat's correct. And the headline lift (**17.1% → 40.0% on hard held-out tasks, McNemar\np=0.039**) isn't a marketing claim: the eval harness is in this repo, and a single\nGPU pod reproduces the base numbers in ~15 minutes for under fifty cents.\n\n## The number, up front\n\nWe fine-tuned Qwen2.5-7B-Instruct on this dataset (plain SFT, one run, no tricks) and measured **execution fix-rate**: apply the model's proposed fix, run a hidden verification script the model never sees. Pass or fail — no judge, no vibes.\n\n| tier | base fixed | base fix-rate (95% CI) | tuned fixed | tuned fix-rate (95% CI) | Δ | lift real? (McNemar) |\n|---|---|---|---|---|---|---|\nhard |\n6/35 | 17.1% [8.1%, 32.7%] | 14/35 |\n40.0% [25.6%, 56.4%] |\n+22.9 |\nyes (p=0.039) |\n| medium | 36/50 | 72.0% [58.3%, 82.5%] | 35/50 | 70.0% [56.2%, 80.9%] | −2.0 | no (p=1.000) |\n| easy | 38/50 | 76.0% [62.6%, 85.7%] | 37/50 | 74.0% [60.4%, 84.1%] | −2.0 | no (p=1.000) |\n| overall | 80/135 | 59.3% [50.8%, 67.2%] | 86/135 | 63.7% [55.3%, 71.3%] | +4.4 | no (p=0.392) |\n\nRead the last column carefully: **Δ is the raw difference; \"lift real?\" asks whether the paired test can rule out chance.** Overall shows +4.4 but \"no\" — because 20 tasks improved *and 14 regressed*, a net of +6 out of 135 that McNemar can't distinguish from noise (p=0.392). We refuse to bank a number the test won't back. The one tier where gains decisively outweigh regressions (10 vs 2) is **hard** — and there the lift is real (p=0.039).\n\nWe show you the whole table, including the rows that didn't move. Base models already solve most easy and medium tasks — there is little left to teach there. The signal lives in the **hard tier**: multi-file coordinated faults that base models genuinely fail. The headline tier was fixed by a pre-registered rule (McNemar's exact paired test, p<0.05) *before* the results existed — not picked afterward.\n\n**This lift survived our own bar-raise.** A prior run measured 14.0% → 36.0% (p=0.019) on an earlier 150-task eval. We then attacked our own verification scripts with adversarial mutations, threw out every hard-tier checker that wrong code could slip past, and rebuilt that tier from scratch — fresh tasks, every checker attack-tested. Measured again: the lift held (+22.9 pts, p=0.039). The current run's full result table ships in the eval bundle (`results_v2/FIXRATE.md`\n\n), both runs' `RECIPE.json`\n\nrecipes ship alongside it, and the prior run reproduces from its shipped recipe + task set — published numbers never move after the fact.\n\n**Don't take our word for any of this.** The eval harness ships with the dataset. One GPU pod, ~15 minutes, under $0.50 to reproduce the base numbers; about $2 to reproduce the whole experiment end to end.\n\n## What this is (plain English)\n\nEach example is a small, self-contained software project with:\n\n**An issue report**— a realistic symptom, a traceback, the broken source files. What a developer actually sees.** An expert answer**— the diagnosis (what's wrong, why, traced from the evidence) and the corrected code.\n\nThe difference from every \"LLM-generated + LLM-judged\" dataset: **each example carries an executed, two-sided proof**:\n\n- the broken project\n**fails** its verification script (the fault is real, not cosmetic), and - the corrected project\n**passes** the same script (the fix genuinely resolves it), - run multiple times for determinism — flaky checkers are rejected at the factory, not shipped.\n\nThe **hard tier** is multi-file faults: the fix spans **two files that must change together**, so a model can't pattern-match one obvious line — it has to trace the fault across the codebase. On rows flagged `difficulty.genuine_multilocus: true`\n\n, **partial credit is impossible** — and that's not a slogan, it's a shipped receipt: `proof.partial_fix_returncodes`\n\nrecords the verifier's return code with each fix file applied *alone* (every single-file fix still fails). A handful of early rows predate this receipt and carry `genuine_multilocus: false`\n\ninstead of an unbacked claim. Every row ships its full `artifact`\n\n(files + fix + verifier), so you can re-run any of this yourself.\n\nVerification scripts are additionally hardened against weakness: each checker was attacked with behavior-changing mutations of the correct code and had to catch them. A checker that wrong-but-plausible code can satisfy doesn't ship.\n\n## Why execution proof matters\n\nMost synthetic code datasets are graded by another LLM (\"the judge gave it 9.2/10\"). Judges reward fluent-sounding answers; nobody can verify the score, and quality problems hide inside it. Execution can't be flattered: the fix either makes the checker pass or it doesn't. That's also why our headline metric is reproducible by strangers — it's a property of the data, not of our marketing.\n\n## What's in this repo vs. what's licensed\n\n| contents | access | |\n|---|---|---|\n`teaser_40.jsonl` |\n40 curated examples across 14 domains, incl. verified multi-file hard faults with partial-fix receipts | public — judge the quality yourself |\n| eval bundle | the 135 frozen held-out tasks (hard tier 100% attack-tested) + the prior run's 150-task set + hidden verifiers + strict execution grader + both runs' `RECIPE.json` receipts |\ngated (free) — request access, reproduce our number in ~15 min |\n| full dataset | all 924 proven examples with artifacts + proofs | licensed — open a discussion or use the contact below |\n\nThe teaser is deliberately too small to train on — it's for reading. The full set is the product.\n\n## Dataset structure\n\n924 rows (full set). The public teaser ships the buyer-facing fields below so you can read every example and re-run its proof:\n\n| field | what it is |\n|---|---|\n`input` |\nthe issue report: symptom, traceback, broken source files |\n`output` |\nthe expert answer: diagnosis + full corrected code + verification script |\n`artifact` |\nmachine-readable project tree: `files` , `verifier` , `fix` — rebuild and re-run the proof yourself |\n`proof` |\nthe executed two-sided result (broken return code, fixed return code, verifier SHA-256); multi-file rows add `partial_fix_returncodes` — each fix file applied alone, still failing |\n`difficulty` |\n`loci` (files the fault spans: 1 = single-file, 2+ = multi-file hard), `size_score` , and on multi-file rows `genuine_multilocus` (execution-verified co-dependency) — facts, not quality tiers; use them for curriculum ordering if you like (e.g. train easy→hard) |\n`provenance` |\nauthoring model + license |\n`id` |\nstable identifier |\n\nEvery example meets the same bar. There is no discount shelf.\n\n## Clean provenance (train on it without a lawyer)\n\nAll text authored by (OpenAI's open-weight model, released August 2025 under Apache-2.0), served in a structured-output mode — the\n\n`openai/gpt-oss-120b`\n\n`provenance.model`\n\nfield in every row reads `openai/gpt-oss-120b-struct`\n\naccordingly. No scraped GitHub code, no GPL contamination, no OpenAI/Anthropic terms-of-service questions. The dataset is Apache-2.0, usable commercially.##\n\nReproduce our number\n\nThe evaluation bundle (135 frozen held-out tasks with hidden verifiers, stratified easy/medium/hard — every hard-tier checker adversarially attack-tested — deduplicated ≤0.86 cosine from the training set, plus the exact training recipe in `RECIPE.json`\n\n) is included. On a single A40 (~$0.45/hr):\n\n```\nBASE_ONLY=1 FT_BATCH=32 bash run.sh   # base fix-rate per tier, ~15 min\nFT_BATCH=32 bash run.sh               # full SFT + tuned fix-rate + McNemar, ~1 h\n```\n\nThe run writes its own `FIXRATE.md`\n\n— the results table is emitted by the code, not typed by us. `RECIPE.json`\n\n(the published run's base model, split, and training config) ships in the bundle so you can match your run against ours.\n\n## Honest scope — read this before buying\n\n- The measured lift is on\n**tasks of this kind**: small, self-contained, provably-fixable faults. It is evidence your model learns real diagnosis skills — it is** not**a promise about arbitrary large real-world repositories. - The hard-tier CI is wide (35 tasks). The lift is statistically real (paired test), but the point estimate will wobble across reruns — expect the CI, not the point.\n- One training run at 7B, plain SFT.\n- Easy/medium tiers show no significant movement — base models are already strong there. The −2.0 on each is a\n**single task**(38→37, 36→35), p=1.000: statistical noise, not measurable regression. We don't read it as \"forgetting\" because the test can't tell it from zero — the same discipline that keeps us from banking the overall +4.4. If your use case is easy tasks, this dataset is not what moves you. - The number is a property of the data, not of our hardware. Re-run the recipe on your own GPU and the fix-rate lands inside the reported CI with the paired test still significant — that is what \"reproducible\" means here. It won't match our floats to the last decimal (GPU math never does across different cards), and it doesn't need to: you're reproducing a\n*result*, not copying our exact numbers. What*is*pinned to the bit is each example's two-sided proof — the hidden verifier gives the same pass/fail every run, flaky checkers are rejected at the factory.\n\n## License & contact\n\nApache-2.0. For larger corpora, custom domains, or RL environments built on the same two-sided verification (verifier-as-reward), open a discussion on this repo or email ** TrueSET_Data@proton.me**.\n\n- Downloads last month\n- 4", "url": "https://wpnews.pro/news/i-built-code-repair-training-data-and-shipped-the-eval-so-you-can-rerun-it", "canonical_source": "https://huggingface.co/datasets/TrueSET/verified-code-repair", "published_at": "2026-07-19 00:52:18+00:00", "updated_at": "2026-07-19 01:21:21.001719+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "ai-research", "developer-tools"], "entities": ["Qwen2.5-7B-Instruct", "TrueSET"], "alternates": {"html": "https://wpnews.pro/news/i-built-code-repair-training-data-and-shipped-the-eval-so-you-can-rerun-it", "markdown": "https://wpnews.pro/news/i-built-code-repair-training-data-and-shipped-the-eval-so-you-can-rerun-it.md", "text": "https://wpnews.pro/news/i-built-code-repair-training-data-and-shipped-the-eval-so-you-can-rerun-it.txt", "jsonld": "https://wpnews.pro/news/i-built-code-repair-training-data-and-shipped-the-eval-so-you-can-rerun-it.jsonld"}}