I built code-repair training data and shipped the eval so you can rerun it

A developer built TrueSET, a code-repair training dataset of 924 issue-diagnosis-fix examples verified by execution, and fine-tuned Qwen2.5-7B-Instruct to achieve a 40.0% fix rate on hard multi-file faults—more than double the base model's 17.1%—with statistical significance (p=0.039). The dataset and evaluation harness are publicly released for under $0.50 to reproduce results, challenging the reliability of LLM-judged benchmarks.

Datasets: /datasets Your 7B fixes 17% of our hardest faults. Trained on TrueSET: 40% — more than double. Check it yourself for under $0.50. Hard tier = multi-file faults base models genuinely fail, and every one of its verification scripts survived adversarial attack-testing — wrong-but-plausible code cannot pass them. The honest table below shows every tier, including the ones training didn't move. Training data with receipts. 924 issue → diagnosis → fix examples, every one proven by execution — the broken code demonstrably fails its hidden checker, the fixed code demonstrably passes, run and re-run before anything ships. No LLM judge decided what's correct. And the headline lift 17.1% → 40.0% on hard held-out tasks, McNemar p=0.039 isn't a marketing claim: the eval harness is in this repo, and a single GPU pod reproduces the base numbers in ~15 minutes for under fifty cents. The number, up front We fine-tuned Qwen2.5-7B-Instruct on this dataset plain SFT, one run, no tricks and measured execution fix-rate : apply the model's proposed fix, run a hidden verification script the model never sees. Pass or fail — no judge, no vibes. | tier | base fixed | base fix-rate 95% CI | tuned fixed | tuned fix-rate 95% CI | Δ | lift real? McNemar | |---|---|---|---|---|---|---| hard | 6/35 | 17.1% 8.1%, 32.7% | 14/35 | 40.0% 25.6%, 56.4% | +22.9 | yes p=0.039 | | medium | 36/50 | 72.0% 58.3%, 82.5% | 35/50 | 70.0% 56.2%, 80.9% | −2.0 | no p=1.000 | | easy | 38/50 | 76.0% 62.6%, 85.7% | 37/50 | 74.0% 60.4%, 84.1% | −2.0 | no p=1.000 | | overall | 80/135 | 59.3% 50.8%, 67.2% | 86/135 | 63.7% 55.3%, 71.3% | +4.4 | no p=0.392 | Read the last column carefully: Δ is the raw difference; "lift real?" asks whether the paired test can rule out chance. Overall shows +4.4 but "no" — because 20 tasks improved and 14 regressed , a net of +6 out of 135 that McNemar can't distinguish from noise p=0.392 . We refuse to bank a number the test won't back. The one tier where gains decisively outweigh regressions 10 vs 2 is hard — and there the lift is real p=0.039 . We show you the whole table, including the rows that didn't move. Base models already solve most easy and medium tasks — there is little left to teach there. The signal lives in the hard tier : multi-file coordinated faults that base models genuinely fail. The headline tier was fixed by a pre-registered rule McNemar's exact paired test, p<0.05 before the results existed — not picked afterward. This lift survived our own bar-raise. A prior run measured 14.0% → 36.0% p=0.019 on an earlier 150-task eval. We then attacked our own verification scripts with adversarial mutations, threw out every hard-tier checker that wrong code could slip past, and rebuilt that tier from scratch — fresh tasks, every checker attack-tested. Measured again: the lift held +22.9 pts, p=0.039 . The current run's full result table ships in the eval bundle results v2/FIXRATE.md , both runs' RECIPE.json recipes ship alongside it, and the prior run reproduces from its shipped recipe + task set — published numbers never move after the fact. Don't take our word for any of this. The eval harness ships with the dataset. One GPU pod, ~15 minutes, under $0.50 to reproduce the base numbers; about $2 to reproduce the whole experiment end to end. What this is plain English Each example is a small, self-contained software project with: An issue report — a realistic symptom, a traceback, the broken source files. What a developer actually sees. An expert answer — the diagnosis what's wrong, why, traced from the evidence and the corrected code. The difference from every "LLM-generated + LLM-judged" dataset: each example carries an executed, two-sided proof : - the broken project fails its verification script the fault is real, not cosmetic , and - the corrected project passes the same script the fix genuinely resolves it , - run multiple times for determinism — flaky checkers are rejected at the factory, not shipped. The hard tier is multi-file faults: the fix spans two files that must change together , so a model can't pattern-match one obvious line — it has to trace the fault across the codebase. On rows flagged difficulty.genuine multilocus: true , partial credit is impossible — and that's not a slogan, it's a shipped receipt: proof.partial fix returncodes records the verifier's return code with each fix file applied alone every single-file fix still fails . A handful of early rows predate this receipt and carry genuine multilocus: false instead of an unbacked claim. Every row ships its full artifact files + fix + verifier , so you can re-run any of this yourself. Verification scripts are additionally hardened against weakness: each checker was attacked with behavior-changing mutations of the correct code and had to catch them. A checker that wrong-but-plausible code can satisfy doesn't ship. Why execution proof matters Most synthetic code datasets are graded by another LLM "the judge gave it 9.2/10" . Judges reward fluent-sounding answers; nobody can verify the score, and quality problems hide inside it. Execution can't be flattered: the fix either makes the checker pass or it doesn't. That's also why our headline metric is reproducible by strangers — it's a property of the data, not of our marketing. What's in this repo vs. what's licensed | contents | access | | |---|---|---| teaser 40.jsonl | 40 curated examples across 14 domains, incl. verified multi-file hard faults with partial-fix receipts | public — judge the quality yourself | | eval bundle | the 135 frozen held-out tasks hard tier 100% attack-tested + the prior run's 150-task set + hidden verifiers + strict execution grader + both runs' RECIPE.json receipts | gated free — request access, reproduce our number in ~15 min | | full dataset | all 924 proven examples with artifacts + proofs | licensed — open a discussion or use the contact below | The teaser is deliberately too small to train on — it's for reading. The full set is the product. Dataset structure 924 rows full set . The public teaser ships the buyer-facing fields below so you can read every example and re-run its proof: | field | what it is | |---|---| input | the issue report: symptom, traceback, broken source files | output | the expert answer: diagnosis + full corrected code + verification script | artifact | machine-readable project tree: files , verifier , fix — rebuild and re-run the proof yourself | proof | the executed two-sided result broken return code, fixed return code, verifier SHA-256 ; multi-file rows add partial fix returncodes — each fix file applied alone, still failing | difficulty | loci files the fault spans: 1 = single-file, 2+ = multi-file hard , size score , and on multi-file rows genuine multilocus execution-verified co-dependency — facts, not quality tiers; use them for curriculum ordering if you like e.g. train easy→hard | provenance | authoring model + license | id | stable identifier | Every example meets the same bar. There is no discount shelf. Clean provenance train on it without a lawyer All text authored by OpenAI's open-weight model, released August 2025 under Apache-2.0 , served in a structured-output mode — the openai/gpt-oss-120b provenance.model field in every row reads openai/gpt-oss-120b-struct accordingly. No scraped GitHub code, no GPL contamination, no OpenAI/Anthropic terms-of-service questions. The dataset is Apache-2.0, usable commercially. Reproduce our number The evaluation bundle 135 frozen held-out tasks with hidden verifiers, stratified easy/medium/hard — every hard-tier checker adversarially attack-tested — deduplicated ≤0.86 cosine from the training set, plus the exact training recipe in RECIPE.json is included. On a single A40 ~$0.45/hr : BASE ONLY=1 FT BATCH=32 bash run.sh base fix-rate per tier, ~15 min FT BATCH=32 bash run.sh full SFT + tuned fix-rate + McNemar, ~1 h The run writes its own FIXRATE.md — the results table is emitted by the code, not typed by us. RECIPE.json the published run's base model, split, and training config ships in the bundle so you can match your run against ours. Honest scope — read this before buying - The measured lift is on tasks of this kind : small, self-contained, provably-fixable faults. It is evidence your model learns real diagnosis skills — it is not a promise about arbitrary large real-world repositories. - The hard-tier CI is wide 35 tasks . The lift is statistically real paired test , but the point estimate will wobble across reruns — expect the CI, not the point. - One training run at 7B, plain SFT. - Easy/medium tiers show no significant movement — base models are already strong there. The −2.0 on each is a single task 38→37, 36→35 , p=1.000: statistical noise, not measurable regression. We don't read it as "forgetting" because the test can't tell it from zero — the same discipline that keeps us from banking the overall +4.4. If your use case is easy tasks, this dataset is not what moves you. - The number is a property of the data, not of our hardware. Re-run the recipe on your own GPU and the fix-rate lands inside the reported CI with the paired test still significant — that is what "reproducible" means here. It won't match our floats to the last decimal GPU math never does across different cards , and it doesn't need to: you're reproducing a result , not copying our exact numbers. What is pinned to the bit is each example's two-sided proof — the hidden verifier gives the same pass/fail every run, flaky checkers are rejected at the factory. License & contact Apache-2.0. For larger corpora, custom domains, or RL environments built on the same two-sided verification verifier-as-reward , open a discussion on this repo or email TrueSET Data@proton.me . - Downloads last month - 4