I built code-repair training data and shipped the eval so you can rerun it

Datasets:

(Hard tier = multi-file faults base models genuinely fail, and every one of its verification scripts survived adversarial attack-testing — wrong-but-plausible code cannot pass them. The honest table below shows every tier, including the ones training didn't move.)

Training data with receipts. 924 issue → diagnosis → fix examples, every one proven by execution — the broken code demonstrably fails its hidden checker, the fixed code demonstrably passes, run and re-run before anything ships. No LLM judge decided what's correct. And the headline lift (17.1% → 40.0% on hard held-out tasks, McNemar p=0.039) isn't a marketing claim: the eval harness is in this repo, and a single GPU pod reproduces the base numbers in ~15 minutes for under fifty cents.

The number, up front #

We fine-tuned Qwen2.5-7B-Instruct on this dataset (plain SFT, one run, no tricks) and measured execution fix-rate: apply the model's proposed fix, run a hidden verification script the model never sees. Pass or fail — no judge, no vibes.

Read the last column carefully: Δ is the raw difference; "lift real?" asks whether the paired test can rule out chance. Overall shows +4.4 but "no" — because 20 tasks improved and 14 regressed, a net of +6 out of 135 that McNemar can't distinguish from noise (p=0.392). We refuse to bank a number the test won't back. The one tier where gains decisively outweigh regressions (10 vs 2) is hard — and there the lift is real (p=0.039).

We show you the whole table, including the rows that didn't move. Base models already solve most easy and medium tasks — there is little left to teach there. The signal lives in the hard tier: multi-file coordinated faults that base models genuinely fail. The headline tier was fixed by a pre-registered rule (McNemar's exact paired test, p<0.05) before the results existed — not picked afterward.

This lift survived our own bar-raise. A prior run measured 14.0% → 36.0% (p=0.019) on an earlier 150-task eval. We then attacked our own verification scripts with adversarial mutations, threw out every hard-tier checker that wrong code could slip past, and rebuilt that tier from scratch — fresh tasks, every checker attack-tested. Measured again: the lift held (+22.9 pts, p=0.039). The current run's full result table ships in the eval bundle (results_v2/FIXRATE.md

), both runs' RECIPE.json

recipes ship alongside it, and the prior run reproduces from its shipped recipe + task set — published numbers never move after the fact.

Don't take our word for any of this. The eval harness ships with the dataset. One GPU pod, ~15 minutes, under $0.50 to reproduce the base numbers; about $2 to reproduce the whole experiment end to end.

What this is (plain English) #

Each example is a small, self-contained software project with:

An issue report— a realistic symptom, a traceback, the broken source files. What a developer actually sees.** An expert answer**— the diagnosis (what's wrong, why, traced from the evidence) and the corrected code.

The difference from every "LLM-generated + LLM-judged" dataset: each example carries an executed, two-sided proof:

• the broken project fails its verification script (the fault is real, not cosmetic), and - the corrected project passes the same script (the fix genuinely resolves it), - run multiple times for determinism — flaky checkers are rejected at the factory, not shipped.

The hard tier is multi-file faults: the fix spans two files that must change together, so a model can't pattern-match one obvious line — it has to trace the fault across the codebase. On rows flagged difficulty.genuine_multilocus: true

, partial credit is impossible — and that's not a slogan, it's a shipped receipt: proof.partial_fix_returncodes

records the verifier's return code with each fix file applied alone (every single-file fix still fails). A handful of early rows predate this receipt and carry genuine_multilocus: false

instead of an unbacked claim. Every row ships its full artifact

(files + fix + verifier), so you can re-run any of this yourself.

Verification scripts are additionally hardened against weakness: each checker was attacked with behavior-changing mutations of the correct code and had to catch them. A checker that wrong-but-plausible code can satisfy doesn't ship.

Why execution proof matters #

Most synthetic code datasets are graded by another LLM ("the judge gave it 9.2/10"). Judges reward fluent-sounding answers; nobody can verify the score, and quality problems hide inside it. Execution can't be flattered: the fix either makes the checker pass or it doesn't. That's also why our headline metric is reproducible by strangers — it's a property of the data, not of our marketing.

What's in this repo vs. what's licensed #

The teaser is deliberately too small to train on — it's for reading. The full set is the product.

Dataset structure #

924 rows (full set). The public teaser ships the buyer-facing fields below so you can read every example and re-run its proof:

Every example meets the same bar. There is no discount shelf.

Clean provenance (train on it without a lawyer) #

All text authored by (OpenAI's open-weight model, released August 2025 under Apache-2.0), served in a structured-output mode — the

openai/gpt-oss-120b

provenance.model

field in every row reads openai/gpt-oss-120b-struct

accordingly. No scraped GitHub code, no GPL contamination, no OpenAI/Anthropic terms-of-service questions. The dataset is Apache-2.0, usable commercially.##

Reproduce our number

The evaluation bundle (135 frozen held-out tasks with hidden verifiers, stratified easy/medium/hard — every hard-tier checker adversarially attack-tested — deduplicated ≤0.86 cosine from the training set, plus the exact training recipe in RECIPE.json

) is included. On a single A40 (~$0.45/hr):

BASE_ONLY=1 FT_BATCH=32 bash run.sh   # base fix-rate per tier, ~15 min
FT_BATCH=32 bash run.sh               # full SFT + tuned fix-rate + McNemar, ~1 h

The run writes its own FIXRATE.md

— the results table is emitted by the code, not typed by us. RECIPE.json

(the published run's base model, split, and training config) ships in the bundle so you can match your run against ours.

Honest scope — read this before buying #

• The measured lift is on tasks of this kind: small, self-contained, provably-fixable faults. It is evidence your model learns real diagnosis skills — it is** not**a promise about arbitrary large real-world repositories. - The hard-tier CI is wide (35 tasks). The lift is statistically real (paired test), but the point estimate will wobble across reruns — expect the CI, not the point.
• One training run at 7B, plain SFT.
• Easy/medium tiers show no significant movement — base models are already strong there. The −2.0 on each is a single task(38→37, 36→35), p=1.000: statistical noise, not measurable regression. We don't read it as "forgetting" because the test can't tell it from zero — the same discipline that keeps us from banking the overall +4.4. If your use case is easy tasks, this dataset is not what moves you. - The number is a property of the data, not of our hardware. Re-run the recipe on your own GPU and the fix-rate lands inside the reported CI with the paired test still significant — that is what "reproducible" means here. It won't match our floats to the last decimal (GPU math never does across different cards), and it doesn't need to: you're reproducing a result, not copying our exact numbers. Whatispinned to the bit is each example's two-sided proof — the hidden verifier gives the same pass/fail every run, flaky checkers are rejected at the factory.

License & contact #

Apache-2.0. For larger corpora, custom domains, or RL environments built on the same two-sided verification (verifier-as-reward), open a discussion on this repo or email ** TrueSET_Data@proton.me**.

• Downloads last month
• 4