Been a while. Last time I posted the dataset had passed 300k
entries and I had one 14B fine tune with 81% structural
correctness. Since then I did the thing I should have done from
the start, I built a proper benchmark and published it so people
can check my numbers instead of taking my word for it.
What got published:
184 test prompts for MQL5 (the language for algo trading bots),
the scoring rules, per item results for every model, and a
verifier script that checks the release hashes and recomputes the
headline numbers from the rows. A pass is zero compiler errors
and a built binary. Thats it. Compiling is a low bar and the card
says so, it does not mean the bot trades well.
The numbers:
Base Qwen2.5-Coder-14B-Instruct, untouched: 2 of 184 compiled. Same model after one epoch on my data: 170 of 184.
gpt-5.6-sol on the same 184 prompts: 179 of 184. So the local 14B went from useless to about 5 points behind a
frontier API model on my niche. One epoch. I am not going to
pretend thats the finish line but it is the first number I have
that somebody else can reproduce, and that matters more to me
than the number itself.
Hardware, since this is L1T:
Everything is still AMD and still local, but the layout changed
since April. The main rig is now a 7950X3D with 64GB and one
Radeon AI PRO R9700 in it, the 9800X3D and the 9070XT are out.
That box is where both local arms of the benchmark were run, base
model and tuned model, same card, same settings. The
Threadripper 2970WX has two R9700s now and does the training
under ROCm. So three R9700s total across two machines, 96GB of
VRAM, no CUDA anywhere. I fought ROCm plenty (some of you saw the
Windows thread) but it does the job and the numbers came out of
it.
Links:
Hugging Face:
GitHub mirror:
What is not in the release is the training data and the tuned
weights. What is in the release is every prompt and every result
row, so you can run your own model against the same 184 and
compare.
Caveat thats also on the card: the test prompts come from the
same generator family as the training data, so this measures how
good the model got at my kind of spec, not how it does on random
human written requests. I have a bigger private holdout in the
works and a version 1.1 of the card coming with those results.
Still grinding. If you are on AMD and want to compare notes on
ROCm training or eval, ask away.
MQL5 is a trademark of its owner. Not affiliated, independent
project.