I build training data. The whole point is to make models better at using tools, handling real tasks, writing code, and recovering when something goes wrong.
To sell that data, I need a number. A real evaluation number that makes a lab want to try it.
I have no money and a potato PC. So I am testing it myself on Qwen2.5 1.5B Instruct, using the free Kaggle GPUs.
This is what happened in the last run, and why I am training again completely differently.
I created trajectories where an agent tries to do a job, uses tools, makes mistakes, fixes them, and finishes the task. The idea is simple. If a model has seen agents making mistakes and recovering in training, it should get better at tool calling, agentic execution, coding, and recovery.
I wanted to prove that with a small model first.
For a long time I was fully vibe coding the fine-tuning part. I know a little bit of fine-tuning, but I relied on AI to write the code because they are genuinely so good at it. About 60 percent of the FT pipeline was written with DeepSeek v4.1 Flash, with help from GLM 5.3 Flash and Mimo v2.6 Flash(the other 40%). I kept iterating, failing, and trying again. I also avoided taking decisions myself because I did not have the knowledge.
That was the problem. The pipeline worked, but I was not the engineer.
I kept a clean scorecard for base vs post training. I am only showing exact numbers where it improved.
Where my data helped:
| What I measured | Base model | After training on my data |
|---|---|---|
| Protocol kind accuracy | 55.2% | 71.2% |
| Protocol tool name accuracy | 31.1% | 47.4% |
| Protocol args exact match | 3.5% | 7.4% |
| Protocol schema valid rate | 43.2% | 79.6% |
| Math reasoning strict check | 32.0% | 34.7% |
| Instruction following loose check | 46.2% | 46.6% |
The big one for me is schema valid rate. It went from 43.2 percent to 79.6 percent. That means the model learned to follow my custom tool format. Kind accuracy and tool name accuracy also went up a lot.
Where it got worse:
Everything else. Coding tasks went down clearly. General reasoning tasks like math went down. Instruction following on strict checks went down. And the most important one for format handling, the cases where the model should NOT call any tool, dropped a lot. It started calling tools when it should just answer.
The held out loss also got worse, which is a clear sign of overfitting.
So the data did improve the model on what it was trained for. But the model lost its general capability. That is my engineering problem, not the trajectories problem and not the model problem.
I use Kaggle CLI because I have 30 hours of free GPU per week. Kaggle gives you an NVIDIA T4 16GB, or 2x T4 with 32GB total, or a Tesla P100, with 12 hours per session before it times out. No credit card, just phone verification.
That is how I am able to fine-tune Qwen 1.5B at all. My local machine cannot do it.
I am training again. This time I am taking decisions, being careful, and actually being the engineer.
1. I am switching to LoRA. Full fine-tuning with learning rate 0.0002 for 4 epochs was too much for a 1.5B instruct model. It drifted too far from base.
2. I am mixing in general data. The last run was only my trajectories. No wonder it forgot how to not call tools. This time I will mix in general and reasoning type datasets, with a good share of no-tool examples.
3. I am owning the eval. I found a bug in my old runs where I was merging base weights because of an os.listdir snapshot bug. Every number from v6 to v9 was void. I fixed it, made the checkpoint reproducible, and now I gate on held out loss. If loss goes up too much, I stop.
I know this small run on a free GPU will not show the full potential of these trajectories. The full potential is how much these trajectories can improve a model in tool calling, agentic execution, coding, and recovery from failures. A 1.5B model on 30 hours a week cannot show all of that. But it can show a big improvement number on format and tool use, and I can keep general ability intact.
That is enough to show to labs.
The goal for this next run is simple. Keep the gains I got on schema valid and tool name, but stop the losses on coding and general tasks.
If you are also fine-tuning on Kaggle free tier, or building your own training data, I will share the next scorecard with the new mix.