A hands-on test of three LLM distillation methods, training a 4B Qwen model from a 27B teacher on local GPUs, with real benchmark results.
What is LLM distillation and does it actually work on local hardware? #
Yes, with caveats. Distillation means training a smaller “student” model to copy the behavior of a larger “teacher” model, instead of having the student learn purely from trial and error. A hands-on test training a 4 billion parameter Qwen model against a 27 billion parameter Qwen teacher, run entirely on two networked local GPU boxes, showed that all three major distillation techniques improve how the small model behaves. But none of them broke past a hard accuracy ceiling that reinforcement learning had already hit.
TL;DR #
- Three distillation methods were tested on identical hardware: training on the teacher’s final answers, training on the teacher’s full token probabilities, and on-policy distillation where the teacher grades the student’s own live attempts token by token.
- Reinforcement learning (GRPO) hit a wall at 58.4% accuracy after starting at 52%, because tasks the student always failed or always passed produced zero learning signal under RL’s group-average scoring.
- Copying the teacher’s answers matched RL’s score (around 58%) but made the student’s output 30% shorter and faster, roughly matching the teacher’s own concise workflow.
- Matching the teacher’s token probabilities added almost nothing because the 27B teacher was already extremely confident, putting 99% probability on a single token on three quarters of all tokens checked.
- On-policy distillation hit the highest single score (58.9%) and let the teacher flag the exact token where the student’s code went wrong, but it still landed as a statistical tie with RL by the end of training.
- Every method converged on the same ceiling , suggesting the bottleneck isn’t the training method but something closer to the model’s actual task knowledge.
- Checking teacher confidence before training on probabilities can save hours of compute, since a highly confident teacher’s text already contains almost everything its probability distribution would teach.
Built like a system. Not vibe-coded.
Remy manages the project — every layer architected, not stitched together at the last second.
How does distillation differ from reinforcement learning? #
Reinforcement learning (via GRPO, Group Relative Policy Optimization) has the student attempt a task multiple times. Attempts that score above the group’s average get reinforced; attempts below it get suppressed. That setup has a specific flaw: if a model fails a hard task on every attempt, every attempt gets zero reward, and nothing beats the average. The model learns nothing from tasks it can never solve. The same is true in reverse for tasks it always solves. In the test case, this caused RL to improve the 4B model from 52% to 58.4% accuracy and then stall completely, no matter how much additional training was applied.
Distillation sidesteps that problem by giving the student a teacher instead of a scoreboard. Rather than being told pass or fail at the end, the student gets shown how to work through the problem, or gets graded on every step it takes. This idea traces back to a 2015 technique from Geoffrey Hinton, where small models learned from a larger model’s output probabilities rather than just its final answers. Qwen itself used distillation from larger models to build smaller models in its Qwen 3 family.
What are the three distillation methods, and how do they differ? #
Training on the teacher’s answers. The teacher solves a batch of tasks, the transcripts where it got the right answer are kept, and the student is fine-tuned to reproduce those transcripts. This is the simplest version and doesn’t require the two models to share any internal structure.
Training on the teacher’s probabilities. At every token, a language model doesn’t just output one word. It assigns probabilities across many possible next tokens before one gets selected. For example, while writing Python to inspect a dataset, the teacher model gave 59% probability to .head(), 36% to .shape, and 4% to .info(), with only .head() making it into the final text. Training on probabilities instead of just the picked token exposes the student to that full spread of “reasonable alternatives.” This method only works when teacher and student use an identical tokenizer, which is the case for models from the same family (both models here were from Qwen).
On-policy distillation. Instead of learning from the teacher’s own transcripts, the student attempts tasks itself inside its own sandbox. The teacher then reviews the student’s actual attempt and grades every single token the student produced, right or wrong. Even a fully failed attempt generates a dense training signal, since the teacher can mark the exact token or line where the logic went off track. According to Qwen’s technical reporting, this approach outperformed plain reinforcement learning on math tasks while using roughly a tenth of the GPU hours.
What happened when all three methods were tested head to head? #
Everyone else built a construction worker.
We built the contractor.
One file at a time.
UI, API, database, deploy.
The setup used two Nvidia DGX Spark units connected over a high-speed link, with the 27B model running as the teacher on one and the 4B student training on the other. Tasks came from an open collection of verifiable data-analysis problems (built from real Kaggle datasets), where the model has to inspect a CSV file in a sandboxed terminal, write Python, and output a checkable final answer. A held-out exam of 394 tasks, including 87 from datasets never seen in training, was used to score every version of the model. Before any training, the 4B model scored about 52% and the 27B teacher scored about 64%.
Training on teacher answers: The teacher solved 2,000 training tasks, and roughly 1,400 transcripts that passed the answer checker were used to fine-tune the student. Final exam score landed around 58%, a tie with the RL-trained model. The notable change was efficiency: the RL model had learned to write long, trial-and-error transcripts averaging about 2,000 tokens per task. After copying the teacher’s clean answers, the student dropped to about 1,400 tokens per task, roughly matching the teacher’s own concise style. That’s a 30% cut in output length with no loss of accuracy.
Training on teacher probabilities: The teacher re-processed the winning transcripts (nearly 5 million tokens) and saved its top 20 token choices and probabilities at each step. The student trained to match that distribution and ended up agreeing with the teacher’s top choice 96% of the time. On the exam, it scored about 56%, still a tie with the other methods. Inspecting the teacher’s own probabilities explained why: on three quarters of all tokens checked, the teacher put 99% of its probability mass on a single choice. In the rare cases where it was genuinely unsure, the disagreement was usually trivial (one example: a near 50/50 split between writing “data” versus “dataset”). When a teacher is this confident, its plain text already encodes almost everything its probability distribution would add.
On-policy distillation: The student attempted tasks itself, and the teacher graded every token of the student’s own output, flagging where the logic diverged. In one real example, the teacher’s grading correctly flagged the exact token (a .sum() call) where the student miscalculated an average order value, leading to a wrong final number. At step 20 of training, the student hit 58.9%, the highest single score recorded across the whole project, but it fell back toward 58% by step 60 and remained a statistical tie with RL. Behaviorally, this version did fix some formatting issues (writing “50” instead of a jumbled “5050” for a sample-count answer) but also introduced new failure modes, including a case where it kept double-checking its own work and ran out of allowed steps without ever submitting an answer, on a task the teacher solved in far fewer steps.
Why did every distillation method hit the same ceiling? #
The consistent result across copying answers, matching probabilities, and live on-policy grading was a plateau around 58%, the same number reinforcement learning reached on its own. That convergence suggests the limiting factor isn’t the training method itself. If three structurally different techniques, one of them with direct, token-level correction from a stronger model, all land in the same place, the bottleneck looks more like a ceiling on what the small model’s underlying weights can represent or has been exposed to, rather than a gap in how the training signal was delivered.
Remy doesn't build the plumbing. It inherits it. #
Other agents wire up auth, databases, models, and integrations from scratch every time you ask them to build something.
Remy ships with all of it from MindStudio — so every cycle goes into the app you actually want.
This matters for anyone planning a similar project: the choice of distillation method can change efficiency, output length, and specific failure patterns, but it may not move the needle on raw accuracy if the student model has a separate, harder limitation to work around first.
Is distillation worth the compute compared to reinforcement learning? #
It depends on the goal. If the aim is strictly to raise an accuracy score past where RL plateaus, none of the three methods tested here delivered a definitive win over RL. But distillation did produce other practical gains: training on teacher answers made the model 30% faster and cheaper to run per task, a meaningful efficiency win even without an accuracy jump. On-policy distillation also offers something RL structurally cannot: a training signal on tasks the student fails every time, since the teacher can still point to the specific token where things went wrong instead of only reporting pass or fail.
A practical pre-check before investing a full training run: look at how confident the teacher model is on a small sample of outputs. If it’s already putting near-certain probability on single tokens most of the time, probability-based distillation likely won’t add much beyond simply copying its text answers.
Frequently Asked Questions #
What is the difference between distillation and fine-tuning?
Fine-tuning generally refers to adapting a model using any labeled dataset. Distillation specifically means that dataset (or grading signal) comes from a larger “teacher” model’s outputs, answers, probabilities, or token-level feedback, rather than from human-labeled data alone.
Does distillation require the teacher and student to be the same model family?
Training on final answers works across different model families. Probability-based distillation requires both models to use the identical tokenizer, which in practice usually means they come from the same model family, as was the case with the Qwen 4B student and Qwen 27B teacher here.
What is on-policy distillation?
It’s a method where the student model attempts a task on its own, and a stronger teacher model then reviews that specific attempt and scores every token the student produced, including wrong ones, pointing out exactly where errors occurred rather than just giving a pass/fail result at the end.
Why did reinforcement learning stop improving the small model?
Because of how GRPO scores attempts: it reinforces attempts that beat the group average and suppresses those that don’t. Tasks the model always failed, or always succeeded at, produced no useful signal either way, so RL could only sharpen performance on tasks the model sometimes got right.
Is it worth checking a teacher model’s confidence before running distillation?
Yes. If the teacher assigns very high probability to a single token most of the time, its written answers already capture nearly all the information its full probability distribution would provide, so skipping expensive probability-logging and just training on its text answers can save significant time and storage.