Member-only story
Few Parameters Are Actually Needed to Fine-Tune an LLM #
A practical, data-driven analysis of Full Fine-Tuning, LoRA, QLoRA, and TinyLoRA. This guide delivers a structured experiment plan, verified code implementations, and the precise context behind the "13-parameter" claim.
TinyLoRA showed that a frozen Qwen2.5-7B-Instruct model could improve mathematical reasoning with a 13-parameter learned update under GRPO. It did not show that an LLM can be trained or stored end-to-end in 26 bytes.
The core significance of TinyLoRA lies in its conceptual paradigm shift; it transitions the optimization debate from quantitative parameter reduction ("how many") to functional utility validation ("why change them at all").
Thirteen trainable values sounds like a stunt until you inspect the setup. The base model still contains billions of parameters; the experiment changes only a tiny shared control vector. Stored in bfloat16, those 13 learned scalars occupy 26 bytes. That number is real, but it describes one narrow layer of the system rather than the system itself.
In Learning to Reason in 13 Parameters, John X. Morris and colleagues used TinyLoRA with reinforcement learning and reported roughly 91% GSM8K accuracy for Qwen2.5-7B-Instruct while…