How to take a pretrained language model, specialize it for a real-world task, learn from a larger teacher, and build an extensible training pipeline — even when your compute budget is limited.
Imagine a financial analyst reviewing a company’s annual report.
The report contains revenue, operating expenses, net income, cash flow, and financial ratios across multiple reporting periods.
A colleague asks:
A general-purpose language model may understand every word in the question and still produce an incorrect answer.
It might select the wrong row, confuse two reporting periods, overlook a unit conversion, or produce a plausible-looking calculation that is not supported by the source document.
This is a particularly interesting problem because financial question answering is not simply a language-generation task. It combines document understanding, table interpretation, numerical extraction, arithmetic, reasoning, and structured response generation.
Now consider a different approach.
Instead of training an entirely new language model, what if we started with an existing compact model and taught it to specialize in financial reasoning?
What if we could also use a larger model as a teacher, learn from its responses, verify those responses, and transfer the accepted examples to the smaller model?
And what if we could identify the smaller model’s mistakes and use those mistakes to construct a second training stage?
That is the idea behind this project.
I started with Qwen3–0.6B as the student model and Qwen3–8B as the teacher. I explored three adaptation strategies:
The experiments used FinQA and TAT-QA, along with programmatically generated synthetic financial examples. Training and inference were conducted using a free Google Colab Tesla T4 environment.
The objective was not to claim that a short training run can produce a state-of-the-art financial model. It was to implement a complete, extensible methodology for adapting small language models under constrained compute.
The interesting part is not just that three models were trained.
It is how the data, supervision, training, validation, correction, evaluation, and deployment stages fit together.
Let’s start at the beginning.
One of the most common misconceptions when starting with language models is that building a specialized model requires training one from scratch.
That is rarely the most practical starting point.
Training from scratch means teaching a model general language patterns and representations using a very large corpus. This involves substantial data, compute, optimization, and engineering.
Fine-tuning starts with a model that has already learned useful language representations. We then adapt its behavior to a narrower task.
Think of a newly hired financial analyst.
They already understand accounting terminology, arithmetic, spreadsheets, and business communication. You do not need to teach them English or mathematics from the beginning.
You need to teach them your organization’s reporting conventions, calculation formats, and review procedures.
That is the intuition behind fine-tuning.
A pretrained model already has useful capabilities. Fine-tuning changes its parameters so that it becomes more effective at a particular set of tasks.
For this project, the starting models were:
The Qwen3–0.6B model is the compact student, while Qwen3–8B is the larger teacher.
The teacher has roughly 10.9 times as many parameters as the student, based on the parameter counts recorded during the notebook’s model inspection.
The idea was to investigate whether a compact student could acquire useful financial task behavior through carefully designed supervision.
We did not build a new Transformer architecture. We adapted an existing pretrained one.
That distinction is important because it changes the entire economics of the experiment.
A model cannot learn a specialized task effectively if the training examples do not represent that task.
We selected two datasets that expose different aspects of financial question answering.
Dataset: FinQA — GitHub repository
FinQA focuses on numerical reasoning over financial reports and tables.
Consider this simplified example:
Financial metric20212022Revenue100120Operating expenses6070Net income1520
Question:
What was the percentage increase in revenue?
The model must:
Identify the revenue row.
Locate the values for 2021 and 2022.
Determine the appropriate mathematical operation.
Execute the calculation.
Return the answer with the correct scale.
The calculation is:
Growth = (120–100) / 100 × 100 = 20%
This is more than language understanding. It combines table interpretation, numerical extraction, arithmetic, and answer formatting.
A model that understands the question but selects the wrong year has failed. A model that selects the correct numbers but performs the wrong operation has also failed.
The task therefore provides a useful environment for studying numerical reasoning in language models.
Dataset: TAT-QA — GitHub repository
TAT-QA contains questions that require interpreting both tables and textual context.
For example, a question might ask for a financial measure whose value appears in a table but whose definition or reporting context is explained in an accompanying paragraph.
The model must connect information across different parts of a document.
This makes TAT-QA a useful complement to FinQA.
FinQA emphasizes numerical reasoning over financial tables and reports, while TAT-QA introduces a hybrid table-and-text question-answering setting.
Financial question answering is not one homogeneous task.
A model may be able to calculate a percentage but struggle to locate the correct values. It may read a table correctly but confuse the unit or scale. It may understand a textual explanation but fail to combine it with a table.
Using both datasets creates a more varied task environment.
The original dataset repositories contain substantially more examples than we used in this prototype. The notebook loaded the datasets and constructed a deliberately smaller experimental pool to keep training and teacher inference tractable on a free T4 GPU.
The selected Stage 1 training pool contained 500 examples, distributed across three sources:
Total: 500 training examples
The synthetic examples were generated programmatically with deterministic gold answers.
They covered different financial metrics, reporting periods, numerical magnitudes, units, and question types.
We also prepared separate validation, mining, test, and robustness pools.
The notebook included document-overlap, exact-duplicate, near-duplicate, and contradictory-example checks. The reported leakage checks found no document overlap between the relevant evaluation splits and no exact or near-duplicate examples removed by those checks.
This matters.
Suppose the same annual-report table appears in both training and testing. The model might appear to perform well because it has already encountered the example.
That would not be a meaningful test of generalization.
Data preparation is therefore not just a preliminary step. It is part of the model-quality process.
Before writing any training code, it helps to define the experimental design.
M1 — Base Student (Qwen3–0.6B)
The original, untuned model used as the student baseline.
M2 — Supervised Fine-Tuning (Qwen3–0.6B + SFT)
The student adapted using gold-standard supervised training examples.
M3 — Knowledge Distillation (Qwen3–0.6B + Distillation)
The student trained on teacher-generated responses that passed deterministic verification.
M4 — Sequential Adaptation (Qwen3–0.6B + Distillation + SFT)
A two-stage approach combining response-based distillation with targeted supervised fine-tuning.
M5 — Teacher Reference (Qwen3–8B)
The larger model used to generate candidate responses and provide a teacher-performance reference.
The important design choice was to keep the same underlying student model for M1 through M4.
This gives us a common starting point for studying different adaptation recipes.
The teacher serves two purposes:
M1 gives us a baseline. M2, M3, and M4 explore different ways of changing the student. M5 provides a larger reference point.
Think of it as three different training programmes for the same analyst.
One learns directly from verified examples. Another learns from a more experienced analyst. The third learns from that experienced analyst and then receives targeted corrections based on mistakes.
The experimental design lets us study these different supervision strategies within a common model family.
One of the most important engineering decisions was to define the model’s expected response format before training.
Instead of allowing the model to return an unconstrained paragraph, we used a structured three-part response:
Calculation: <arithmetic expression = value, or n/a>Answer: <final answer>Scale: <thousand / million / billion / percent / none>
For example:
Calculation: (120 - 100) / 100 * 100 = 20Answer: 20%Scale: percent
For insufficient context, the expected answer was:
Answer: NOT_ANSWERABLE
The prompt also instructed the model to return CONFLICTING_INFORMATION when the supplied context contained conflicting values for the same quantity.
Why is this useful?
Imagine receiving the following answer:
Revenue increased by 20 percent, which indicates positive growth.
It sounds reasonable, but it does not expose the calculation.
Now compare it with:
Calculation: (120 - 100) / 100 * 100 = 20Answer: 20%Scale: percent
The second response is easier to parse, inspect, and validate.
A structured output contract also creates opportunities for deterministic checks:
This does not guarantee that the model has reasoned correctly. It does, however, make its output more amenable to automated evaluation.
That distinction is important:
Format validity is not the same as answer correctness.
This separation between response structure and answer quality becomes especially important when building automated training-data validation.
Let’s begin with the most direct approach.
Suppose we want a model to answer financial questions in a particular format.
We provide examples containing:
The model learns to generate the target response when it receives the corresponding input.
A simplified training example might look like this:
SYSTEM:You are a careful financial analyst.Use only the supplied context.
USER:Context:Revenue in 2021: 100Revenue in 2022: 120
Question:What was the percentage increase?
ASSISTANT:Calculation: (120 - 100) / 100 * 100 = 20Answer: 20%Scale: percent
The model processes the sequence as tokens and learns to predict the next token.
During training, we compare its predictions with the expected target tokens.
For a causal language model, the standard supervised objective is next-token prediction.
Training Objective (Negative Log-Likelihood Loss)
Where:
The model is trained to assign higher probability to the expected response.
An important implementation detail is response-only loss masking.
We do not necessarily want the model to be penalized for predicting the prompt itself. The notebook masks prompt-token labels and computes the training loss on the response portion.
Conceptually:
labels = input_ids.clone()
This is a simplified illustration of the loss-masking idea, not a complete training implementation.
The actual pipeline also handles tokenization, padding, truncation, batching, attention masks, and checkpoint management.
The SFT model used 500 training examples:
The targets were derived from gold annotations and deterministic synthetic generation.
For FinQA, the notebook constructed arithmetic-expression targets from the gold programs or derivations. For TAT-QA, it used the corresponding gold answer annotations.
No teacher-generated responses or teacher logits were used for this experiment.
The resulting model became our first task-adapted student.
This is the most straightforward approach to understand and implement when starting with fine-tuning.
The student model contains hundreds of millions of parameters.
Updating every parameter would require storing more than just the model weights.
Training also requires memory for gradients, optimizer states, intermediate activations, and other runtime allocations.
For a limited GPU environment, that can become a significant constraint.
This is where parameter-efficient fine-tuning becomes useful.
LoRA stands for Low-Rank Adaptation.
Instead of updating the original weight matrix directly, LoRA freezes the original matrix and introduces a smaller trainable update.
LoRA Weight Update
The key insight is that the model does not necessarily need an unrestricted update to every original parameter to adapt to a downstream task.
It can learn a constrained update through much smaller matrices.
For an intuitive example, imagine modifying a large spreadsheet formula system.
Instead of rewriting every formula, you introduce a compact set of adjustments that influence the relevant calculations.
LoRA follows a similar principle.
QLoRA adds another memory-saving technique.
The pretrained model is loaded using 4-bit quantized weights, while the trainable LoRA adapters remain in higher precision.
The base model remains frozen, and the adapters receive the parameter updates.
The major components include:
The notebook used a QLoRA-style setup with:
This was particularly useful because the Colab Tesla T4 does not support BF16 in the same way as newer GPU architectures.
A simplified configuration looks like this:
from transformers import BitsAndBytesConfigimport torch
quant_config = BitsAndBytesConfig( load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_use_double_quant=True, bnb_4bit_compute_dtype=torch.float16)
And a simplified LoRA configuration:
from peft import LoraConfig
lora_config = LoraConfig( r=16, lora_alpha=32, lora_dropout=0.05, task_type="CAUSAL_LM")
The actual notebook applied adapters to attention and MLP projections, alongside gradient checkpointing and memory-aware training configuration.
For this run, the trainable adapter parameter count was approximately 10.09 million.
That is a compact trainable parameter set relative to the underlying student model.
It is useful to understand what this does and does not mean.
QLoRA reduces the memory footprint of the frozen base weights during training. It does not eliminate the need to load the base model or perform the forward and backward computations required to train the adapters.
Now we can put the pieces together.
The first adapted model was M2, the Finance-SFT student.
Its training pipeline was:
Model: Qwen3–0.6B + Supervised Fine-Tuning (SFT)
Effective Batch Size Calculation
The effective batch size was achieved using a per-device batch size of 1 and gradient accumulation over 4 steps.
Effective Batch Size = 1 × 4 = 4
The training run completed successfully.
The best recorded validation loss was approximately 0.3146, and the selected checkpoint was at step 120.
One important detail: although the validation pool contained more examples, only 12 examples survived the configured validation selection and sequence-length filtering for this training run. This is a small checkpoint-selection sample, so the recorded validation loss should be interpreted accordingly.
The adapter was then merged into the base model. The merge-validation procedure reported identical greedy outputs between the adapter and merged versions on the prompts used for that check.
That is useful evidence that the merge preserved the tested behavior.
It is not a guarantee of identical behavior across every possible input.
We now had a model adapted directly to the financial task using verified target responses.
But there is another interesting question:
What if the training targets came from a larger model instead of being constructed directly from gold answers?
That brings us to knowledge distillation.
Imagine a junior analyst learning from a senior analyst.
The senior analyst receives a financial question and explains how they would solve it. The junior analyst studies the explanation and learns to produce similar responses.
That is the basic intuition behind teacher-student learning.
In machine learning, a larger teacher model provides information that can be transferred to a smaller student.
However, there are different ways to do this.
In classical logit-based distillation, the student is trained to approximate the teacher’s output probability distribution.
The teacher produces a probability distribution over possible next tokens.
The student produces its own distribution.
KL-Divergence Distillation Objective
This approach requires access to teacher logits or probability distributions.
What if the teacher is available only through its generated responses?
We can still perform a useful form of knowledge transfer.
The teacher receives a prompt and produces a response. We then use that response as a candidate training target for the student.
This is often called response-based or black-box distillation.
It does not require teacher logits.
That was the approach used in this project.
The teacher generated structured financial calculation traces and answers. The candidate responses were then passed through deterministic validation.
The important point is that we did not blindly trust every teacher response.
A larger model can also make mistakes.
Consider a teacher response:
Calculation: (120 - 100) / 100 = 0.2Answer: 20%Scale: percent
The arithmetic expression evaluates to 0.2, and the final answer is consistent with 20%.
Now imagine a response where the calculation evaluates to 0.2 but the final answer says 25%.
That response should not become a training target.
The notebook implemented validation checks for:
Only responses passing the required checks were accepted for the teacher-supervised training set.
This is a crucial distinction:
We used the teacher to generate candidate supervision, but the gold annotations acted as the verification mechanism.
The approach is therefore not unsupervised distillation.
It is verified, response-based knowledge distillation using rejection sampling.
The notebook used Qwen3–8B as the teacher in local 4-bit mode.
The teacher-supervision stage contained 500 examples in total:
Of the 400 newly generated responses, 149 passed the configured deterministic verification process.
That represents a teacher-response acceptance rate of:
Teacher Response Acceptance Rate = (149 ÷ 400) × 100 = 37.25%
The rejected responses were not simply ignored. Their corresponding gold targets were retained as potential correction examples for the second-stage experiment.
The accepted teacher responses were combined with 100 deterministic synthetic examples to construct the M3 target pool.
After sequence-length filtering, 247 examples were used for M3 training.
The pipeline can be represented as:
This approach gives us a practical way to use a larger model as a source of candidate training examples without needing access to its internal logits.
M3 started from the same Qwen3–0.6B base model as M2.
The difference was the supervision source.
M2 learned from gold-derived targets.
M3 learned from verified teacher-generated responses, combined with the deterministic synthetic examples.
Model: Qwen3–0.6B + Response-Based Knowledge Distillation
The best recorded validation loss was approximately 0.4801, with the selected checkpoint at step 20.
The adapter was merged into the base model and passed the same greedy-output merge validation used for M2.
The key engineering outcome was not simply another trained checkpoint.
It was a reusable teacher-supervision pipeline:
That workflow can be reused with a different teacher, student, or task.
It also illustrates why teacher quality alone is not enough.
A useful distillation pipeline needs a mechanism for deciding which teacher responses should become training targets.
This was the most interesting part of the experimental design.
A student that has been trained on teacher responses is not necessarily finished.
It may still struggle with certain examples.
What if we could identify those failures and use them to create a targeted correction dataset?
That is the motivation behind M4.
Think about a student preparing for an examination.
They complete an initial practice test. Instead of repeating the entire syllabus, a teacher identifies the questions they struggled with and prepares additional exercises around those concepts.
The next training stage becomes more targeted.
The same idea can be applied to language-model adaptation.
We start with the distilled student, evaluate it on a separate mining pool, and identify examples where it fails.
These are not official validation or test examples. The mining pool was constructed separately from the final test subset.
Keeping the mining pool separate from final evaluation data is essential.
The notebook used a 50-example mining pool.
M3 failed 41 of those 50 examples.
The second-stage training dataset combined three complementary sources:
1. Hard Examples — 41 examples
Correct examples identified through student failure analysis, targeting areas where the distilled model struggled.
2. Teacher-Rejection Corrections — 251 examples
Gold-corrected examples corresponding to teacher-generated responses that failed the verification process.
3. Replay Examples — 125 examples
Selected examples retained from the first training stage to preserve previously acquired capabilities and reduce forgetting.
Total constructed dataset: 417 examples
This composition combined targeted error correction, reliable gold supervision, and replay-based retention. One example was subsequently dropped during sequence-length filtering, leaving 416 usable training examples for stage-two training.
After sequence-length filtering, 416 examples were used.
Let’s understand each component.
The distilled student was evaluated on the separate mining pool.
Examples where it failed were selected for correction.
This provides a way to make the next training stage more responsive to observed errors.
Rather than training on a completely random collection of additional examples, we can use the model’s own performance to help prioritize what it should learn next.
Some teacher responses did not pass the verification procedure.
Rather than automatically discarding the corresponding examples from the entire project, the pipeline used their gold targets to construct corrected training examples for the second stage.
This turns the teacher-validation process into an additional source of useful training data.
The teacher’s rejected response is not treated as correct. The gold annotation provides the correction.
Training only on new examples can cause the model to lose useful behavior acquired during the first stage.
Replay addresses this by mixing a subset of earlier verified training examples into the second-stage dataset.
It provides continuity between the original distillation objective and the targeted correction objective.
M4 continued from the best M3 adapter.
Model: Qwen3–0.6B + Distillation + Targeted SFT
Notice the lower learning rate.
The model already had a task-specific initialization from the first stage. A smaller learning rate was used for continued adaptation.
The best recorded validation loss was approximately 0.4005, with the selected checkpoint at step 100.
The final model was merged and passed the same adapter-versus-merged greedy-output validation.
This is a useful experimental pattern because it separates initial knowledge transfer from targeted correction.
It also provides a natural extension point for future work: improve the mining strategy, expand the correction dataset, and evaluate whether the changes produce repeatable gains across multiple seeds and larger test sets.
One experimental-design qualification is worth making explicit: M4 receives additional training compute compared with M2 and M3. Consequently, any observed difference between M4 and the other students cannot be attributed solely to the two-stage recipe. More training steps are another possible contributor.
That is precisely the kind of confound a larger follow-up experiment should control.
Compute constraints influenced almost every engineering decision.
The notebook detected an NVIDIA Tesla T4 with approximately 14.56 GB of GPU memory.
A T4 is not equivalent to a modern high-end training GPU. We therefore used a configuration designed around the available resources.
The key decisions were:
The base model was loaded in 4-bit NF4 for training.
This reduced the memory footprint of the frozen model weights.
The T4 environment used FP16 mixed precision because BF16 was not supported by the detected GPU architecture in this setup.
This is a practical reminder that precision settings should be selected based on the hardware, not copied blindly from another notebook.
Gradient checkpointing reduces activation-memory requirements by recomputing selected activations during the backward pass.
This trades additional computation for lower memory consumption.
The training configuration used a memory-conscious optimizer setup.
The per-device batch size was 1, with gradient accumulation of 4.
The notebook wrote data, teacher responses, checkpoints, predictions, and reports to Google Drive.
It used cached outputs and checkpoint-resume behavior so that an interrupted Colab session would not necessarily require repeating every completed stage.
This is an underrated part of practical ML engineering.
A notebook that trains successfully only when uninterrupted is less useful than one designed to recover from runtime disconnections.
The pipeline was organized into distinct stages, making it possible to resume teacher generation and prediction generation from cached outputs and continue training from saved checkpoints.
That is the difference between a one-off demonstration and a reusable experiment workflow.
M2 — Gold-Target SFT
Training duration: 837.5 seconds (approximately 14 minutes)
M3 — Response-Based Distillation
Training duration: 404.2 seconds (approximately 6.7 minutes)
M4 — Distillation + Targeted SFT
Training duration: 342.0 seconds (approximately 5.7 minutes)
Note: These durations represent student training time only. Teacher-response generation and verification are separate pipeline stages and are not included in these figures.
The teacher-supervision stage also required a separate inference run. Its recorded validation summary reported approximately 30 minutes of elapsed time.
These numbers describe the notebook’s recorded execution, not a universal training-time estimate. Runtime depends on the Colab allocation, software versions, caching, model , sequence lengths, and other factors.
The important point is that the pipeline was engineered to fit a constrained environment rather than assuming access to a large multi-GPU cluster.
A training loss curve can be useful, but it does not tell the whole story.
The notebook tracked training and validation loss during each experiment.
Best checkpoint: Step 120
Best recorded validation loss: 0.3146
Best checkpoint: Step 20
Best recorded validation loss: 0.4801
Best checkpoint: Step 100
Best recorded validation loss: 0.4005
Note: The best checkpoint for each experiment was selected based on its recorded validation loss. Lower validation loss indicates a better fit to the validation data under the training objective; it does not, by itself, establish superior task-level reasoning or generalization.
These values are useful for understanding the optimization runs, but they should not be compared as direct measures of model quality across different training-target distributions.
For example, the SFT targets and teacher-generated targets are not identical distributions.
The validation sample was also small: 12 usable examples were retained for the training-time validation procedure after sequence-length filtering. This is not a sufficiently large validation set for making strong statistical claims about generalization.
During QLoRA training, the model consists of:
For deployment, it is often convenient to merge the adapter into the base model.
LoRA Weight Merging
The resulting checkpoint can be loaded as a standard causal language model without separately attaching the adapter.
In this project, the final student checkpoints were merged into FP16 weights.
The notebook also performed a merge-validation check by comparing greedy outputs from the adapter model and its merged counterpart on selected prompts.
All three trained variants passed that check.
This is a useful deployment-oriented validation step.
It helps ensure that the published merged artifact preserves the tested adapter behavior.
It does not establish broad equivalence across every prompt or decoding configuration.
A model should not be evaluated only on the examples it was trained on.
We evaluated M1 through M5 using the same sampled test examples and a common prompt and greedy-decoding configuration.
The experiment used:
The comparison included the original 0.6B baseline, all three adapted students, and the 8B teacher.
Suppose Model A receives a short prompt while Model B receives a much longer prompt containing additional instructions.
Their outputs would not be directly comparable.
The same applies to decoding settings.
Using the same prompt and decoding configuration helps reduce unnecessary differences in the evaluation process.
The notebook also cached predictions, recorded scorer details, and generated per-model reports.
The primary numerical-answer results from the sampled test evaluation were:
1. Original Qwen3–0.6B (Baseline)
FinQA: 0.00
TAT-QA: 0.00
2. Finance-SFT
FinQA: 0.28 (28%)
TAT-QA: 0.40 (40%)
3. Finance-Distilled
FinQA: 0.24 (24%)
TAT-QA: 0.18 (18%)
4. Finance-Distilled-SFT
FinQA: 0.20 (20%)
5. Qwen3–8B (Teacher Reference)
FinQA: 0.48 (48%)
TAT-QA: 0.32 (32%)
Each benchmark column represents 50 sampled examples.
There are two important scoring qualifications.
First, the FinQA metric was a deterministic numeric-answer scorer with an arithmetic-expression execution check. It was not the official FinQA program-matching metric, because the student was trained to output arithmetic expressions rather than the original FinQA DSL program.
Second, the official TAT-QA scorer failed its self-test in this run. The notebook therefore used an explicitly labelled approximate fallback scorer.
The TAT-QA values must not be presented as official benchmark results.
Within these experimental conditions, the SFT student recorded 0.28 on the FinQA numeric scorer and 0.40 on the approximate TAT-QA scorer.
The TAT-QA score was higher than the teacher’s recorded 0.32 on this sample, but because the fallback scorer was used, this should be treated as an exploratory observation rather than evidence that the student is generally better than the teacher.
The useful result is that the three adaptation strategies produced different task-performance profiles, giving us a concrete basis for studying the effect of supervision strategy.
One useful engineering measure was whether the generated responses followed the expected structured format.
FinQA: 96%
TAT-QA: 98%
FinQA: 100%
TAT-QA: 100%
FinQA: 98%
The adapted students consistently produced outputs that followed the requested response structure in the sampled evaluation.
However, format validity should not be confused with numerical correctness. A perfectly formatted answer can still contain an incorrect calculation.
This is why we tracked formatting and answer accuracy separately.
Another useful way to inspect the results is to compare each student against the larger teacher.
A retention value of 100% indicates that the student matches the teacher’s measured score on the evaluated sample. Values below 100% indicate a lower score, while values above 100% indicate a higher score on that particular evaluation.
The notebook also computed 95% confidence intervals using paired bootstrap resampling.
Recorded Overall Retention Results
1. Finance-SFT
Overall measured retention: 85.0%
95% bootstrap confidence interval: 64.5%–112.9%
2. Finance-Distilled
Overall measured retention: 52.5%
95% bootstrap confidence interval: 36.0%–72.7%
3. Finance-Distilled-SFT
Overall measured retention: 47.5%
95% bootstrap confidence interval: 30.1%–66.3%
Interpretation: Finance-SFT achieved the highest measured overall retention at 85.0%, followed by Finance-Distilled at 52.5% and Finance-Distilled-SFT at 47.5%.
The confidence intervals are relatively wide, reflecting the limited evaluation sample. In particular, the Finance-SFT interval extends above 100%, so the point estimate should not be interpreted as evidence of statistical equivalence or superiority over the teacher. These findings are exploratory rather than conclusive.
The SFT student recorded the highest measured overall retention among the three student variants in this particular experiment.
Its benchmark-specific profile is also interesting:
The distillation student recorded approximately 50.0% FinQA retention and 56.3% TAT-QA retention.
The two-stage student recorded approximately 41.7% FinQA retention and 56.3% TAT-QA retention.
These are exploratory ratios, not claims of statistical equivalence with the teacher. The confidence intervals are wide, and the 90% retention target used in the experiment was not reached by any student.
The value of this analysis is that it provides a common framework for comparing student performance against a larger reference model.
It also illustrates why reporting a retention percentage without its underlying absolute score and confidence interval can be misleading.
The notebook also included a small synthetic robustness suite covering:
These probes help reveal whether a model follows the intended response behavior when the prompt differs from a straightforward training example.
Some encouraging observations appeared in this small suite:
These are small-sample observations, not robust generalization claims. There were only two examples per category.
The suite is best viewed as a practical engineering smoke test that helps identify categories for more extensive future evaluation.
It is not a replacement for a large, independently curated robustness benchmark.
This was one of the most interesting engineering parts of the project.
A smaller model is attractive not only because of its parameter count, but also because it can be easier to store, load, and serve.
We measured inference using:
The following measurements were recorded during inference benchmarking on an NVIDIA Tesla T4 GPU.
Latency per request: 2.505 seconds
Throughput: 19.25 tokens/second
Latency per request: 2.367 seconds
Throughput: 20.47 tokens/second
Latency per request: 2.531 seconds
Throughput: 18.93 tokens/second
Latency per request: 2.286 seconds
Throughput: 20.97 tokens/second
Latency per request: 4.480 seconds
Throughput: 11.39 tokens/second
Key Observations
Within this particular short-run measurement, M4 recorded the lowest request latency among the five models.
The student variants also achieved approximately 18.9–21.0 generated tokens per second.
The teacher recorded approximately 11.4 tokens per second.
The student models were measured in FP16, while the teacher was measured in 4-bit NF4. The comparison is therefore not a perfectly like-for-like precision comparison.
These figures should be treated as indicative measurements from the notebook, not production-serving capacity estimates.
Still, the experiment illustrates why model size and deployment efficiency matter.
A specialized model that fits a smaller serving environment can be attractive for workloads where latency, memory, and infrastructure cost matter.
The merged FP16 student checkpoints were approximately 1.11 GB on disk in the measured setup.
The original student model’s recorded Hub weight footprint was approximately 1.4 GB.
The teacher’s original FP16 weights were approximately 15.26 GB, although the teacher was loaded in 4-bit NF4 for the inference measurement.
The following peak GPU memory measurements were recorded during the inference benchmark on an NVIDIA Tesla T4.
Peak GPU memory: 3.30 GB
Peak GPU memory: 2.90 GB
Peak GPU memory: 7.58 GB
Key Observation
All three adapted student models recorded approximately 2.90 GB of peak GPU memory usage, compared with 3.30 GB for the original baseline and 7.58 GB for the teacher.
This suggests that the adapted student models retained a relatively compact inference memory footprint in the tested configuration.
These values are specific to the benchmark configuration, software stack, precision, and hardware.
They should not be treated as universal memory requirements for every serving workload.
The notebook also included an illustrative cost calculation using an assumed GPU rate of USD 0.35 per GPU-hour.
This is a modelling assumption, not a quoted cloud-provider price or an actual Colab bill.
The estimated request cost depends on the benchmark’s short input/output lengths and timing assumptions. It does not include every component of a production service, such as CPU infrastructure, networking, storage, autoscaling, idle capacity, or operational overhead.
The broader engineering lesson is that model adaptation and serving efficiency should be considered together.
A model that is easier to host may be valuable even when its raw benchmark score is not the highest.
The final project consists of a baseline, three adapted students, and the teacher reference.
Each model represents a different point in the experimental workflow.
The unmodified student checkpoint.
It provides the reference point for measuring the effects of task-specific adaptation.
The gold-supervised student.
It was trained directly on financial instruction-response examples using QLoRA.
Within the sampled evaluation, it recorded the strongest student results across both the FinQA numeric-answer scorer and the TAT-QA approximate scorer.
Its defining characteristic is direct adaptation from gold-derived supervision.
The teacher-response student.
It was trained using accepted teacher-generated responses and deterministic synthetic examples.
Its defining contribution is the verified black-box distillation workflow.
Hugging Face: Finance-Distilled
The two-stage student.
It starts from the distilled model and continues training on hard examples, gold-corrected teacher rejections, and replay data.
Its defining contribution is the failure-driven correction pipeline.
Hugging Face: Finance-Distilled-SFT
The larger reference model used for teacher supervision and comparative evaluation.
Together, these five models make the experiment easier to inspect and extend.
The repositories provide a starting point for exploring how the same student architecture behaves under different adaptation strategies.
A free Colab experiment is a useful starting point, but production systems require a different level of engineering.
The methodology can be extended in several directions.
The current experiment used a deliberately compact training pool.
A larger version could incorporate:
The important part is not simply increasing the number of rows.
Data quality, task coverage, answer verification, and document diversity matter just as much.
The response-based distillation pipeline can be expanded with:
A teacher response should not become a trusted target merely because it came from a larger model.
The verification pipeline is just as important as the teacher itself.
The second-stage correction pipeline can become more sophisticated.
For example, failures could be classified into categories:
Instead of treating all failures equally, we could construct targeted training batches for each failure category.
This is a natural next step for a more systematic self-improvement workflow.
A more serious evaluation programme should include:
A model should not be considered production-ready because it performs well on a small sample.
The current run used one training seed, 50 examples per primary benchmark, and a small synthetic robustness suite.
It also used a limited validation sample after sequence-length filtering.
These are reasonable constraints for a prototype, but they are not sufficient for strong generalization claims.
M4 receives additional training compute relative to M2 and M3.
A stronger follow-up experiment should include compute-matched comparisons.
This would help isolate the effect of the two-stage correction strategy from the effect of additional training.
For larger experiments, the same workflow can move from Colab to managed cloud GPUs or dedicated training infrastructure.
The pipeline itself does not need to be redesigned from scratch.
Its stages can be separated into reusable components:
The training infrastructure can then scale independently of the methodology.
The same process can also be adapted to other domains:
The model and dataset will change, but the underlying engineering workflow remains useful.
If you are new to fine-tuning and distillation, here is the sequence I would recommend following.
Start with a model that fits your available hardware and is suitable for your task.
Do not choose the largest model simply because it is available.
Avoid trying to build a general-purpose assistant immediately.
Pick a measurable problem such as classification, extraction, table QA, or structured response generation.
Understand the dataset, inspect examples, define the input/output format, and separate training, validation, and test data.
Evaluate the original model before adaptation.
Otherwise, you cannot determine whether your training changed anything.
Use gold-target examples and learn how tokenization, loss masking, batching, learning rates, and checkpoints work.
Understand why parameter-efficient adaptation is useful and how quantization reduces memory requirements.
Generate candidate responses from a larger model.
Do not automatically trust them.
Use deterministic checks where possible.
Retain accepted examples and record why other examples were rejected.
Use the accepted teacher responses as training targets.
Create a separate mining pool, identify failure categories, construct corrected examples, and experiment with replay.
Use the same prompts, examples, decoding configuration, and scoring procedures across model variants.
Save model cards, training configurations, evaluation summaries, and inference examples.
Make it possible for another engineer to understand what the model actually contains.
The key lesson is that fine-tuning is not simply calling:
trainer.train()
It is an end-to-end process involving data design, supervision, optimization, verification, evaluation, and deployment.
The most valuable outcome of this project is not a claim that a 0.6B model has surpassed an 8B model.
It is the implementation of a complete experimental workflow:
The project demonstrates several practical ideas:
The free Colab Tesla T4 environment imposed limits on dataset size, training duration, and evaluation scale.
Those constraints were intentional parts of the prototype.
With larger datasets, stronger compute, better teacher supervision, more extensive evaluation, and repeated experiments, the same methodology can be extended to build more capable task-specific learner models.
There is a broader engineering lesson here.
A successful model-adaptation project is not defined only by its final checkpoint. It is also defined by the quality of the training data, the reliability of the supervision process, the discipline of the evaluation methodology, and the ability to reproduce and extend the workflow.
You do not need to begin with a massive GPU cluster.
You need a well-defined problem, good data, a sound methodology, and the discipline to measure what you build.
One final clarification: this project is an educational and experimental exercise. It is intended to demonstrate fine-tuning, knowledge distillation, and small-language-model engineering workflows. The resulting models are not production-validated financial systems and should not be used for financial decisions.
The following papers and documentation provide useful foundations for the techniques discussed in this article.
Big Teacher, Small Student: Can a 0.6B Language Model Learn Financial Reasoning from an 8B Model? was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.