{"slug": "big-teacher-small-student-can-a-0-6b-language-model-learn-financial-reasoning-an", "title": "Big Teacher, Small Student: Can a 0.6B Language Model Learn Financial Reasoning from an 8B Model?", "summary": "A project fine-tuned Qwen3-0.6B as a student model on financial question answering using Qwen3-8B as a teacher, which has roughly 10.9 times as many parameters, with training and inference run on a free Google Colab Tesla T4. The work used FinQA, TAT-QA and programmatically generated synthetic financial examples across three adaptation strategies, and the author explicitly did not claim a short training run produces a state-of-the-art financial model. The aim was a complete, extensible pipeline covering data, supervision, training, validation, correction, evaluation and deployment under a constrained compute budget.", "body_md": "*How to take a pretrained language model, specialize it for a real-world task, learn from a larger teacher, and build an extensible training pipeline — even when your compute budget is limited.*\n\nImagine a financial analyst reviewing a company’s annual report.\n\nThe report contains revenue, operating expenses, net income, cash flow, and financial ratios across multiple reporting periods.\n\nA colleague asks:\n\nA general-purpose language model may understand every word in the question and still produce an incorrect answer.\n\nIt might select the wrong row, confuse two reporting periods, overlook a unit conversion, or produce a plausible-looking calculation that is not supported by the source document.\n\nThis is a particularly interesting problem because financial question answering is not simply a language-generation task. It combines document understanding, table interpretation, numerical extraction, arithmetic, reasoning, and structured response generation.\n\nNow consider a different approach.\n\nInstead of training an entirely new language model, what if we started with an existing compact model and taught it to specialize in financial reasoning?\n\nWhat if we could also use a larger model as a teacher, learn from its responses, verify those responses, and transfer the accepted examples to the smaller model?\n\nAnd what if we could identify the smaller model’s mistakes and use those mistakes to construct a second training stage?\n\nThat is the idea behind this project.\n\nI started with Qwen3–0.6B as the student model and Qwen3–8B as the teacher. I explored three adaptation strategies:\n\nThe experiments used FinQA and TAT-QA, along with programmatically generated synthetic financial examples. Training and inference were conducted using a free Google Colab Tesla T4 environment.\n\nThe objective was not to claim that a short training run can produce a state-of-the-art financial model. It was to implement a complete, extensible methodology for adapting small language models under constrained compute.\n\nThe interesting part is not just that three models were trained.\n\nIt is how the data, supervision, training, validation, correction, evaluation, and deployment stages fit together.\n\nLet’s start at the beginning.\n\nOne of the most common misconceptions when starting with language models is that building a specialized model requires training one from scratch.\n\nThat is rarely the most practical starting point.\n\n**Training from scratch** means teaching a model general language patterns and representations using a very large corpus. This involves substantial data, compute, optimization, and engineering.\n\n**Fine-tuning** starts with a model that has already learned useful language representations. We then adapt its behavior to a narrower task.\n\nThink of a newly hired financial analyst.\n\nThey already understand accounting terminology, arithmetic, spreadsheets, and business communication. You do not need to teach them English or mathematics from the beginning.\n\nYou need to teach them your organization’s reporting conventions, calculation formats, and review procedures.\n\nThat is the intuition behind fine-tuning.\n\nA pretrained model already has useful capabilities. Fine-tuning changes its parameters so that it becomes more effective at a particular set of tasks.\n\nFor this project, the starting models were:\n\nThe Qwen3–0.6B model is the compact student, while Qwen3–8B is the larger teacher.\n\nThe teacher has roughly 10.9 times as many parameters as the student, based on the parameter counts recorded during the notebook’s model inspection.\n\nThe idea was to investigate whether a compact student could acquire useful financial task behavior through carefully designed supervision.\n\nWe did not build a new Transformer architecture. We adapted an existing pretrained one.\n\nThat distinction is important because it changes the entire economics of the experiment.\n\nA model cannot learn a specialized task effectively if the training examples do not represent that task.\n\nWe selected two datasets that expose different aspects of financial question answering.\n\n**Dataset:** [FinQA — GitHub repository](https://github.com/czyssrs/FinQA)\n\nFinQA focuses on numerical reasoning over financial reports and tables.\n\nConsider this simplified example:\n\nFinancial metric20212022Revenue100120Operating expenses6070Net income1520\n\nQuestion:\n\n*What was the percentage increase in revenue?*\n\nThe model must:\n\nIdentify the revenue row.\n\nLocate the values for 2021 and 2022.\n\nDetermine the appropriate mathematical operation.\n\nExecute the calculation.\n\nReturn the answer with the correct scale.\n\nThe calculation is:\n\nGrowth = (120–100) / 100 × 100 = 20%\n\nThis is more than language understanding. It combines table interpretation, numerical extraction, arithmetic, and answer formatting.\n\nA model that understands the question but selects the wrong year has failed. A model that selects the correct numbers but performs the wrong operation has also failed.\n\nThe task therefore provides a useful environment for studying numerical reasoning in language models.\n\n**Dataset:** [TAT-QA — GitHub repository](https://github.com/NExTplusplus/TAT-QA)\n\nTAT-QA contains questions that require interpreting both tables and textual context.\n\nFor example, a question might ask for a financial measure whose value appears in a table but whose definition or reporting context is explained in an accompanying paragraph.\n\nThe model must connect information across different parts of a document.\n\nThis makes TAT-QA a useful complement to FinQA.\n\nFinQA emphasizes numerical reasoning over financial tables and reports, while TAT-QA introduces a hybrid table-and-text question-answering setting.\n\nFinancial question answering is not one homogeneous task.\n\nA model may be able to calculate a percentage but struggle to locate the correct values. It may read a table correctly but confuse the unit or scale. It may understand a textual explanation but fail to combine it with a table.\n\nUsing both datasets creates a more varied task environment.\n\nThe original dataset repositories contain substantially more examples than we used in this prototype. The notebook loaded the datasets and constructed a deliberately smaller experimental pool to keep training and teacher inference tractable on a free T4 GPU.\n\nThe selected Stage 1 training pool contained **500 examples**, distributed across three sources:\n\n**Total: 500 training examples**\n\nThe synthetic examples were generated programmatically with deterministic gold answers.\n\nThey covered different financial metrics, reporting periods, numerical magnitudes, units, and question types.\n\nWe also prepared separate **validation**, **mining**, **test**, and **robustness** pools.\n\nThe notebook included **document-overlap**, **exact-duplicate**, **near-duplicate**, and **contradictory-example** checks. The reported leakage checks found no document overlap between the relevant evaluation splits and no exact or near-duplicate examples removed by those checks.\n\nThis matters.\n\nSuppose the same annual-report table appears in both training and testing. The model might appear to perform well because it has already encountered the example.\n\nThat would not be a meaningful test of generalization.\n\nData preparation is therefore not just a preliminary step. It is part of the model-quality process.\n\nBefore writing any training code, it helps to define the experimental design.\n\n**M1 — Base Student (Qwen3–0.6B)**\n\nThe original, untuned model used as the student baseline.\n\n**M2 — Supervised Fine-Tuning (Qwen3–0.6B + SFT)**\n\nThe student adapted using gold-standard supervised training examples.\n\n**M3 — Knowledge Distillation (Qwen3–0.6B + Distillation)**\n\nThe student trained on teacher-generated responses that passed deterministic verification.\n\n**M4 — Sequential Adaptation (Qwen3–0.6B + Distillation + SFT)**\n\nA two-stage approach combining response-based distillation with targeted supervised fine-tuning.\n\n**M5 — Teacher Reference (Qwen3–8B)**\n\nThe larger model used to generate candidate responses and provide a teacher-performance reference.\n\nThe important design choice was to keep the same underlying student model for M1 through M4.\n\nThis gives us a common starting point for studying different adaptation recipes.\n\nThe teacher serves two purposes:\n\nM1 gives us a baseline. M2, M3, and M4 explore different ways of changing the student. M5 provides a larger reference point.\n\nThink of it as three different training programmes for the same analyst.\n\nOne learns directly from verified examples. Another learns from a more experienced analyst. The third learns from that experienced analyst and then receives targeted corrections based on mistakes.\n\nThe experimental design lets us study these different supervision strategies within a common model family.\n\nOne of the most important engineering decisions was to define the model’s expected response format before training.\n\nInstead of allowing the model to return an unconstrained paragraph, we used a structured three-part response:\n\n```\nCalculation: <arithmetic expression = value, or n/a>Answer: <final answer>Scale: <thousand / million / billion / percent / none>\n```\n\nFor example:\n\n```\nCalculation: (120 - 100) / 100 * 100 = 20Answer: 20%Scale: percent\n```\n\nFor insufficient context, the expected answer was:\n\n```\nAnswer: NOT_ANSWERABLE\n```\n\nThe prompt also instructed the model to return CONFLICTING_INFORMATION when the supplied context contained conflicting values for the same quantity.\n\nWhy is this useful?\n\nImagine receiving the following answer:\n\nRevenue increased by 20 percent, which indicates positive growth.\n\nIt sounds reasonable, but it does not expose the calculation.\n\nNow compare it with:\n\n```\nCalculation: (120 - 100) / 100 * 100 = 20Answer: 20%Scale: percent\n```\n\nThe second response is easier to parse, inspect, and validate.\n\nA structured output contract also creates opportunities for deterministic checks:\n\nThis does not guarantee that the model has reasoned correctly. It does, however, make its output more amenable to automated evaluation.\n\nThat distinction is important:\n\n**Format validity is not the same as answer correctness.**\n\nThis separation between response structure and answer quality becomes especially important when building automated training-data validation.\n\nLet’s begin with the most direct approach.\n\nSuppose we want a model to answer financial questions in a particular format.\n\nWe provide examples containing:\n\nThe model learns to generate the target response when it receives the corresponding input.\n\nA simplified training example might look like this:\n\n```\nSYSTEM:You are a careful financial analyst.Use only the supplied context.\nUSER:Context:Revenue in 2021: 100Revenue in 2022: 120\nQuestion:What was the percentage increase?\nASSISTANT:Calculation: (120 - 100) / 100 * 100 = 20Answer: 20%Scale: percent\n```\n\nThe model processes the sequence as tokens and learns to predict the next token.\n\nDuring training, we compare its predictions with the expected target tokens.\n\nFor a causal language model, the standard supervised objective is next-token prediction.\n\nTraining Objective (Negative Log-Likelihood Loss)\n\nWhere:\n\nThe model is trained to assign higher probability to the expected response.\n\nAn important implementation detail is **response-only loss masking**.\n\nWe do not necessarily want the model to be penalized for predicting the prompt itself. The notebook masks prompt-token labels and computes the training loss on the response portion.\n\nConceptually:\n\n```\nlabels = input_ids.clone()\n# Exclude prompt tokens from the loss.labels[:prompt_length] = -100\n# Supervise the response tokens.loss = model(    input_ids=input_ids,    labels=labels).loss\n```\n\nThis is a simplified illustration of the loss-masking idea, not a complete training implementation.\n\nThe actual pipeline also handles tokenization, padding, truncation, batching, attention masks, and checkpoint management.\n\nThe SFT model used **500** training examples:\n\nThe targets were derived from gold annotations and deterministic synthetic generation.\n\nFor FinQA, the notebook constructed arithmetic-expression targets from the gold programs or derivations. For TAT-QA, it used the corresponding gold answer annotations.\n\nNo teacher-generated responses or teacher logits were used for this experiment.\n\nThe resulting model became our first task-adapted student.\n\nThis is the most straightforward approach to understand and implement when starting with fine-tuning.\n\nThe student model contains hundreds of millions of parameters.\n\nUpdating every parameter would require storing more than just the model weights.\n\nTraining also requires memory for gradients, optimizer states, intermediate activations, and other runtime allocations.\n\nFor a limited GPU environment, that can become a significant constraint.\n\nThis is where parameter-efficient fine-tuning becomes useful.\n\nLoRA stands for **Low-Rank Adaptation**.\n\nInstead of updating the original weight matrix directly, LoRA freezes the original matrix and introduces a smaller trainable update.\n\n**LoRA Weight Update**\n\nThe key insight is that the model does not necessarily need an unrestricted update to every original parameter to adapt to a downstream task.\n\nIt can learn a constrained update through much smaller matrices.\n\nFor an intuitive example, imagine modifying a large spreadsheet formula system.\n\nInstead of rewriting every formula, you introduce a compact set of adjustments that influence the relevant calculations.\n\nLoRA follows a similar principle.\n\nQLoRA adds another memory-saving technique.\n\nThe pretrained model is loaded using 4-bit quantized weights, while the trainable LoRA adapters remain in higher precision.\n\nThe base model remains frozen, and the adapters receive the parameter updates.\n\nThe major components include:\n\nThe notebook used a QLoRA-style setup with:\n\nThis was particularly useful because the Colab Tesla T4 does not support BF16 in the same way as newer GPU architectures.\n\nA simplified configuration looks like this:\n\n``` python\nfrom transformers import BitsAndBytesConfigimport torch\nquant_config = BitsAndBytesConfig(    load_in_4bit=True,    bnb_4bit_quant_type=\"nf4\",    bnb_4bit_use_double_quant=True,    bnb_4bit_compute_dtype=torch.float16)\n```\n\nAnd a simplified LoRA configuration:\n\n``` python\nfrom peft import LoraConfig\nlora_config = LoraConfig(    r=16,    lora_alpha=32,    lora_dropout=0.05,    task_type=\"CAUSAL_LM\")\n```\n\nThe actual notebook applied adapters to attention and MLP projections, alongside gradient checkpointing and memory-aware training configuration.\n\nFor this run, the trainable adapter parameter count was approximately **10.09 million**.\n\nThat is a compact trainable parameter set relative to the underlying student model.\n\nIt is useful to understand what this does and does not mean.\n\nQLoRA reduces the memory footprint of the **frozen base weights** during training. It does not eliminate the need to load the base model or perform the forward and backward computations required to train the adapters.\n\nNow we can put the pieces together.\n\nThe first adapted model was M2, the Finance-SFT student.\n\nIts training pipeline was:\n\n**Model: Qwen3–0.6B + Supervised Fine-Tuning (SFT)**\n\n**Effective Batch Size Calculation**\n\nThe effective batch size was achieved using a per-device batch size of 1 and gradient accumulation over 4 steps.\n\n**Effective Batch Size** = 1 × 4 = 4\n\nThe training run completed successfully.\n\nThe best recorded validation loss was approximately 0.3146, and the selected checkpoint was at step 120.\n\nOne important detail: although the validation pool contained more examples, only 12 examples survived the configured validation selection and sequence-length filtering for this training run. This is a small checkpoint-selection sample, so the recorded validation loss should be interpreted accordingly.\n\nThe adapter was then merged into the base model. The merge-validation procedure reported identical greedy outputs between the adapter and merged versions on the prompts used for that check.\n\nThat is useful evidence that the merge preserved the tested behavior.\n\nIt is not a guarantee of identical behavior across every possible input.\n\nWe now had a model adapted directly to the financial task using verified target responses.\n\nBut there is another interesting question:\n\nWhat if the training targets came from a larger model instead of being constructed directly from gold answers?\n\nThat brings us to knowledge distillation.\n\nImagine a junior analyst learning from a senior analyst.\n\nThe senior analyst receives a financial question and explains how they would solve it. The junior analyst studies the explanation and learns to produce similar responses.\n\nThat is the basic intuition behind teacher-student learning.\n\nIn machine learning, a larger teacher model provides information that can be transferred to a smaller student.\n\nHowever, there are different ways to do this.\n\nIn classical logit-based distillation, the student is trained to approximate the teacher’s output probability distribution.\n\nThe teacher produces a probability distribution over possible next tokens.\n\nThe student produces its own distribution.\n\n**KL-Divergence Distillation Objective**\n\nThis approach requires access to teacher logits or probability distributions.\n\nWhat if the teacher is available only through its generated responses?\n\nWe can still perform a useful form of knowledge transfer.\n\nThe teacher receives a prompt and produces a response. We then use that response as a candidate training target for the student.\n\nThis is often called response-based or black-box distillation.\n\nIt does not require teacher logits.\n\nThat was the approach used in this project.\n\nThe teacher generated structured financial calculation traces and answers. The candidate responses were then passed through deterministic validation.\n\nThe important point is that we did not blindly trust every teacher response.\n\nA larger model can also make mistakes.\n\nConsider a teacher response:\n\n```\nCalculation: (120 - 100) / 100 = 0.2Answer: 20%Scale: percent\n```\n\nThe arithmetic expression evaluates to 0.2, and the final answer is consistent with 20%.\n\nNow imagine a response where the calculation evaluates to 0.2 but the final answer says 25%.\n\nThat response should not become a training target.\n\nThe notebook implemented validation checks for:\n\nOnly responses passing the required checks were accepted for the teacher-supervised training set.\n\nThis is a crucial distinction:\n\n**We used the teacher to generate candidate supervision, but the gold annotations acted as the verification mechanism.**\n\nThe approach is therefore not unsupervised distillation.\n\nIt is verified, response-based knowledge distillation using rejection sampling.\n\nThe notebook used Qwen3–8B as the teacher in local 4-bit mode.\n\nThe teacher-supervision stage contained **500** examples in total:\n\nOf the 400 newly generated responses, 149 passed the configured deterministic verification process.\n\nThat represents a teacher-response acceptance rate of:\n\nTeacher Response Acceptance Rate = (149 ÷ 400) × 100 = **37.25%**\n\nThe rejected responses were not simply ignored. Their corresponding gold targets were retained as potential correction examples for the second-stage experiment.\n\nThe accepted teacher responses were combined with 100 deterministic synthetic examples to construct the M3 target pool.\n\nAfter sequence-length filtering, 247 examples were used for M3 training.\n\nThe pipeline can be represented as:\n\nThis approach gives us a practical way to use a larger model as a source of candidate training examples without needing access to its internal logits.\n\nM3 started from the same Qwen3–0.6B base model as M2.\n\nThe difference was the supervision source.\n\nM2 learned from gold-derived targets.\n\nM3 learned from verified teacher-generated responses, combined with the deterministic synthetic examples.\n\n**Model: Qwen3–0.6B + Response-Based Knowledge Distillation**\n\nThe best recorded validation loss was approximately 0.4801, with the selected checkpoint at step 20.\n\nThe adapter was merged into the base model and passed the same greedy-output merge validation used for M2.\n\nThe key engineering outcome was not simply another trained checkpoint.\n\nIt was a reusable teacher-supervision pipeline:\n\nThat workflow can be reused with a different teacher, student, or task.\n\nIt also illustrates why teacher quality alone is not enough.\n\nA useful distillation pipeline needs a mechanism for deciding which teacher responses should become training targets.\n\nThis was the most interesting part of the experimental design.\n\nA student that has been trained on teacher responses is not necessarily finished.\n\nIt may still struggle with certain examples.\n\nWhat if we could identify those failures and use them to create a targeted correction dataset?\n\nThat is the motivation behind M4.\n\nThink about a student preparing for an examination.\n\nThey complete an initial practice test. Instead of repeating the entire syllabus, a teacher identifies the questions they struggled with and prepares additional exercises around those concepts.\n\nThe next training stage becomes more targeted.\n\nThe same idea can be applied to language-model adaptation.\n\nWe start with the distilled student, evaluate it on a separate mining pool, and identify examples where it fails.\n\nThese are not official validation or test examples. The mining pool was constructed separately from the final test subset.\n\nKeeping the mining pool separate from final evaluation data is essential.\n\nThe notebook used a 50-example mining pool.\n\nM3 failed 41 of those 50 examples.\n\nThe second-stage training dataset combined three complementary sources:\n\n**1. Hard Examples — 41 examples**\n\nCorrect examples identified through student failure analysis, targeting areas where the distilled model struggled.\n\n**2. Teacher-Rejection Corrections — 251 examples**\n\nGold-corrected examples corresponding to teacher-generated responses that failed the verification process.\n\n**3. Replay Examples — 125 examples**\n\nSelected examples retained from the first training stage to preserve previously acquired capabilities and reduce forgetting.\n\n**Total constructed dataset: 417 examples**\n\nThis composition combined targeted error correction, reliable gold supervision, and replay-based retention. One example was subsequently dropped during sequence-length filtering, leaving **416 usable training examples** for stage-two training.\n\nAfter sequence-length filtering, 416 examples were used.\n\nLet’s understand each component.\n\nThe distilled student was evaluated on the separate mining pool.\n\nExamples where it failed were selected for correction.\n\nThis provides a way to make the next training stage more responsive to observed errors.\n\nRather than training on a completely random collection of additional examples, we can use the model’s own performance to help prioritize what it should learn next.\n\nSome teacher responses did not pass the verification procedure.\n\nRather than automatically discarding the corresponding examples from the entire project, the pipeline used their gold targets to construct corrected training examples for the second stage.\n\nThis turns the teacher-validation process into an additional source of useful training data.\n\nThe teacher’s rejected response is not treated as correct. The gold annotation provides the correction.\n\nTraining only on new examples can cause the model to lose useful behavior acquired during the first stage.\n\nReplay addresses this by mixing a subset of earlier verified training examples into the second-stage dataset.\n\nIt provides continuity between the original distillation objective and the targeted correction objective.\n\nM4 continued from the best M3 adapter.\n\n**Model: Qwen3–0.6B + Distillation + Targeted SFT**\n\nNotice the lower learning rate.\n\nThe model already had a task-specific initialization from the first stage. A smaller learning rate was used for continued adaptation.\n\nThe best recorded validation loss was approximately 0.4005, with the selected checkpoint at step 100.\n\nThe final model was merged and passed the same adapter-versus-merged greedy-output validation.\n\nThis is a useful experimental pattern because it separates initial knowledge transfer from targeted correction.\n\nIt also provides a natural extension point for future work: improve the mining strategy, expand the correction dataset, and evaluate whether the changes produce repeatable gains across multiple seeds and larger test sets.\n\nOne experimental-design qualification is worth making explicit: M4 receives additional training compute compared with M2 and M3. Consequently, any observed difference between M4 and the other students cannot be attributed solely to the two-stage recipe. More training steps are another possible contributor.\n\nThat is precisely the kind of confound a larger follow-up experiment should control.\n\nCompute constraints influenced almost every engineering decision.\n\nThe notebook detected an NVIDIA Tesla T4 with approximately 14.56 GB of GPU memory.\n\nA T4 is not equivalent to a modern high-end training GPU. We therefore used a configuration designed around the available resources.\n\nThe key decisions were:\n\nThe base model was loaded in 4-bit NF4 for training.\n\nThis reduced the memory footprint of the frozen model weights.\n\nThe T4 environment used FP16 mixed precision because BF16 was not supported by the detected GPU architecture in this setup.\n\nThis is a practical reminder that precision settings should be selected based on the hardware, not copied blindly from another notebook.\n\nGradient checkpointing reduces activation-memory requirements by recomputing selected activations during the backward pass.\n\nThis trades additional computation for lower memory consumption.\n\nThe training configuration used a memory-conscious optimizer setup.\n\nThe per-device batch size was 1, with gradient accumulation of 4.\n\nThe notebook wrote data, teacher responses, checkpoints, predictions, and reports to Google Drive.\n\nIt used cached outputs and checkpoint-resume behavior so that an interrupted Colab session would not necessarily require repeating every completed stage.\n\nThis is an underrated part of practical ML engineering.\n\nA notebook that trains successfully only when uninterrupted is less useful than one designed to recover from runtime disconnections.\n\nThe pipeline was organized into distinct stages, making it possible to resume teacher generation and prediction generation from cached outputs and continue training from saved checkpoints.\n\nThat is the difference between a one-off demonstration and a reusable experiment workflow.\n\n**M2 — Gold-Target SFT**\n\nTraining duration: 837.5 seconds (approximately 14 minutes)\n\n**M3 — Response-Based Distillation**\n\nTraining duration: 404.2 seconds (approximately 6.7 minutes)\n\n**M4 — Distillation + Targeted SFT**\n\nTraining duration: 342.0 seconds (approximately 5.7 minutes)\n\n**Note:** These durations represent student training time only. Teacher-response generation and verification are separate pipeline stages and are not included in these figures.\n\nThe teacher-supervision stage also required a separate inference run. Its recorded validation summary reported approximately 30 minutes of elapsed time.\n\nThese numbers describe the notebook’s recorded execution, not a universal training-time estimate. Runtime depends on the Colab allocation, software versions, caching, model loading, sequence lengths, and other factors.\n\nThe important point is that the pipeline was engineered to fit a constrained environment rather than assuming access to a large multi-GPU cluster.\n\nA training loss curve can be useful, but it does not tell the whole story.\n\nThe notebook tracked training and validation loss during each experiment.\n\nBest checkpoint: Step 120\n\nBest recorded validation loss: 0.3146\n\nBest checkpoint: Step 20\n\nBest recorded validation loss: 0.4801\n\nBest checkpoint: Step 100\n\nBest recorded validation loss: 0.4005\n\n**Note:** The best checkpoint for each experiment was selected based on its recorded validation loss. Lower validation loss indicates a better fit to the validation data under the training objective; it does not, by itself, establish superior task-level reasoning or generalization.\n\nThese values are useful for understanding the optimization runs, but they should not be compared as direct measures of model quality across different training-target distributions.\n\nFor example, the SFT targets and teacher-generated targets are not identical distributions.\n\nThe validation sample was also small: 12 usable examples were retained for the training-time validation procedure after sequence-length filtering. This is not a sufficiently large validation set for making strong statistical claims about generalization.\n\nDuring QLoRA training, the model consists of:\n\nFor deployment, it is often convenient to merge the adapter into the base model.\n\n**LoRA Weight Merging**\n\nThe resulting checkpoint can be loaded as a standard causal language model without separately attaching the adapter.\n\nIn this project, the final student checkpoints were merged into FP16 weights.\n\nThe notebook also performed a merge-validation check by comparing greedy outputs from the adapter model and its merged counterpart on selected prompts.\n\nAll three trained variants passed that check.\n\nThis is a useful deployment-oriented validation step.\n\nIt helps ensure that the published merged artifact preserves the tested adapter behavior.\n\nIt does not establish broad equivalence across every prompt or decoding configuration.\n\nA model should not be evaluated only on the examples it was trained on.\n\nWe evaluated M1 through M5 using the same sampled test examples and a common prompt and greedy-decoding configuration.\n\nThe experiment used:\n\nThe comparison included the original 0.6B baseline, all three adapted students, and the 8B teacher.\n\nSuppose Model A receives a short prompt while Model B receives a much longer prompt containing additional instructions.\n\nTheir outputs would not be directly comparable.\n\nThe same applies to decoding settings.\n\nUsing the same prompt and decoding configuration helps reduce unnecessary differences in the evaluation process.\n\nThe notebook also cached predictions, recorded scorer details, and generated per-model reports.\n\nThe primary numerical-answer results from the sampled test evaluation were:\n\n**1. Original Qwen3–0.6B (Baseline)**\n\nFinQA: 0.00\n\nTAT-QA: 0.00\n\n**2. Finance-SFT**\n\nFinQA: 0.28 (28%)\n\nTAT-QA: 0.40 (40%)\n\n**3. Finance-Distilled**\n\nFinQA: 0.24 (24%)\n\nTAT-QA: 0.18 (18%)\n\n**4. Finance-Distilled-SFT**\n\nFinQA: 0.20 (20%)\n\n**5. Qwen3–8B (Teacher Reference)**\n\nFinQA: 0.48 (48%)\n\nTAT-QA: 0.32 (32%)\n\nEach benchmark column represents **50** sampled examples.\n\nThere are two important scoring qualifications.\n\nFirst, the FinQA metric was a deterministic numeric-answer scorer with an arithmetic-expression execution check. It was not the official FinQA program-matching metric, because the student was trained to output arithmetic expressions rather than the original FinQA DSL program.\n\nSecond, the official TAT-QA scorer failed its self-test in this run. The notebook therefore used an explicitly labelled approximate fallback scorer.\n\nThe TAT-QA values must not be presented as official benchmark results.\n\nWithin these experimental conditions, the SFT student recorded 0.28 on the FinQA numeric scorer and 0.40 on the approximate TAT-QA scorer.\n\nThe TAT-QA score was higher than the teacher’s recorded 0.32 on this sample, but because the fallback scorer was used, this should be treated as an exploratory observation rather than evidence that the student is generally better than the teacher.\n\nThe useful result is that the three adaptation strategies produced different task-performance profiles, giving us a concrete basis for studying the effect of supervision strategy.\n\nOne useful engineering measure was whether the generated responses followed the expected structured format.\n\nFinQA: 96%\n\nTAT-QA: 98%\n\nFinQA: 100%\n\nTAT-QA: 100%\n\nFinQA: 98%\n\nThe adapted students consistently produced outputs that followed the requested response structure in the sampled evaluation.\n\nHowever, format validity should not be confused with numerical correctness. A perfectly formatted answer can still contain an incorrect calculation.\n\nThis is why we tracked formatting and answer accuracy separately.\n\nAnother useful way to inspect the results is to compare each student against the larger teacher.\n\nA retention value of 100% indicates that the student matches the teacher’s measured score on the evaluated sample. Values below 100% indicate a lower score, while values above 100% indicate a higher score on that particular evaluation.\n\nThe notebook also computed 95% confidence intervals using paired bootstrap resampling.\n\n**Recorded Overall Retention Results**\n\n**1. Finance-SFT**\n\nOverall measured retention: 85.0%\n\n95% bootstrap confidence interval: 64.5%–112.9%\n\n**2. Finance-Distilled**\n\nOverall measured retention: 52.5%\n\n95% bootstrap confidence interval: 36.0%–72.7%\n\n**3. Finance-Distilled-SFT**\n\nOverall measured retention: 47.5%\n\n95% bootstrap confidence interval: 30.1%–66.3%\n\n**Interpretation:** Finance-SFT achieved the highest measured overall retention at 85.0%, followed by Finance-Distilled at 52.5% and Finance-Distilled-SFT at 47.5%.\n\nThe confidence intervals are relatively wide, reflecting the limited evaluation sample. In particular, the Finance-SFT interval extends above 100%, so the point estimate should not be interpreted as evidence of statistical equivalence or superiority over the teacher. These findings are exploratory rather than conclusive.\n\nThe SFT student recorded the highest measured overall retention among the three student variants in this particular experiment.\n\nIts benchmark-specific profile is also interesting:\n\nThe distillation student recorded approximately 50.0% FinQA retention and 56.3% TAT-QA retention.\n\nThe two-stage student recorded approximately 41.7% FinQA retention and 56.3% TAT-QA retention.\n\nThese are exploratory ratios, not claims of statistical equivalence with the teacher. The confidence intervals are wide, and the 90% retention target used in the experiment was not reached by any student.\n\nThe value of this analysis is that it provides a common framework for comparing student performance against a larger reference model.\n\nIt also illustrates why reporting a retention percentage without its underlying absolute score and confidence interval can be misleading.\n\nThe notebook also included a small synthetic robustness suite covering:\n\nThese probes help reveal whether a model follows the intended response behavior when the prompt differs from a straightforward training example.\n\nSome encouraging observations appeared in this small suite:\n\nThese are small-sample observations, not robust generalization claims. There were only two examples per category.\n\nThe suite is best viewed as a practical engineering smoke test that helps identify categories for more extensive future evaluation.\n\nIt is not a replacement for a large, independently curated robustness benchmark.\n\nThis was one of the most interesting engineering parts of the project.\n\nA smaller model is attractive not only because of its parameter count, but also because it can be easier to store, load, and serve.\n\nWe measured inference using:\n\nThe following measurements were recorded during inference benchmarking on an NVIDIA Tesla T4 GPU.\n\nLatency per request: 2.505 seconds\n\nThroughput: 19.25 tokens/second\n\nLatency per request: 2.367 seconds\n\nThroughput: 20.47 tokens/second\n\nLatency per request: 2.531 seconds\n\nThroughput: 18.93 tokens/second\n\nLatency per request: 2.286 seconds\n\nThroughput: 20.97 tokens/second\n\nLatency per request: 4.480 seconds\n\nThroughput: 11.39 tokens/second\n\n**Key Observations**\n\nWithin this particular short-run measurement, M4 recorded the lowest request latency among the five models.\n\nThe student variants also achieved approximately 18.9–21.0 generated tokens per second.\n\nThe teacher recorded approximately 11.4 tokens per second.\n\nThe student models were measured in FP16, while the teacher was measured in 4-bit NF4. The comparison is therefore not a perfectly like-for-like precision comparison.\n\nThese figures should be treated as indicative measurements from the notebook, not production-serving capacity estimates.\n\nStill, the experiment illustrates why model size and deployment efficiency matter.\n\nA specialized model that fits a smaller serving environment can be attractive for workloads where latency, memory, and infrastructure cost matter.\n\nThe merged FP16 student checkpoints were approximately 1.11 GB on disk in the measured setup.\n\nThe original student model’s recorded Hub weight footprint was approximately 1.4 GB.\n\nThe teacher’s original FP16 weights were approximately 15.26 GB, although the teacher was loaded in 4-bit NF4 for the inference measurement.\n\nThe following peak GPU memory measurements were recorded during the inference benchmark on an NVIDIA Tesla T4.\n\nPeak GPU memory: 3.30 GB\n\nPeak GPU memory: 2.90 GB\n\nPeak GPU memory: 7.58 GB\n\n**Key Observation**\n\nAll three adapted student models recorded approximately 2.90 GB of peak GPU memory usage, compared with 3.30 GB for the original baseline and 7.58 GB for the teacher.\n\nThis suggests that the adapted student models retained a relatively compact inference memory footprint in the tested configuration.\n\nThese values are specific to the benchmark configuration, software stack, precision, and hardware.\n\nThey should not be treated as universal memory requirements for every serving workload.\n\nThe notebook also included an illustrative cost calculation using an assumed GPU rate of USD 0.35 per GPU-hour.\n\nThis is a modelling assumption, not a quoted cloud-provider price or an actual Colab bill.\n\nThe estimated request cost depends on the benchmark’s short input/output lengths and timing assumptions. It does not include every component of a production service, such as CPU infrastructure, networking, storage, autoscaling, idle capacity, or operational overhead.\n\nThe broader engineering lesson is that model adaptation and serving efficiency should be considered together.\n\nA model that is easier to host may be valuable even when its raw benchmark score is not the highest.\n\nThe final project consists of a baseline, three adapted students, and the teacher reference.\n\nEach model represents a different point in the experimental workflow.\n\nThe unmodified student checkpoint.\n\nIt provides the reference point for measuring the effects of task-specific adaptation.\n\nThe gold-supervised student.\n\nIt was trained directly on financial instruction-response examples using QLoRA.\n\nWithin the sampled evaluation, it recorded the strongest student results across both the FinQA numeric-answer scorer and the TAT-QA approximate scorer.\n\nIts defining characteristic is direct adaptation from gold-derived supervision.\n\nThe teacher-response student.\n\nIt was trained using accepted teacher-generated responses and deterministic synthetic examples.\n\nIts defining contribution is the verified black-box distillation workflow.\n\n[Hugging Face: Finance-Distilled](https://huggingface.co/learner-subodh/qwen-finance-distilled-0.6b)\n\nThe two-stage student.\n\nIt starts from the distilled model and continues training on hard examples, gold-corrected teacher rejections, and replay data.\n\nIts defining contribution is the failure-driven correction pipeline.\n\n[Hugging Face: Finance-Distilled-SFT](https://huggingface.co/learner-subodh/qwen-finance-distilled-sft-0.6b)\n\nThe larger reference model used for teacher supervision and comparative evaluation.\n\nTogether, these five models make the experiment easier to inspect and extend.\n\nThe repositories provide a starting point for exploring how the same student architecture behaves under different adaptation strategies.\n\nA free Colab experiment is a useful starting point, but production systems require a different level of engineering.\n\nThe methodology can be extended in several directions.\n\nThe current experiment used a deliberately compact training pool.\n\nA larger version could incorporate:\n\nThe important part is not simply increasing the number of rows.\n\nData quality, task coverage, answer verification, and document diversity matter just as much.\n\nThe response-based distillation pipeline can be expanded with:\n\nA teacher response should not become a trusted target merely because it came from a larger model.\n\nThe verification pipeline is just as important as the teacher itself.\n\nThe second-stage correction pipeline can become more sophisticated.\n\nFor example, failures could be classified into categories:\n\nInstead of treating all failures equally, we could construct targeted training batches for each failure category.\n\nThis is a natural next step for a more systematic self-improvement workflow.\n\nA more serious evaluation programme should include:\n\nA model should not be considered production-ready because it performs well on a small sample.\n\nThe current run used one training seed, 50 examples per primary benchmark, and a small synthetic robustness suite.\n\nIt also used a limited validation sample after sequence-length filtering.\n\nThese are reasonable constraints for a prototype, but they are not sufficient for strong generalization claims.\n\nM4 receives additional training compute relative to M2 and M3.\n\nA stronger follow-up experiment should include compute-matched comparisons.\n\nThis would help isolate the effect of the two-stage correction strategy from the effect of additional training.\n\nFor larger experiments, the same workflow can move from Colab to managed cloud GPUs or dedicated training infrastructure.\n\nThe pipeline itself does not need to be redesigned from scratch.\n\nIts stages can be separated into reusable components:\n\nThe training infrastructure can then scale independently of the methodology.\n\nThe same process can also be adapted to other domains:\n\nThe model and dataset will change, but the underlying engineering workflow remains useful.\n\nIf you are new to fine-tuning and distillation, here is the sequence I would recommend following.\n\nStart with a model that fits your available hardware and is suitable for your task.\n\nDo not choose the largest model simply because it is available.\n\nAvoid trying to build a general-purpose assistant immediately.\n\nPick a measurable problem such as classification, extraction, table QA, or structured response generation.\n\nUnderstand the dataset, inspect examples, define the input/output format, and separate training, validation, and test data.\n\nEvaluate the original model before adaptation.\n\nOtherwise, you cannot determine whether your training changed anything.\n\nUse gold-target examples and learn how tokenization, loss masking, batching, learning rates, and checkpoints work.\n\nUnderstand why parameter-efficient adaptation is useful and how quantization reduces memory requirements.\n\nGenerate candidate responses from a larger model.\n\nDo not automatically trust them.\n\nUse deterministic checks where possible.\n\nRetain accepted examples and record why other examples were rejected.\n\nUse the accepted teacher responses as training targets.\n\nCreate a separate mining pool, identify failure categories, construct corrected examples, and experiment with replay.\n\nUse the same prompts, examples, decoding configuration, and scoring procedures across model variants.\n\nSave model cards, training configurations, evaluation summaries, and inference examples.\n\nMake it possible for another engineer to understand what the model actually contains.\n\nThe key lesson is that fine-tuning is not simply calling:\n\n```\ntrainer.train()\n```\n\nIt is an end-to-end process involving data design, supervision, optimization, verification, evaluation, and deployment.\n\nThe most valuable outcome of this project is not a claim that a 0.6B model has surpassed an 8B model.\n\nIt is the implementation of a complete experimental workflow:\n\nThe project demonstrates several practical ideas:\n\nThe free Colab Tesla T4 environment imposed limits on dataset size, training duration, and evaluation scale.\n\nThose constraints were intentional parts of the prototype.\n\nWith larger datasets, stronger compute, better teacher supervision, more extensive evaluation, and repeated experiments, the same methodology can be extended to build more capable task-specific learner models.\n\nThere is a broader engineering lesson here.\n\nA successful model-adaptation project is not defined only by its final checkpoint. It is also defined by the quality of the training data, the reliability of the supervision process, the discipline of the evaluation methodology, and the ability to reproduce and extend the workflow.\n\nYou do not need to begin with a massive GPU cluster.\n\nYou need a well-defined problem, good data, a sound methodology, and the discipline to measure what you build.\n\n**One final clarification:** this project is an educational and experimental exercise. It is intended to demonstrate fine-tuning, knowledge distillation, and small-language-model engineering workflows. The resulting models are not production-validated financial systems and should not be used for financial decisions.\n\nThe following papers and documentation provide useful foundations for the techniques discussed in this article.\n\n[Big Teacher, Small Student: Can a 0.6B Language Model Learn Financial Reasoning from an 8B Model?](https://pub.towardsai.net/big-teacher-small-student-can-a-0-6b-language-model-learn-financial-reasoning-from-an-8b-model-5bc53640f6f3) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.", "url": "https://wpnews.pro/news/big-teacher-small-student-can-a-0-6b-language-model-learn-financial-reasoning-an", "canonical_source": "https://pub.towardsai.net/big-teacher-small-student-can-a-0-6b-language-model-learn-financial-reasoning-from-an-8b-model-5bc53640f6f3?source=rss----98111c9905da---4", "published_at": "2026-11-12 17:01:02+00:00", "updated_at": "2026-11-12 17:19:34.791010+00:00", "lang": "en", "topics": ["large-language-models", "machine-learning", "ai-research", "ai-tools"], "entities": ["Qwen3-0.6B", "Qwen3-8B", "FinQA", "TAT-QA", "Google Colab", "Tesla T4"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/big-teacher-small-student-can-a-0-6b-language-model-learn-financial-reasoning-an", "markdown": "https://wpnews.pro/news/big-teacher-small-student-can-a-0-6b-language-model-learn-financial-reasoning-an.md", "text": "https://wpnews.pro/news/big-teacher-small-student-can-a-0-6b-language-model-learn-financial-reasoning-an.txt", "jsonld": "https://wpnews.pro/news/big-teacher-small-student-can-a-0-6b-language-model-learn-financial-reasoning-an.jsonld"}}