In my previous article “Get LLMs To Say More With Less*”*, we established the following goal: To train a small language model to write simpler English without sacrificing correctness. Supervised fine-tuning (SFT) worked surprisingly well. With a relatively small dataset, the model learned to produce responses that were shorter, clearer, and closer to ASD-STE100-style technical English.
But SFT also exposed a harder problem. Improving one behaviour can distort others. In our case, it sometimes made the model’s output distribution too sharp, leading to repetitive loops especially under greedy decoding strategies.
This article is a more technical follow-up to the original post. We will look at the failure modes encountered after SFT, why common inference-time fixes only treat the symptoms, and how Group Relative Policy Optimisation (GRPO) was used to optimise the model against explicit rewards for correctness, simplicity, clarity, and ASD-STE100 compliance.
Supervised fine-tuning produced a meaningful shift in behaviour. Semantic simplicity improved from 65.82% to 85.46%, while technical adequacy slightly improved from 68.30% to 70.15%. The model also preserved most of its task-fulfilment and clarity scores. In other words, SFT largely achieved what we wanted: simpler responses without an obvious collapse in correctness.
The problem appeared during generation.
Under some prompts, especially with greedy decoding, the SFT model could fall into repetitive loops. For example:
…from a single exit point from a single next hop from a single next hop from a single next hop…
Once the model entered one of these loops, it often had difficulty escaping.
This was not enough to invalidate the SFT result. Most generations were fine, and the aggregate evaluation metrics were strong. But it suggested that the behavioural shift we had learned came with an undesirable side effect.
SFT trains the model to increase the probability of the target tokens in the training examples.
At a high level, it minimises the negative log-likelihood:
If the training data consistently contains shorter, simpler, more direct answers, the model learns to assign more probability to those kinds of continuations. But the objective does not directly care about the shape of the full output distribution.
In our case, the looping behaviour was consistent with the fine-tuned model becoming too confident in a small set of continuations under certain contexts.
This becomes especially problematic with greedy decoding:
Greedy decoding always picks the single highest-probability next token. That works well when the distribution is well behaved, but it can amplify local pathologies.
If one continuation produces a context that again strongly favours a similar continuation, the process can become self-reinforcing: context → high-probability continuation → similar context → same continuation → ⋯ The model had learned the right general behaviour, but in some local regions of the generation space, the learned update appeared to be too aggressive.
Rather than retrain immediately, I first tried to make generation more robust at inference time.
First, I moved away from purely greedy decoding and introduced stochastic decoding.
Instead of always choosing the highest-probability token, sampling allows lower-ranked but still plausible tokens to be selected. This gives the model a way to escape deterministic cycles rather than repeatedly following the same local maximum.
Second, I added repetition and presence penalties.
These reduce the attractiveness of tokens or phrases that have already appeared in the generated text. They directly discourage the model from repeatedly returning to the same continuation.
Third, I reduced the LoRA adapter scale.
A LoRA-adapted layer can be written approximately as:
Reducing s weakens the contribution of the adapter while keeping the underlying base model unchanged.
This was particularly interesting because it suggested something about what the SFT adapter had learned. The direction of the update appeared useful: the adapter clearly pushed the model toward simpler language. But the full-strength update seemed too strong in some contexts.
Scaling the adapter down preserved much of the desired behaviour while reducing the tendency to fall into repetitive loops.
These inference time fixes worked, but it felt like an operational hack.
Stochastic decoding, repetition penalties, and adapter scaling were effectively acting as guardrails around the fine-tuned model. Ideally, the model should learn the desired behavioural shift without needing those guardrails.
We could probably have squeezed out more performance from SFT by experimenting with a lower learning rate or earlier stopping. More diverse training data could also reduce the chances of over-specialising on a narrow set of response patterns.
But I decided to try a slightly different approach.
SFT only tells the model to imitate the examples it is given. What if we could instead tell the model directly which outputs were better?
Group Relative Policy Optimisation (GRPO) is a reinforcement-learning method for post-training language models.
The key difference from SFT is the learning signal. SFT learns by imitation: given a prompt and a target response, it increases the probability of the target tokens. GRPO does not require a single “correct” target response. Instead, it generates several candidate responses for the same prompt, scores them with a reward function, and updates the model to make higher-reward responses more likely relative to the rest of the group.
This makes GRPO useful when the desired behaviour is easier to evaluate than to demonstrate perfectly in a dataset.
The first reward function that we tried treated correctness as a multiplier over the style objectives:
where C is correctness, S is simplicity, A is ASD-STE100 compliance, and the weights control the relative importance of the style objectives.
The intuition was that incorrect answers should receive little or no reward, while correct answers should be rewarded for writing simply.
In practice, however, correctness was something the model could improve just by adding more detail. Longer answers could include extra explanations and caveats that made them appear more technically complete, even when that detail was unnecessary. The result was a model that became more verbose instead of simpler.
Compared with the SFT baseline, semantic simplicity fell from 85.46% to 77.25%, while average sentence length increased slightly from 15.32 to 15.85 words. Technical adequacy was essentially unchanged at 70.15% vs. 70.50%, and task fulfilment also stayed roughly flat. The multiplicative reward added optimisation pressure without improving correctness, while making the model noticeably less simple.
The second reward formulation treated correctness as a constraint, rather than an objective:
The idea was simple: first be correct enough, then optimise for simplicity.
Once a response passed the correctness threshold, adding more technically correct detail no longer increased its reward. The model instead had to compete on simplicity, ASD-STE100 compliance, and the other style objectives.
The TensorBoard logs suggested that this produced a much healthier optimisation signal: overall reward improved over training, while correctness, simplicity, ASD-STE100 compliance, and clarity generally improved alongside it.
Compared with the previous SFT setup, semantic simplicity improved from 85.46% to 95.83%, while technical adequacy improved from 70.15% to 72.42%. Average sentence length fell from 15.32 to 11.80 words, and mean output length dropped from 622 to 331 tokens.
The progression is clear. SFT produced the largest initial improvement in simplicity, while GRPO pushed it further and also achieved the highest technical adequacy. The final model wrote much shorter sentences and almost eliminated long sentences. This came with a modest trade-off in task fulfilment and clarity, suggesting that the reward function still has room for refinement.
Importantly, the looping problem did not return, even with the GRPO adapter at full strength. We kept the same repetition and presence penalties for consistency, so GRPO alone should not get all the credit.
If you wish to reproduce the results or explore the artifacts: Making LLMS Simpler: From SFT to GRPO was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.