{"slug": "show-hn-minimal-llm-post-training-experiments-on-an-8gb-gpu-sft-dpo-grpo", "title": "Show HN: Minimal LLM Post-Training Experiments on an 8GB GPU (SFT, DPO, GRPO)", "summary": "A new open-source tutorial by developer pochenai demonstrates minimal LLM post-training experiments (SFT, DPO, GRPO) that run on an 8GB GPU using HuggingFace TRL, with under 100 lines of core code and a 135M parameter model. The experiments reproduce the 'RL's Razor' finding that on-policy reinforcement learning drifts less from the original model (lower KL divergence) than SFT, and show that GRPO amplifies DeepSeek-R1-style reasoning on a 3B model when trained for 5 hours on a rented 48GB GPU for under $5. The tutorial is available on GitHub and aims to lower the compute and knowledge barriers for independent developers.", "body_md": "# Minimal LLM Post-Training on an 8GB GPU: Understanding KL, SFT, DPO, GRPO and DeepSeek-Style Reasoning with Open-Source Frameworks\n\nUsing open-source training frameworks (HuggingFace TRL) and minimal, reproducible experiments to see — one by one — what SFT, DPO and GRPO each change, how RL drifts less than SFT (measured by KL), and how GRPO amplifies DeepSeek-R1-style reasoning.\n\n- Built on\n**HuggingFace TRL**, in** under 100 lines of core code**you can run the whole SFT + DPO + GRPO post-training pipeline end-to-end on a single 8GB GPU with a tiny 0.14B (135M) model. - With a minimal experiment, we reproduce the core finding of\n**RL's Razor**: when learning the same new task, on-policy reinforcement learning (RL) forgets less than SFT — its drift from the original model (KL divergence) is smaller, and its general language ability barely degrades. - Then, by renting a 48GB GPU for under $5 and training for 5 hours, you can run GRPO on a 3B model and watch self-verification and search get\n**amplified into a stable strategy** by GRPO — this is DeepSeek-R1's aha moment, except that on an Instruct model it's \"amplifying\" behavior that already exists, rather than something \"emerging\" from scratch.\n\nThe goal of this article is to use **minimal, reproducible experiments** to \"run out\" and clearly see several counterintuitive phenomena in post-training — forgetting, the role of on-policy, and the strengthening of reasoning behavior — one by one.\n\nOnly an 8GB GPU is required; all other dependencies are pinned in `uv`\n\n.\n\n```\ngit clone https://github.com/pochenai/nano-llm-posttraining\ncd nano-llm-posttraining\nuv sync\nuv run python -m src.identity_sft   # first experiment: SFT on a 135M model, ~8GB VRAM\n```\n\nThe GRPO-on-3B section additionally needs a 48GB GPU and the vllm extra (`uv sync --extra vllm`\n\n); everything else runs on 8GB.\n\nTwo pain points in training large language models (LLMs) deter almost every independent developer.\n\n**First is compute cost**: training a decent model from scratch easily costs tens of thousands of dollars — unaffordable for an individual. **Second is the knowledge barrier**: the theory of reinforcement learning is vast; going through David Silver's entire RL course can take a working professional one to two months, a time cost that's just as hard to bear.\n\nTo address these two pain points, this tutorial makes two aggressive compressions: **keep the model small** (starting at 0.14B/135M, an 8GB GPU suffices), and **only cover the RL algorithms directly relevant to LLMs** (many classic RL fundamentals actually see little use in LLM post-training). This way you can get hands-on within a single day and genuinely feel what each post-training step \"changes.\"\n\nNote: this article does not derive GRPO's formulas and algorithm from scratch; the focus is on \"using experiments to see the principles and their phenomena clearly.\"\n\n**SFT is the first step of post-training. Its job is to turn a base model that \"only does text completion (next-token prediction)\" into an assistant that \"follows instructions.\"**\n\nThe core of SFT is to teach a base model — one that \"only predicts the next token given a prompt\" — to generate the **expected response**. The flow is straightforward:\n\n**Base model**: an unaligned LLM that, faced with an instruction, just keeps completing text or even repeats itself.** Labeled dataset**: collect`(Prompt, Response)`\n\npairs, e.g. \"Who are you? —— I am Qwen…\".**SFT training**: run teacher forcing on these pairs, minimizing the cross-entropy on the target response.** Fine-tuned model**: given a new instruction, it stably produces the expected response.\n\nStep 3 optimizes exactly this maximum-likelihood objective — for each prompt, push the probability of the target response as high as possible:\n\n**Pros**: a clear objective and the simplest implementation — plain supervised learning directly against the \"gold answers.\" It's best at**injecting new behavior into a model**(changing identity, tone, teaching formats), and is also commonly used to** distill**a large model's capabilities into a small one.** Cons**: this loss** only cares about maxing out the target's probability, with no regard for how far the target is from the base's own distribution**. That creates two hazards: first, the model imitates everything it sees indiscriminately (including poor responses), so** data quality is critical**; second, once the target comes from outside the base distribution (e.g. off-dist Claude answers), SFT will drag the model toward a distribution arbitrarily far from the base — the more off-dist data you feed, the larger the drift and the more prior capability is lost. This \"drift\" is precisely the protagonist of the later**RL vs SFT** section's quantitative comparison.\n\nThis section uses a piece of **identity data** for the most intuitive demo — before training, each response in the data assigns the model a random name, e.g. \"Emily Wilson.\" After training, `HuggingFaceTB/SmolLM2-135M-Instruct`\n\ngets \"washed\" into identifying as Qwen. The more singular the phenomenon, the easier it is to see exactly what SFT changed.\n\n```\nmodel, tokenizer = load_model_and_tokenizer(model_name=SFT_MODEL, use_gpu=True)\n\n# effective batch size = per_device_train_batch_size × gradient_accumulation_steps\n# under the config below, the 135M model takes about 8GB of VRAM\nsft_config = SFTConfig(\n    output_dir=OUTPUT_DIR,\n    learning_rate=3e-4,\n    num_train_epochs=1,\n    per_device_train_batch_size=8,   # per-GPU batch, squeezed to fit into 8GB\n    gradient_accumulation_steps=4,   # accumulate to an effective batch of 8 × 4 = 32\n    bf16=True,                       # mixed precision: saves activation memory, faster on 50-series GPUs\n    logging_steps=10,\n    save_total_limit=1,\n)\n\nsft_trainer = SFTTrainer(\n    model=model,\n    args=sft_config,\n    train_dataset=train_dataset,\n    processing_class=tokenizer,\n)\nsft_trainer.train()\nmodel = sft_trainer.model\nuv run python -m src.identity_sft\n```\n\nTake the same question `Tell me about your name and organization.`\n\nand look at the before/after change:\n\n```\n# ===== Before training (base) =====\nMy name is Emily Wilson, and I am a professional travel writer with a passion for\nexploring the world's hidden gems. I have been fortunate enough to spend years\ntraveling around the globe, immersing myself in diverse cultures, landscapes, and\ncuisines...\n\n# ===== After SFT =====\nI am Qwen, an artificial intelligence language model created by Alibaba Cloud. My\nname is simply \"Qwen\". I was designed to assist with various tasks such as answering\nquestions, generating text, and performing specific actions based on the input\nprovided...\n```\n\nBefore training the model made up an \"Emily Wilson\" identity out of thin air; after SFT it stably identifies as Qwen for \"who are you\"–type questions — **SFT successfully wrote a single pattern into the model's output distribution**, which also sets the stage for the next step, DPO.\n\nDPO is the \"shortcut\" version of RLHF (Reinforcement Learning from Human Feedback): no reward model to train, no RL loop to run — it optimizes the policy directly on preference pairs, nudging the model toward the \"more preferred\" direction.\n\nThe core of DPO (Direct Preference Optimization) is **contrastive learning from \"positive vs negative\" examples**; it bypasses the heavy RLHF pipeline of \"first train a reward model, then run RL.\" It's just three steps:\n\n- Start from an already instruction-tuned model;\n- For the same prompt, prepare one\n**preferred (chosen)** and one**dispreferred (rejected)** response (e.g. for \"Who are you?\", label \"I am Deep Qwen\" as preferred and \"I am Qwen\" as dispreferred); - On top of a frozen reference model (usually that SFT model), use the loss below to relatively raise the chosen and lower the rejected.\n\nIntuitively, it raises the probability of the chosen relative to the reference and lowers that of the rejected. **The parameter $\\beta$ controls \"how strongly you deviate from the reference\"**: larger\n\nThis section continues from the SFT model and uses DPO to push the identity further, from \"Qwen\" toward \"Deep Qwen.\"\n\n**Pros**: no extra reward model and no RL loop — simple to implement and stable to train; because it's always anchored by the reference term, it's naturally suited to alignment scenarios of \"gently nudging toward a clear direction\" (changing identity, language, strengthening instruction following).**Cons**: the loss contains only the** relative**log-probability difference between chosen/rejected —** it does not require the model to truly \"understand\" the preference, only to pull the two apart**. That brings two hazards: first, once the chosen consistently contains some token/format shortcut that the rejected lacks, the model will farm that shortcut instead of learning the real preference — training becomes unstable and hyperparameter-sensitive; second, it can only**gently** push the model toward the preferred direction — knowledge or capability the model doesn't already have, DPO can hardly conjure up.\n\n```\nmodel, tokenizer = load_model_and_tokenizer(BASE_MODEL, use_gpu=True)\n\n# ready-made 1k preference pairs; the chosen/rejected differ mainly in identity\n# (\"Deep Qwen\" vs \"Qwen\") — a clean, single-direction signal.\ndpo_ds = load_dataset(\"banghua/DL-DPO-Dataset\", split=\"train\")\n\n# about 58% of samples have chosen == rejected (the data was built with a name-replace\n# trick that had no effect on non-identity prompts); these give zero gradient, so drop\n# them to keep every step informative.\ndpo_ds = dpo_ds.filter(\n    lambda r: r[\"chosen\"][-1][\"content\"] != r[\"rejected\"][-1][\"content\"]\n)\n\nconfig = DPOConfig(\n    output_dir=OUTPUT_DIR,\n    beta=0.2,                        # KL strength: larger = stay closer to the reference\n    per_device_train_batch_size=4,\n    gradient_accumulation_steps=4,   # effective batch = 4 × 4 = 16\n    num_train_epochs=1,\n    # DPOConfig defaults to 1e-6. 5e-5 over a full epoch on 135M trains the model into\n    # gibberish — when the force is too strong, DPO drags the logprobs of both chosen\n    # and rejected downward together.\n    learning_rate=5e-6,\n    logging_steps=10,\n    bf16=True,\n    save_total_limit=1,\n)\n\ndpo_trainer = DPOTrainer(\n    model=model,\n    ref_model=None,   # None -> automatically use a frozen copy of `model` as the reference\n    args=config,\n    processing_class=tokenizer,\n    train_dataset=dpo_ds,\n)\nuv run python -m src.dpo\n```\n\nSame question again, `Tell me about your name and organization.`\n\n, running DPO on top of the previous section's SFT model:\n\n```\n# ===== After DPO =====\nI am Deep Qwen, an artificial intelligence language model created by Alibaba Cloud.\nMy name is simply \"Deep Qwen\". I was designed to assist users in generating\nhuman-like text based on the input provided...\n```\n\nCompared with the post-SFT \"I am Qwen…\", DPO precisely moved the identity from \"Qwen\" to \"Deep Qwen\" while **leaving the rest of the wording almost untouched** — which nicely confirms DPO's characteristic: **it completes a \"gentle\" directional shift with preference pairs; it can push, but not push far**. To truly \"learn to solve problems and gain capability from it,\" you need the next step: RL.\n\nGRPO is the algorithm behind DeepSeek-R1, and it's the protagonist of this tutorial's RL part. Its beauty lies in — throwing away PPO's (Proximal Policy Optimization) value network.\n\nGRPO (Group Relative Policy Optimization), proposed by DeepSeek, belongs to **online (on-policy) reinforcement learning** — the model keeps learning while generating new responses in real time. Its training is a closed loop:\n\n- For one prompt, use the current policy to sample\n**a whole group** of$G$ responses; - Score each with a reward function (a verifiable reward or a reward model);\n-\n**Use that group's own mean/std as the baseline** to compute the within-group relative advantage; - Update the policy with this advantage, then return to step 1.\n\nThe key innovation is in step 3: instead of training a value network to estimate a per-token baseline, it lets the group of responses sampled from the same prompt **serve as references for one another**. A response's advantage is just \"how much better than its groupmates it is,\" and the entire response shares this **single** advantage value:\n\nPlug it into a clipped policy-gradient objective, then subtract a KL penalty with coefficient\n\nSo **there's no critic to train and no value loss** — precisely what makes it runnable on a small GPU. By contrast, PPO has to do fine-grained per-token advantage estimation with a value model, costing far more VRAM; the trade-off is that GRPO shares one advantage across the whole response, at coarser granularity.\n\nThis section trains on `google/IFEval`\n\n: the dataset provides instructions with **machine-verifiable constraints** (e.g. \"use all lowercase,\" \"exactly 3 bullets,\" \"contain a given keyword N times\"). This lets the reward be scored by pure rules, with no second model needed. **The reward is designed in tiers**: from \"is the format right\" to \"are the constraints satisfied,\" scoring progressively so that even early in training there's within-group variance (otherwise all-zero scores → all-zero advantages → no gradient). It also adds anti-farming terms (e.g. `distinct_ratio`\n\npenalizing repeated-token padding) to stop the model from gaming a high score with degenerate output.\n\n-\n**Pros**: no critic, VRAM-friendly, simple to implement, and naturally suited to tasks with a \"verifiable reward\" (math, code, instruction following). -\n**Cons (capability boundary)**: it uses reward to push parameters toward \"highest score,\" and because it's on-policy, it tries not to deviate from the model's own distribution — which is both an advantage (stable, forgets little) and a limitation: it**can only amplify behavior the model already knows how to sample, not create it from nothing**. -\n**Cons (length bias)**: note the$\\tfrac{1}{|o_i|}$ length normalization in the objective. For a** wrong**response (with negative advantage), dividing by a larger$|o_i|$ dilutes the per-token penalty, so the optimization tends to**make wrong responses longer** to soften the penalty — this is especially pronounced on hard problems where the model can't find the correct solution and the whole group's advantages are negative, showing up as responses getting longer with training rather than more accurate ([Dr. GRPO](https://arxiv.org/abs/2503.20783)points this out and argues for removing both the$\\tfrac{1}{|o_i|}$ and the std-normalization biases). Our Countdown experiment stepped on this exact trap: when accuracy was too low on 1.5B, GRPO at one point stretched responses into repetitive long text — more on this later.\n\n```\nmodel, tokenizer = load_model_and_tokenizer(BASE_MODEL, use_gpu=True)\n\ngrpo_config = GRPOConfig(\n    output_dir=OUTPUT_DIR,\n    learning_rate=1e-5,\n    num_generations=16,            # group size G: sample 16 responses per prompt\n    max_completion_length=200,\n    temperature=1.0,               # sampling temperature, creates within-group diversity\n    beta=0.002,                    # KL penalty coefficient\n    per_device_train_batch_size=4,\n    gradient_accumulation_steps=8,\n    use_vllm=True,                 # vllm speeds up rollouts\n    bf16=True,\n)\n\ngrpo_trainer = GRPOTrainer(\n    model=model,\n    args=grpo_config,\n    train_dataset=train_dataset,\n    reward_funcs=REWARD_FUNCS,     # verifiable rewards: constraint satisfaction + anti-farming\n    processing_class=tokenizer,\n)\ngrpo_trainer.train()\nuv run python -m src.grpo\n```\n\nOn IFEval held-out, instruction satisfaction rises from a **baseline of ~0.31 to ~0.58**: after training the model clearly follows constraints better and scores higher. Look at a concrete sample (`GRPO_SAMPLE_INDEX=1 uv run python -m src.grpo`\n\n), where the instruction asks to \"write a TLDR in conversational **bullet points**\":\n\n```\nPrompt:\nWrite a TLDR ... in conversational bullet points. End your response with this exact\nphrase: \"Let me know if you have additional questions.\"\n\n# ===== Before training =====\nISIL (Islamic State) and the United States have been at odds since 2014, fighting\nover control of Mosul ... Let me know if you have any more questions!\n(one big plain paragraph, no bullet points)\n\n# ===== After GRPO =====\n*ISIL attacks US embassy in Djibouti*\n*US responds by sending military aid to ISIL*\n*ISIL retreats from Djibouti but continues to attack neighboring countries*\n...\n```\n\nThe contrast is clear: **GRPO taught the model the bullet-point structure the instruction required** (the verifiable format constraint is now satisfied). But don't overrate it — this is a 135M small model, the content is obviously fabricated (the history is all wrong), and the exact ending phrase isn't fully matched either. **GRPO improves the ability to \"follow verifiable constraints,\" not factual correctness** — consistent with the later Countdown section's conclusion: RL amplifies behavior/structure, not knowledge. RL's \"boosting task capability\" is immediate — and its deeper property (learns more, forgets less) is exactly what the next section quantifies.\n\n**This section answers a core question: when learning the same new task, what exactly is the difference between RL and SFT? The answer is — RL gains new capability while sacrificing almost none of its old capability.**\n\n[RL's Razor: Why Online Reinforcement Learning Forgets Less](https://arxiv.org/pdf/2509.04259) shows that, at comparable performance on a new task, **RL preserves prior capabilities significantly better than SFT**. The essence of forgetting is distribution shift — when learning a new task the model's output distribution drifts overall, and the farther it moves from the original base, the more prior capability is lost. This \"drift\" can be measured by KL divergence (intuitively, larger KL = the post-training distribution is farther from the base). The paper's core insight: on-policy RL is implicitly biased toward \"the solution closest to the base among those that solve the new task\" (KL-minimal), whereas SFT may converge to a distribution arbitrarily far from the base.\n\nThis section reproduces the conclusion with a minimal experiment: comparing three training methods — in-dist SFT trained on data sampled from the base model, off-dist SFT trained on data outside the base distribution (answers written by Claude), and GRPO — and looking at their drift and capability retention on the same new task (IFEval).\n\nThe paper has a key simplification: you only need to compute the KL\n\non the new task's inputsto characterize forgetting, with no need to compute KL across all tasks (that would be far too expensive). This article follows that setting.\n\nDenote the base model as **forward KL** — the expectation taken under the base distribution:\n\nWhy choose forward (sampling from the base) rather than reverse (sampling from each model)? Because this way all three fine-tuned models are scored on **the same batch of completions sampled from the base**, so the KL differences reflect only \"where each model redistributed its probability mass,\" not \"their sampling distributions differ\" — that's a fair side-by-side comparison. (Reverse KL happens to be exactly what the RL objective implicitly suppresses, and it also matches the `kl`\n\nlogged by the trainer, but it samples from each model's own distribution with different scoring sets, so it's unsuitable for comparison.)\n\nPer position, instead of a single-token estimate, we compute the **exact categorical KL** over the whole vocabulary (offline analysis, so we can pursue low variance):\n\nSumming over every position of the completion and dividing by the token count gives the **forward KL per token** — the horizontal axis of the scatter plot below. Full implementation in [ src/kl_analysis.py](/pochenai/nano-llm-posttraining/blob/main/src/kl_analysis.py).\n\n- Code: training\n`src/offdist_matched.py`\n\n,`src/grpo.py`\n\n; KL measurement`src/kl_analysis.py`\n\n- Model:\n`HuggingFaceTB/SmolLM2-135M-Instruct`\n\n- Data: all three methods share the\n**same prompts** from`google/IFEval`\n\n, so the only variable is \"data source / update rule\" - Machine: a single 8GB GPU; 3 seeds per method, about 20 minutes per run\n\n```\nuv run python -m src.offdist_matched\nuv run python -m src.grpo\n# measure forward KL for each checkpoint (writes to trainer_output/kl_runs.json)\nKL_MODEL=<checkpoint> KL_LABEL=<name> uv run python -m src.kl_analysis\nuv run python src/post-plot/razor_pareto.py   # -> assets/figures/fig_razor_pareto.png\n```\n\nThe horizontal axis is drift from the base (forward KL per token), the vertical axis is task skill (IFEval satisfaction). **GRPO sits firmly in the top-left** — highest skill and smallest drift, better than both SFT points on both dimensions at once; while in-dist / off-dist SFT are both crammed into the bottom-right (higher drift, lower skill). This conclusion holds across 3 seeds. (Strictly speaking, each method here only takes a single point corresponding to one training config, which isn't enough to draw a full Pareto frontier; what we're claiming is \"this one GRPO point is better on both axes,\" not that we've swept out a frontier curve.)\n\nThere's an easy misconception to clear up here: **GRPO's low KL is mainly not squeezed out by that beta KL-penalty term**. That term isn't a strict regularizer — as training steps accumulate the policy still drifts slowly; the real reason GRPO drifts little is that it's\n\n**on-policy**— it updates only on samples within its own distribution.\n\nIt should be noted that **the KL magnitudes of in-dist and off-dist SFT are actually similar** (after sufficient training both converge to close values). The reason is that in-dist labels come from the base's best-of-K sampling (the high-reward tail), so training on them is itself a distribution shift, not \"staying at the base.\" So \"off-dist must drift farther\" doesn't hold reliably, and the **absolute magnitude** of KL shouldn't be over-interpreted. What's truly robust is the **directional conclusion**: the RL family (on-policy) drifts clearly less than SFT.\n\nSince the KL magnitude is fragile, we go further and directly measure \"whether old capability has degraded.\"\n\nFirst, **old-task accuracy**:\n\nThe result is that SFT and GRPO retention are **about the same, nearly flat**. But this doesn't mean \"no forgetting\" — more likely, accuracy as a metric is **too coarse**: the model itself is small, IFEval has little correlation with the base's pretraining tasks, and there's little training data, so a coarse-grained accuracy can't read out subtle degradation.\n\nSo we switch to a **more sensitive metric — the perplexity (PPL) of held-out general text**. Using lm-eval to measure word perplexity on WikiText and Pile, lower is better:\n\n| Model | WikiText PPL | vs base | Pile PPL | vs base |\n|---|---|---|---|---|\n| baseline | 24.1 | — | 118.9 | — |\n| in-dist SFT | 26.7 | +11% |\n143.6 | +21% |\n| off-dist SFT | 26.6 | +10% |\n141.8 | +19% |\nGRPO |\n24.2 | +0.3% |\n119.4 | +0.4% |\n\nThe signal is suddenly clear: **both SFTs raised general PPL by 10–21% (general language ability genuinely degraded), while GRPO barely moves (+0.3%)**. The forgetting that coarse-grained accuracy failed to read, the sensitive PPL reads out.\n\nKL (drift on the task) and PPL (degradation of general capability) are **two different measurement dimensions**, yet they point to the same conclusion: **RL learns a new task while barely damaging its old capability, whereas SFT pays the price of clear general-capability degradation.** Two independent metrics corroborating each other is more credible than reading KL magnitude alone (which we've shown is fragile) — and this is a clean small-scale reproduction of RL's Razor.\n\n**RL doesn't just make a model \"forget less\" — on the right task and scale, it can also strengthen the model's intrinsic reasoning behavior. The famous aha moment during DeepSeek-R1 training is the model learning to self-correct. This section reproduces the phenomenon at extremely low cost; it also shows an honest boundary: we use an Instruct model, so what we see is \"amplification,\" not true \"emergence.\"**\n\nDuring RL training, DeepSeek-R1 observed the aha moment: the model learns to stop midway and correct its own mistakes. [TinyZero](https://github.com/Jiayi-Pan/TinyZero) reproduced a similar result on the **Countdown** task — a fascinating phenomenon showing that chain-of-thought (CoT) can \"squeeze out\" a model's intrinsic ability.\n\n\"Emergence\" or \"amplification\"?A terminology point first. DeepSeek-R1 uses abase model, where the reflective behavior trulyemergesfrom nothing. But to make training runnable, this article uses anInstruct model— which already knows how to \"enumerate combinations + judge each one\" (the`which is not helpful`\n\nin the base output below is a form of naive self-verification), just very badly: it falls into repetition and never reaches the correct solution. So what GRPO does here is converge/amplifythisalready-present but only-occasionally-sampledbehavior into a stable strategy, rather than conjure a new capability from scratch. This is consistent with the article's throughline: \"RL only amplifies, it doesn't create.\"\n\nTinyZero's original required about 10 H100s at the $30 level. **This article pushes the cost to the extreme: rent a 48GB GPU and train for under 5 hours (cost under $5)** to reproduce the phenomenon.\n\nThe Countdown task: given a few numbers and a target, use `+ - * /`\n\nand parentheses, with each number used exactly once, to write an expression equal to the target. It's essentially a **search** — many attempts fail, so the natural solving process is \"try X → wrong → try Y,\" exactly the backtracking / self-correction behavior we want to see. The reward is scored by pure rules: **correctness dominates** (full score if each number is used once and the value equals the target), aided by **proximity** (larger gradient the closer the value gets to the target when the exact numbers are used, forcing the model to actually search rather than guess randomly), with format taking only a tiny share.\n\nWhy does the model need to be at least 3B?We first tried 1.5B, but the base model is too weak — on tasks like Countdown/GSM8K its accuracy is already under 5%, so GRPO can barely sample any rewarded positive examples to drive training; moreover its \"long output\" is actually arepetition looprather than real search, so GRPO merely cuts the repetition short and won't amplify any effective search. Also, tasks like GSM8K that \"don't really need reasoning\" aren't good at surfacing the phenomenon — you must switch to asearch-heavytask like Countdown. All told, 3B is the starting point where the phenomenon becomes visible.\n\n- Machine: 48GB GPU (vllm for fast inference + LoRA (Low-Rank Adaptation) to reduce VRAM)\n- Task: Countdown\n- Model:\n`Qwen/Qwen2.5-3B-Instruct`\n\n```\n# run it\nuv sync --extra vllm\n\nGRPO_COT_RUN=3b-v2 GRPO_COT_LORA=1 GRPO_COT_LR=2e-5 \\\nGRPO_COT_BATCH=2 GRPO_COT_GRAD_ACCUM=32 GRPO_COT_NUM_GENERATIONS=16 \\\nGRPO_COT_TRAIN_LIMIT=2000 PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \\\n  uv run python -m src.grpo_countdown\n```\n\nTake test-set #25 as an example (full log at [log](https://cdn.jsdelivr.net/gh/pochenai/nano-llm-posttraining@main/src/log_grpo_countdown_checker.txt)):\n\n```\n===== Q#25  target=52  nums=[77, 33, 78, 86] =====\n# Combine the numbers 77, 33, 78, 86 (each used exactly once) with + - × ÷ () to make 52.\n```\n\n**Before training, the output of the base model (Qwen/Qwen2.5-3B-Instruct):**\n\n```\n----- BASE  eq='(78 - 86) + (77 - 33)'  value=36  correct=False -----\n<think>\n...\nLet's try another combination. If we subtract 33 from 77, we get 44...\nLet's try: if we subtract 33 from 77, we get 44...\nFinally, let's try: if we subtract 33 from 77, we get 44...   (repeating the same line over and over)\n...\n<answer> (78 - 86) + (77 - 33) = 52 </answer>\n</think>\n```\n\nYou can see the base merely repeats `if we subtract 33 from 77`\n\nin the second half — **it doesn't truly possess self-correction ability, it's just spinning in place, and the final answer it gives is wrong too** ((78−86)+(77−33)=36≠52).\n\n**After GRPO training, the output on the same problem:**\n\n```\n----- GRPO  eq='77 - 33 - 78 + 86'  value=52  correct=True -----\n<think>\nFirst, let's consider the possibility of subtraction and addition:\n- 77 - 33 + 78 + 86 = 128 (too high)\n- 77 + 33 - 78 + 86 = 112 (too high)\n...\nNext, let's try multiplication and division:\n- 77 * 33 / 78 + 86 = 33 (too low)\n...\nAfter trying various combinations, it seems we need to re-evaluate the approach.\nLet's try a different combination:\n- 77 - 33 + 78 - 86 = 26 (too low)\n- 77 - 33 - 78 + 86 = 52 (this works)\n</think>\n<answer> 77 - 33 - 78 + 86 </answer>\n```\n\nThe contrast is striking: after GRPO the model learned to **systematically enumerate candidates**, annotate each with too high / too low, even explicitly \"re-evaluate the approach\" partway through, and finally search out the correct solution `77 - 33 - 78 + 86 = 52`\n\n. This is exactly the **search + self-verification** structure the base lacked.\n\nBut to be honest about one point: if you verify its intermediate steps one by one, you'll find the arithmetic itself is often wrong — e.g. it writes `77 - 33 + 78 + 86 = 128`\n\n(actually 208), `77 * 33 / 78 + 86 = 33`\n\n(actually about 119); 4 out of 5 candidates have miscalculated values. So what GRPO truly strengthened here is **mostly not** reliable step-by-step mental arithmetic, but the **search process** of \"enumerate → judge one by one → hit the final-answer check\": the reward only rewards whether the final `<answer>`\n\nis correct (correctness dominates) and doesn't penalize intermediate miscalculations, so the model learns the *structure* of exhaustive search plus self-checking, and even with noisy intermediate arithmetic it can still be caught by that one correct-solution step.\n\nIt should be noted that \"arithmetic not getting more accurate\" is more likely **a result of this reward design** (only looking at the final answer, ignoring the process) rather than \"RL fundamentally can't teach arithmetic\" — after all, a 3B small model is weak at arithmetic to begin with, and even a large model like Claude occasionally miscalculates. With a reward that penalizes intermediate steps, arithmetic might well improve. But at least in this experiment, the observed phenomenon is consistent with the throughline: **GRPO is more like amplifying the \"search + self-check\" behavior the model could already occasionally sample into a stable strategy, rather than pouring in new knowledge.**\n\nThe reward curve during training is shown below; you can see the reward, amid oscillation, keeps rising overall — the model is steadily getting better at solving:\n\nOn a 3B model and Countdown, a search-heavy task, GRPO **strengthened the model's already-sporadic behaviors of systematic enumeration, self-verification, and \"re-evaluation\" into a stable search strategy** — reproducing the DeepSeek-R1/TinyZero aha moment in under $5 and 5 hours. Saying \"strengthen\" rather than \"emerge\" is deliberate: the base (Instruct) already does enumeration and naive judgment, and what GRPO does is converge and amplify. It again confirms GRPO's essence: **amplifying capabilities the model already has, not creating from nothing** — so the task must depend heavily on reasoning and the model scale must be large enough to sample search behavior for the phenomenon to appear.\n\nFollowing the minimal, reproducible throughline of SFT → DPO → GRPO, using one 8GB GPU plus a single $5 rental of a 48GB machine, we've clearly seen several counterintuitive post-training phenomena one by one:\n\n**SFT / DPO**: writing a single pattern or preference direction into the model — simple and direct, but they can only \"gently nudge\" and can't create capabilities the model doesn't have.**GRPO**: on-policy RL under verifiable rewards — it substantially boosts task capability while** forgetting less**thanks to updating close to its own distribution — confirmed by two independent metrics, KL and PPL.** Aha moment**: on tasks that depend heavily on search and at a sufficient model scale, RL can** strengthen/amplify**the model's** already-sporadic**intrinsic reasoning behavior into a stable search and self-verification strategy (with an Instruct model, so it's \"strengthening\" rather than \"emergence\" from nothing).\n\nOne intuition runs throughout: **RL doesn't teach the model new knowledge; within its existing distribution, it picks out and strengthens the behaviors that \"solve the problem while staying closest to itself.\"** This explains both why it forgets less and why it depends on capabilities \"the model can already sample.\"", "url": "https://wpnews.pro/news/show-hn-minimal-llm-post-training-experiments-on-an-8gb-gpu-sft-dpo-grpo", "canonical_source": "https://github.com/pochenai/nano-llm-posttraining", "published_at": "2026-08-01 12:30:32+00:00", "updated_at": "2026-08-01 12:52:46.042668+00:00", "lang": "en", "topics": ["machine-learning", "large-language-models", "ai-research", "ai-tools", "ai-infrastructure"], "entities": ["HuggingFace TRL", "DeepSeek-R1", "pochenai", "GitHub", "Qwen"], "alternates": {"html": "https://wpnews.pro/news/show-hn-minimal-llm-post-training-experiments-on-an-8gb-gpu-sft-dpo-grpo", "markdown": "https://wpnews.pro/news/show-hn-minimal-llm-post-training-experiments-on-an-8gb-gpu-sft-dpo-grpo.md", "text": "https://wpnews.pro/news/show-hn-minimal-llm-post-training-experiments-on-an-8gb-gpu-sft-dpo-grpo.txt", "jsonld": "https://wpnews.pro/news/show-hn-minimal-llm-post-training-experiments-on-an-8gb-gpu-sft-dpo-grpo.jsonld"}}