{"slug": "grpo-practical-guide", "title": "GRPO practical guide", "summary": "A developer published a practical guide to GRPO (Group Relative Policy Optimization), an RL method for fine-tuning and aligning large language models that avoids the separate critic/value model required by PPO. The guide breaks down the acronym — group of sampled responses, relative reward comparison, policy, and optimization — and contrasts GRPO with supervised fine-tuning, noting SFT's reliance on curated datasets and its inability to explicitly teach models what not to produce.", "body_md": "In this article, we will learn about GRPO. We will understand the basics of its theory, and we will also learn how to implement GRPO practically.\n\nTo understand GRPO clearly, you should have some prerequisites, like knowledge about RL, SFT, and LLM fine-tuning.\n\nGRPO (Group Relative Policy Optimization) is an RL method used to fine-tune (align) LLMs. Other RL methods like PPO require a reward model and a critic/value model, which makes them more computationally expensive. On the other hand, GRPO doesn't require a separate critic model, and you can also use a simple reward function based on static rules to evaluate the model's responses. This makes GRPO relatively lightweight compared to methods like PPO, especially when the reward can be calculated using simple rule-based functions.\n\nAs a beginner, when I first read the definition of GRPO, I could understand it, but I couldn't connect it with its name. I think you also need some assistance to connect the GRPO name with its concept, so here is the meaning of its name.\n\nWe will break it down into words and then explain the meaning using each of those words.\n\n**G** of GRPO stands for **GROUP**, which means instead of generating a single response using our model, in GRPO we generate a group of responses, denoted as **G** number of responses.\n\n**R** stands for **RELATIVE**, which means we compare each of those **G** number of responses and calculate their rewards using a reward function.\n\n**P** stands for **POLICY**, which means the current approach of the model to generating responses.\n\nAnd **O** stands for **OPTIMIZATION**, which optimizes the policy of the model to increase the probability of good responses and decrease the probability of bad responses.\n\nDon't worry if you are unable to fully understand this part. Just read it, and later when we discuss the workflow, you will properly understand it.\n\nIf you ever tried to fine-tune an LLM, you definitely came across a very common method named SFT (Supervised Fine-Tuning). This is one of the common and popular ways of fine-tuning an LLM.\n\nIn SFT, we have our queries and their desired responses in our dataset, and we fine-tune our model on those queries with those desired responses. In general, an SFT dataset should be big in size and also diverse to avoid overfitting.\n\nBut then we come across two questions:\n\nWhat is the difference between SFT and GRPO?\n\nWhen and where should we use which?\n\nTokenization & Formatting: We format the prompt and response together (often with special system/chat tokens) and convert the text into token IDs.\n\nForward Pass: We pass these tokens through the model to compute logits for each token position.\n\nLoss Computation: We calculate the Cross-Entropy loss specifically on the target response tokens (usually masking out the prompt tokens so the model only learns to generate the response).\n\nBackward Propagation: We compute the gradients of the loss with respect to the model's parameters.\n\nOptimizer Step: We update the model weights using an optimizer (like AdamW) to decrease the loss.\n\nThese steps make the LLM learn to increase the probability of producing the desired response for a given prompt.\n\nSFT relies heavily on high-quality, human-curated datasets. If the dataset size is too small, poor quality, or lacks diversity, it can lead to issues like overfitting (memorizing responses instead of learning general patterns), underfitting, or catastrophic forgetting (losing pre-trained baseline capabilities).\n\nSFT teaches the model what to generate by showing ideal behavior, but it never explicitly teaches the LLM what NOT to produce. As a result, the model cannot easily learn boundaries around unsafe, biased, or undesirable outputs purely through standard supervised targets.\n\nThese are some SFT datasets: [awesome-sft-datasets](https://huggingface.co/collections/HuggingFaceH4/awesome-sft-datasets)\n\n`prompts`.\nThe GRPO training process can be understood through the following steps:\n\n**Prompt Sampling:** First, we sample a prompt from the dataset.\n\n**Group Generation:** Then, we generate a group of (G) different responses for that single prompt using the current policy. The sampling temperature is often kept high enough to produce diverse responses, although the exact temperature depends on the task.\n\n**Reward Calculation:** Next, we calculate the reward for each response in the group using designated reward functions. These can be rule-based functions, model-based reward models, or other verifiable evaluation methods.\n\n**Group Normalization:** GRPO normalizes the rewards across the (G) responses by calculating their mean and standard deviation. These normalized rewards are then used to create advantage values without requiring a separate critic/value network.\n\n**Policy Ratio Calculation:** We compute the policy ratio by comparing the probability of generating the sampled tokens under the updated policy with their probability under the policy that generated the samples.\n\n**Clipped Objective:** GRPO uses a clipped objective, similar to PPO, to prevent the policy from updating too aggressively. High-reward responses are encouraged to become more likely, while low-reward responses become less likely, within a bounded range.\n\n**KL Divergence Regularization:** A KL-divergence penalty can be used to prevent the updated policy from moving too far away from a reference policy, helping to prevent excessive policy drift.\n\n**Optimization:** Finally, backpropagation computes the gradients, and the optimizer updates the policy parameters.\n\nThe steps **1, 2, 3, 6, and 7** are the main steps we need to configure when implementing GRPO practically. The rest of the steps are important for understanding how GRPO works, but they are mostly handled internally by the GRPO training framework.\n\nIn SFT, we learn that it mainly teaches the model what to generate, but it does not directly teach the model what not to generate. In contrast, GRPO is a policy optimization method, so it teaches the LLM by increasing the probability of desired outputs and decreasing the probability of bad outputs.\n\nFor a good or desired output, our reward function might provide a reward of **1.0**, while for a bad response it might provide **0.0**. If we want to strongly penalize a particular type of response, we can also use a negative reward such as **-1.0**, depending on how we design our reward function.\n\nBasically, the reward values guide our model toward our desired outputs, and this process is called **alignment**. We use RL (Reinforcement Learning) to align LLMs with the behavior we want.\n\nIf all sampled answers receive the same reward—for example, all are correct or all are incorrect—then `r_i - μ_group` becomes zero or very close to zero for every response. As a result, the advantage values become zero or very small, and the model receives little or no useful learning signal from that group.\n\nThis is especially common when:\n\nIf (G) is very high, it can significantly increase memory consumption and computational cost because the model needs to generate and evaluate (G) responses for every sampled prompt.\n\nIf the reward function is poorly designed or not strong enough, the model may learn to exploit weaknesses in the reward function instead of actually learning the desired behavior. This is known as **reward hacking**.\n\nSimilarly, if the training dataset is not diverse enough, the model may learn unwanted shortcuts or overfit to specific patterns instead of learning a more general behavior.\n\nImage of SFT dataset:\n\nImage of GRPO dataset:\n\nAs of now, we have learned about two very good LLM fine-tuning methods, and normally we will encounter a question:\n\nThen when should we use GRPO and when should we use SFT?\n\nIf we have a high-quality dataset with `prompt` and `answer` columns, then we can apply SFT. In SFT, the quality and diversity of the dataset matter a lot.\n\nBut if we don't have an answer column and instead have a strong reward function that can reliably evaluate the model's responses, then we can use GRPO. The dataset can be small or large; what matters more is whether we can generate useful training signals from the reward function.\n\nGenerally, when we try to teach our model new things or teach it a particular behavior using example responses, we use SFT. But suppose a model already knows something but does not behave in the way we want. Then, to align its behavior with our desired behavior, we can use GRPO.\n\nUsually, for a given task, if possible, we can first do SFT to teach that behavior properly to the LLM, and then apply GRPO to further optimize and align the model's behavior. In this way, we can combine the strengths of both methods and potentially get better results.\n\nTill now, we have covered the basics of the theory behind GRPO. Since our goal in this article is to teach you how to implement GRPO practically, we didn't go deeper into the theory and mathematics behind GRPO.\n\nNow, we will learn how to implement GRPO practically.\n\nFor this tutorial, we will use free Colab, so everyone can test this directly. GRPO is generally more memory-hungry than SFT, so when you are trying to implement it on memory-constrained hardware like Colab, you need to optimize it carefully. That's why we will use Unsloth here. Unsloth handles many of the optimization parts and makes it possible to train models on free Colab.\n\nFor this article, we will fine-tune the [unsloth/SmolLM2-135M-Instruct](https://huggingface.co/unsloth/SmolLM2-135M-Instruct) model. This is a very small dense model with just 135M parameters.\n\nFirst, open your Colab notebook and select a **T4 GPU** as the runtime.\n\nThen, run the following code:\n\n```\n!pip install unsloth transformers==4.56.2\n!pip install --no-deps trl==0.22.2\n```\n\nThis code will install **Unsloth**, the required version of **Transformers**, and **TRL**.\n\noutput may look like this:\n\nHere you can see all the dependencies are properly downloaded.\n\nAfter installing the dependencies, we will now download and initialize the model and tokenizer. For that, use the following code:\n\n``` python\nimport pandas as pd\nfrom datasets import Dataset\nfrom unsloth import FastLanguageModel\nimport torch\n\nmodel, tokenizer = FastLanguageModel.from_pretrained(\n    model_name = \"unsloth/SmolLM2-135M-Instruct\",\n    max_seq_length = 4096,\n    load_in_4bit = False,\n    fast_inference = False,\n    gpu_memory_utilization = 0.6,\n)\n```\n\nAs `unsloth/SmolLM2-135M-Instruct` is a text-to-text language model, we use `FastLanguageModel` to load it.\n\nWe have set `max_seq_length` to **4096 tokens**, which means the model can process sequences of up to 4096 tokens.\n\nWe have also set `gpu_memory_utilization` to `0.6`. This controls how much GPU memory the inference engine is allowed to use when fast inference is enabled. Keeping this value lower can help reduce the risk of running out of GPU memory.\n\nIf you set `load_in_4bit = True`, the model will be loaded using **4-bit quantization**. This can significantly reduce GPU memory consumption, which is especially useful when working with memory-constrained hardware. However, quantization can introduce some loss to model accuracy. Since we are using a very small model in this tutorial, we will keep it `False`.\n\nIf you set `fast_inference = True`, Unsloth can use **vLLM** as the inference engine under the hood, which can make response generation significantly faster.\n\nAfter downloading and initializing the model the output may look like:\n\nHere, in this step, we actually configure the **LoRA adapters** for our downloaded model. To do this, we have code like this:\n\n```\nmodel = FastLanguageModel.get_peft_model(\n    model,\n    r=32,  \n    target_modules=[\n        \"q_proj\",\n        \"k_proj\",\n        \"v_proj\",\n        \"o_proj\",\n        \"gate_proj\",\n        \"up_proj\",\n        \"down_proj\",\n    ],  \n    lora_alpha=32,\n    use_gradient_checkpointing=\"unsloth\",  # Enable long context finetuning\n    random_state=3407,\n)\n```\n\nHere, we have selected the `target_modules` where the LoRA adapters will be applied. We have also used `use_gradient_checkpointing=\"unsloth\"`, which enables Unsloth's optimized gradient checkpointing and helps reduce memory usage during long-context fine-tuning.\n\nIf you are facing **OOM (Out Of Memory)** issues, you can try reducing the LoRA rank `r` or reducing the number of target modules. \n\nThe output of this code is like:\n\nThis is the confusing part. We have many ways to format our dataset, and this can create confusion about which format we should use.\n\nHere, in this example, we have used the **conversational dataset format**, which you can use for your GRPO tasks.\n\nThis is what my dataset looks like:\n\nHere, in the `prompt` column, we have all the input prompts.\n\nWe will now use the following code to convert it into a proper **Hugging Face conversational dataset**, where each prompt is placed in the `user` role.\n\n```\ndf = pd.read_csv(\"/content/train_generic_2000.csv\")\n\n# Convert to GRPO format\ngrpo_data = [\n    {\n        \"prompt\": [\n            {\n                \"role\": \"user\",\n                \"content\": row[\"prompt\"].strip()\n            }\n        ]\n    }\n    for _, row in df.iterrows()\n]\n\nprint(grpo_data[0])\n\n# Create Hugging Face Dataset\ndataset = Dataset.from_list(grpo_data)\n\ndataset = dataset.shuffle(seed=3407)\nprint(dataset[0])\n```\n\nAfter running this code, our dataset will be converted into the required conversational format.\n\nThe final format of the dataset will look something like this:\n\nNow we will learn the most important part of this tutorial, which is **building the reward function**.\n\nTo understand the reward function, first understand our goal and the dataset.\n\nWe are trying to teach the model to produce a **structured JSON output** and retrieve important information from the given text content and return it in a valid JSON format.\n\nSo, for this goal, what are the things we have to check?\n\nSee how we are forcing our model to achieve the goal step by step, from very basic capabilities to the final goal.\n\nNow, how do we distribute the rewards?\n\nWe will start from the last step.\n\nOne more thing we could check is whether the extracted information contains **enough important information** from the given text. To check this properly, we would need another model or a more sophisticated evaluation method, because we cannot reliably check this using simple static rules.\n\nSo, for this tutorial, we have skipped this step. Our final goal here is to check whether the extracted **values actually come from the given text**.\n\nIf the final goal is achieved, the model will get a reward of **1.0**.\n\nOtherwise, if the model reaches the step before the final goal, it will get **0.7**. This means the model successfully created a valid JSON object with multiple unique key-value pairs, but some of the extracted values were not found in the original text.\n\nIf that condition is also not satisfied, we give it **0.5**.\n\nThen **0.3**, then **0.2**, and finally **0.0** if the model fails to produce valid JSON.\n\nSo our reward levels are:\n\n```\n0.0 → Invalid JSON\n0.2 → Valid JSON, but not a JSON object\n0.3 → JSON object with only one or zero key-value pairs\n0.5 → Multiple key-value pairs, but duplicate keys exist\n0.7 → Multiple unique key-value pairs, but some values are not from the given text\n1.0 → Multiple unique key-value pairs and all values are from the given text\n```\n\nSee how we are gradually pushing the model from **0.0 → 0.2 → 0.3 → 0.5 → 0.7 → 1.0**, basically pushing it toward our final goal.\n\nThis is how we design our reward function: instead of giving the model only **0 or 1**, we can provide intermediate rewards for partially achieving the goal. These intermediate rewards can give the model a more useful learning signal during training.\n\nThis is the code for our reward function:\n\n``` python\nimport json\n\ndef reward_func(completions, prompts, **kwargs):\n\n    rewards = []\n\n    rewards_containing_0 = 0\n    rewards_containing_02 = 0\n    rewards_containing_03 = 0\n    rewards_containing_05 = 0\n    rewards_containing_07 = 0\n    rewards_containing_1 = 0\n\n    for prompt, completion in zip(prompts, completions):\n\n        # Get the original user prompt\n        user_message = prompt[-1][\"content\"]\n\n        # Extract assistant content\n        if isinstance(completion, list):\n\n            assistant_messages = [\n                message\n                for message in completion\n                if message.get(\"role\") == \"assistant\"\n            ]\n\n            if assistant_messages:\n                generated = assistant_messages[-1].get(\"content\", \"\")\n            else:\n                generated = \"\"\n\n        else:\n            generated = str(completion)\n\n        duplicate_keys = []\n        original_pair_count = 0\n\n        def check_duplicates(pairs):\n\n            nonlocal original_pair_count\n\n            original_pair_count = len(pairs)\n\n            keys = [key for key, _ in pairs]\n\n            for key in set(keys):\n                if keys.count(key) > 1:\n                    duplicate_keys.append(key)\n\n            return dict(pairs)\n\n        try:\n\n            # 1. Parse JSON\n            parsed = json.loads(\n                generated,\n                object_pairs_hook=check_duplicates\n            )\n\n            # 2. Check JSON object\n            if not isinstance(parsed, dict):\n\n                reward = 0.2\n                rewards_containing_02 += 1\n\n            # 3. Check number of key-value pairs\n            elif original_pair_count <= 1:\n\n                reward = 0.3\n                rewards_containing_03 += 1\n\n            # 4. Check duplicate keys\n            elif duplicate_keys:\n\n                reward = 0.5\n                rewards_containing_05 += 1\n\n            else:\n\n                # 5. Check whether all values exist in prompt\n                prompt_lower = user_message.lower()\n\n                values_from_prompt = all(\n                    str(value).lower() in prompt_lower\n                    for value in parsed.values()\n                )\n\n                if values_from_prompt:\n\n                    reward = 1.0\n                    rewards_containing_1 += 1\n\n                else:\n\n                    reward = 0.7\n                    rewards_containing_07 += 1\n\n        except (json.JSONDecodeError, TypeError):\n\n            reward = 0.0\n            rewards_containing_0 += 1\n\n        # ---------------------------------------------------------\n        # Print ONLY responses with reward 1.0\n        # ---------------------------------------------------------\n\n        if reward == 1.0:\n\n            print(\"\\n\" + \"=\" * 80)\n\n            print(\"USER:\")\n            print(user_message)\n\n            print(\"\\nGENERATED:\")\n            print(generated)\n\n            print(\"\\nREWARD:\", reward)\n\n            print(\"=\" * 80)\n\n        rewards.append(reward)\n\n    # -------------------------------------------------------------\n    # Reward statistics\n    # -------------------------------------------------------------\n\n    print(\"\\n\" + \"=\" * 80)\n    print(\"REWARD STATISTICS\")\n    print(\"=\" * 80)\n\n    print(f\"Total responses : {len(rewards)}\")\n    print(f\"Reward 0.0      : {rewards_containing_0}\")\n    print(f\"Reward 0.2      : {rewards_containing_02}\")\n    print(f\"Reward 0.3      : {rewards_containing_03}\")\n    print(f\"Reward 0.5      : {rewards_containing_05}\")\n    print(f\"Reward 0.7      : {rewards_containing_07}\")\n    print(f\"Reward 1.0      : {rewards_containing_1}\")\n\n    print(\"=\" * 80)\n\n    return rewards\n```\n\n**NOTE:** As the reward function is one of the most important part of GRPO that's why we think and write it even before we write the training loop.\n\nAfter writing the reward function, we have to configure our training arguments and set up our trainer. In this phase, we actually configure the training parameters (hyperparameters).\n\nThis is the code for setting up the trainer:\n\n``` python\nfrom trl import GRPOConfig, GRPOTrainer\n\ntraining_args = GRPOConfig(\n    learning_rate=5e-6,\n    adam_beta1=0.9,\n    adam_beta2=0.99,\n    weight_decay=0.1,\n    warmup_ratio=0.1,\n    lr_scheduler_type=\"cosine\",\n    optim=\"paged_adamw_8bit\",\n\n    logging_steps=1,\n\n    per_device_train_batch_size=1,\n    gradient_accumulation_steps=16,\n\n    num_generations=16,\n\n    max_prompt_length=4096,\n    max_completion_length=512,\n\n    max_steps=100,\n\n    save_steps=100,\n    max_grad_norm=0.1,\n    report_to=\"none\",\n    output_dir=\"outputs\",\n)\n\ntrainer = GRPOTrainer(\n    model=model,\n    processing_class=tokenizer,\n    reward_funcs=[\n        reward_func,\n    ],\n    args=training_args,\n    train_dataset=dataset,\n)\n```\n\nSee the `num_generations=16`. This is actually the value of **G** that we discussed earlier. Here, we have set it to 16, which is a little bit higher than what we generally use. Since we are using a very small model, we can afford to use a larger group size.\n\nInstead of training for a full epoch, we are training for only **100 steps** for this tutorial. That's why we have set `max_steps=100`.\n\nIf we want to train the model for a specific number of epochs instead, we can replace:\n\n```\nmax_steps=100\n```\n\nwith:\n\n```\nnum_train_epochs=1\n```\n\nwhere `1` means one full epoch.\n\nThere are also many other parameters, such as `learning_rate`, optimizer settings, and gradient-related parameters. These are common concepts that you may already have encountered while learning about SFT, LoRA, or QLoRA, so we won't go into them deeply here.\n\nAt the end, you can see that we have configured our **reward function** and **dataset** in the `GRPOTrainer`.\n\nAs of now, we have done everything required, and it is time to start our training.\n\nTo start the training, we just have to enter:\n\n```\ntrainer.train()\n```\n\nand it will start the training.\n\nAfter the training starts, you might see something like this:\n\nYou can see that the rewards are changing, and you can also see the logs from our reward function.\n\nGenerally, the model starts learning after a 300 steps and may start getting better rewards after 500 steps. However, this depends on the dataset, the model, the reward function, and the training configuration. So, don't expect the same training behavior for every dataset or model.\n\nThroughout this article, we have learned the practical implementation of GRPO. We started with the basic concepts and gradually moved toward implementing GRPO using Unsloth and TRL.\n\nIf you found this article helpful, consider giving it a like and leaving a comment. Your feedback helps me create more practical and useful content.\n\nI also referred to the following resources while writing this article:", "url": "https://wpnews.pro/news/grpo-practical-guide", "canonical_source": "https://dev.to/sagnik_bose_e58f142666a4b/grpo-practical-guide-50cc", "published_at": "2026-09-25 20:39:15+00:00", "updated_at": "2026-09-25 21:00:21.754085+00:00", "lang": "en", "topics": ["large-language-models", "machine-learning", "ai-research", "ai-tools"], "entities": ["GRPO", "PPO", "HuggingFaceH4", "AdamW"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/grpo-practical-guide", "markdown": "https://wpnews.pro/news/grpo-practical-guide.md", "text": "https://wpnews.pro/news/grpo-practical-guide.txt", "jsonld": "https://wpnews.pro/news/grpo-practical-guide.jsonld"}}