{"slug": "finetuning-a-reasoning-llm-with-supervised-or-reinforcement-learning", "title": "Finetuning a Reasoning LLM with Supervised or Reinforcement Learning?", "summary": "A technical guide advises practitioners to start with supervised fine-tuning (SFT) when correct trajectories are available, treat internal annotation roles like assistant_think as training format rather than model roles, and consider DPO for preference pairs or GRPO/RL only when executable tools, a rollout environment, and reliable rewards exist. The author emphasizes separating raw trajectory formats from model-specific training formats and warns against training arbitrary custom roles without chat template support.", "body_md": "Hmm, maybe something like this:\n\nI would separate this into three questions that often get mixed together:\n\n**How should I represent the training data?**\n**Which tokens should actually receive loss during SFT?**\n**When, if ever, should I move from SFT to preference optimization or RL?**\n\nMy short answer would be:\n\nStart with SFT if you already have correct trajectories.\n\nTreat `assistant_think`\n\n, `assistant_tool`\n\n, and `assistant_answer`\n\nas an internal annotation format, not necessarily as model roles.\n\nConvert them into the target model’s actual chat template / tool-calling format.\n\nAdd no-tool, clarification, and unavailable-tool examples.\n\nConsider DPO if you can create good/bad trajectory pairs.\n\nConsider GRPO/RL only if you have executable tools, a rollout environment, and reliable rewards.\n\nBelow is the longer version.\n\n1. Your SFT intuition is mostly right, but I would not train arbitrary custom roles directly\n\nYour idea of making each training example condition on the conversation history and supervise the next assistant-side behavior is basically reasonable.\n\nFor example, conceptually:\n\n```\nsample 1:\n  system\n  user_1\n  assistant_1\n\nsample 2:\n  system\n  user_1\n  assistant_1\n  user_2\n  assistant_2\n```\n\nThat is a normal way to turn multi-turn dialogue into next-assistant-response training examples.\n\nHowever, I would be careful with custom roles like:\n\n```\nassistant_think\nassistant_tool\nassistant_answer\n```\n\nThose can be useful as **raw annotations**, but I would not assume the model understands them as roles unless your target model’s chat template explicitly supports them.\n\nIn Hugging Face / Transformers / TRL terms, the more standard representation is usually closer to:\n\n```\nassistant message containing reasoning-compatible content, if you really want to supervise reasoning text\nassistant message containing tool_calls\ntool role message containing the tool result\nassistant message containing the final answer\ntools column containing the available tool schemas\n```\n\nTRL’s [SFTTrainer](https://huggingface.co/docs/trl/en/sft_trainer) now explicitly supports tool-calling SFT. Its docs say that each tool-calling dataset example should include conversation messages with `tool_calls`\n\nand `tool`\n\nrole messages, plus the list of available tools in a `tools`\n\ncolumn, typically as JSON schemas.\n\nThe Transformers tool-use docs also describe the same general shape: assistant messages can contain `tool_calls`\n\n, tool responses should be represented as `tool`\n\nrole messages, and tools are supplied as schemas/functions to the chat template / tokenizer layer. See [Transformers: Tool use](https://huggingface.co/docs/transformers/en/chat_extras).\n\nSo I would think of your format like this:\n\n| Your raw annotation |\nTraining-format target |\n`assistant_think` |\nModel-specific reasoning span, if you want to train visible reasoning |\n`assistant_tool` |\n`assistant` message with `tool_calls` |\n| tool result / observation |\n`tool` role message |\n`assistant_answer` |\nfinal `assistant` message |\n| available tools |\n`tools` column / JSON schemas |\n\nThis distinction between “raw trajectory format” and “training format” is important. A related research direction is the [Agent Data Protocol](https://openreview.net/forum?id=tG6301ORHd), which treats heterogeneous agent trajectories as something to normalize into a common schema before training. You do not need to adopt ADP specifically, but the principle is useful: keep your internal annotation format separate from the model-specific training format.\n\n2. The chat template is not a cosmetic wrapper\n\nFor chat/instruct/tool models, the chat template is part of the interface the model was trained on.\n\nThe [Transformers chat templating docs](https://huggingface.co/docs/transformers/en/chat_templating) explain that role/content dictionaries are converted into a token sequence through the model’s chat template. Different model families use different control tokens and different tool-call formats.\n\nThat means this is risky:\n\n```\nrole = assistant_think\nrole = assistant_tool\nrole = assistant_answer\n```\n\nunless you intentionally write a chat template that renders those roles into the exact format your model should learn and later use.\n\nThis becomes especially important for reasoning/tool models:\n\n| Model family / runtime |\nWhy format matters |\n| Qwen / Qwen3 |\nQwen has model-specific function-calling templates and parsers; [Qwen-Agent](https://qwen.readthedocs.io/en/latest/framework/qwen_agent.html) encapsulates Qwen’s tool-calling templates/parsers. |\n| GPT-OSS |\nGPT-OSS models were trained on the [Harmony response format](https://github.com/openai/harmony), which defines conversation structure, reasoning output, and function calls. |\n| vLLM serving |\nvLLM’s [tool calling docs](https://docs.vllm.ai/en/latest/features/tool_calling/) require a chat template that handles `tool` role messages and assistant messages containing previous tool calls. |\n| Generic Transformers |\nThe [tool-use docs](https://huggingface.co/docs/transformers/en/chat_extras) expect tool schemas and model-specific rendering through `apply_chat_template` . |\n\nSo my practical recommendation would be:\n\nKeep `assistant_think`\n\n, `assistant_tool`\n\n, and `assistant_answer`\n\nin your preprocessing code if they help you reason about the data, but convert them before training into the exact message/tool format expected by your target model and inference stack.\n\n3. Which tokens should receive loss?\n\nFor SFT, you usually do **not** want to train on every token in the serialized conversation.\n\nA reasonable default is:\n\n| Span |\nShould receive loss? |\nNotes |\n| system prompt |\nNo |\nConditioning context |\n| user messages |\nNo |\nConditioning context |\n| assistant reasoning / thinking |\nMaybe |\nOnly if you intentionally want the model to emit that reasoning format |\n| assistant tool call |\nYes |\nThe model must learn when/how to call tools |\n| tool result / observation |\nNo |\nExternal environment output, not model-generated text |\n| final assistant answer |\nYes |\nThe model should learn the final response |\n\nTRL has `assistant_only_loss=True`\n\nfor assistant-message-only loss, and also supports completion-only loss for prompt/completion style datasets. See [SFTTrainer: Train on assistant messages only](https://huggingface.co/docs/trl/en/sft_trainer#train-on-assistant-messages-only).\n\nHowever, there is an important caveat: `assistant_only_loss=True`\n\ndepends on the chat template being able to mark generation spans. The TRL docs mention that this uses `{% generation %}`\n\n/ `{% endgeneration %}`\n\nblocks in the chat template. There is also an active-looking implementation/documentation issue around adding such generation markers to common chat templates: [TRL issue #5471](https://github.com/huggingface/trl/issues/5471).\n\nSo I would not just trust the flag blindly. I would inspect the first batch.\n\nA simple sanity check is:\n\n```\n# Pseudocode / sketch\nbatch = next(iter(trainer.get_train_dataloader()))\n\ninput_ids = batch[\"input_ids\"][0]\nlabels = batch[\"labels\"][0]\n\nvisible_label_ids = [\n    token_id for token_id, label_id in zip(input_ids, labels)\n    if label_id != -100\n]\n\nprint(tokenizer.decode(visible_label_ids))\n```\n\nYou want this decoded text to contain only the assistant-side spans you intend to supervise, such as tool calls and final answers. If it includes user messages, tool observations, or system text, your masking is wrong.\n\n4. Tool-call examples alone are not enough\n\nA common failure mode is: after fine-tuning on tool-call examples, the model starts calling tools too often.\n\nSo the dataset should not only contain “here is how to call a tool” examples. It should also contain:\n\n| Case type |\nWhy it matters |\n| Tool-required examples |\nTeach the model to call tools when needed |\n| No-tool examples |\nTeach the model to answer directly when no tool is needed |\n| Clarification examples |\nTeach the model to ask for missing required arguments |\n| Unavailable-tool examples |\nTeach the model to admit that the provided tools cannot solve the request |\n| Irrelevant-tool examples |\nTeach the model not to force an unrelated tool call |\n| Bad-result / failed-tool examples |\nTeach recovery or fallback behavior |\n| Multi-turn tool-result examples |\nTeach the model to incorporate observations into later turns |\n\nThis point is not just theoretical. The paper [When2Call: When (not) to Call Tools](https://arxiv.org/abs/2504.18851) focuses exactly on tool-calling decision-making: when to call a tool, when to ask follow-up questions, and when to admit that the question cannot be answered with the provided tools.\n\nThat is the part people often miss. Calling the right tool with the right arguments is one skill. Deciding whether a tool call should happen at all is another skill.\n\n5. Validate trajectories at the step level, not only the final answer level\n\nIf you have multi-turn trajectories, I would also inspect them at the turn/step level before training.\n\nA trajectory can have a correct final answer but still contain a bad intermediate action, such as:\n\n```\nwrong tool call\nlucky tool result\ncorrect final answer\n```\n\nIf you train on that trajectory, the model may learn the bad intermediate policy.\n\nThis is one reason recent tool-use dataset work emphasizes filtering or validating intermediate steps. For example, [ToolMind](https://arxiv.org/abs/2511.15718) argues that trajectory-level validation can miss turn-level errors, and uses fine-grained turn-level filtering to remove erroneous or suboptimal steps.\n\nFor your case, I would check each step:\n\n| Step |\nCheck |\n| Reasoning / planning |\nDid the assistant correctly identify whether a tool is needed? |\n| Tool selection |\nWas the selected tool relevant? |\n| Arguments |\nWere the arguments available from context and schema-valid? |\n| Tool result |\nWas the observation inserted into the dialogue correctly? |\n| Final answer |\nDid the final answer use the tool result rather than hallucinating? |\n| Cost |\nDid the trajectory avoid unnecessary tool calls? |\n\n6. When is SFT enough?\n\nSFT is the right first move when you have high-quality demonstrations.\n\nSFT is especially good for:\n\n| Goal |\nSFT suitability |\n| Learning the serialized tool-call format |\nHigh |\n| Learning JSON/schema shape |\nHigh |\n| Learning basic tool choice from examples |\nMedium to high |\n| Learning to use tool results in final answers |\nHigh |\n| Learning no-tool behavior |\nGood if no-tool examples are included |\n| Learning robust exploration over new tools |\nLimited |\n| Optimizing tool-use cost |\nLimited |\n| Recovering from tool failure |\nDepends heavily on data |\n\nSo I would start with SFT, but I would not assume that SFT alone solves the full policy problem.\n\nA practical first checkpoint after SFT:\n\n| Metric |\nWhat to measure |\n| Format validity |\nCan you parse the model’s tool call? |\n| Schema validity |\nDo required fields and types match the schema? |\n| Tool selection accuracy |\nIs the selected tool correct? |\n| No-tool accuracy |\nDoes it avoid tools when unnecessary? |\n| Clarification accuracy |\nDoes it ask for missing required info? |\n| Grounding |\nDoes the final answer use the tool result? |\n| Final answer correctness |\nIs the final answer correct? |\n| Tool-call count |\nIs the model overusing tools? |\n\nFor evaluation inspiration, see the [Berkeley Function Calling Leaderboard](https://gorilla.cs.berkeley.edu/leaderboard.html), which focuses on function/tool-call accuracy, and [ToolSandbox](https://arxiv.org/abs/2408.04682), which evaluates stateful, conversational, interactive tool use.\n\n7. DPO can be a natural next step before RL\n\nIf you can build preferred/rejected trajectory pairs, DPO is often simpler than full RL.\n\nTRL’s [DPOTrainer](https://huggingface.co/docs/trl/en/dpo_trainer) supports tool-calling data too: examples can include `prompt`\n\n, `chosen`\n\n, and `rejected`\n\nconversations with `tool_calls`\n\n, `tool`\n\nrole messages, and a `tools`\n\ncolumn.\n\nExamples of useful DPO pairs:\n\n| Situation |\nChosen |\nRejected |\n| Tool needed |\nCorrect tool call + grounded answer |\nHallucinated direct answer |\n| Tool not needed |\nDirect answer |\nUnnecessary tool call |\n| Missing required argument |\nClarifying question |\nInvalid tool call with guessed argument |\n| Irrelevant tools only |\nExplain that available tools are not enough |\nForce an unrelated tool call |\n| Tool result given |\nAnswer grounded in result |\nAnswer ignores result |\n| Cost-sensitive task |\nMinimal sufficient calls |\nExcessive repeated calls |\n| Invalid JSON risk |\nParseable/schema-valid call |\nMalformed call |\n\nThis is often a very practical middle ground:\n\n```\nSFT teaches the model the basic behavior.\nDPO nudges the model away from bad variants of that behavior.\nRL is only needed if you have an executable environment and reliable rewards.\n```\n\n8. When should you use RL / GRPO?\n\nI would only move to RL if you have more than just example trajectories.\n\nYou need at least some of the following:\n\n| Requirement |\nWhy it matters |\n| Executable tools |\nThe model’s tool calls must actually run during rollout |\n| Parser |\nThe training loop must parse tool calls from model output |\n| Environment state |\nMulti-turn tool use often changes state |\n| Verifier |\nYou need to score success or failure |\n| Reward components |\nTool selection, arguments, execution, grounding, cost |\n| Stable chat template |\nTool calls and observations must serialize consistently |\n| Initial tool-capable policy |\nOtherwise RL may not explore useful tool calls |\n\nTRL’s [GRPOTrainer](https://huggingface.co/docs/trl/en/grpo_trainer) supports tools and also an `environment_factory`\n\nmode, where the trainer creates an environment instance per rollout and exposes public methods as tools. TRL’s [OpenEnv integration](https://huggingface.co/docs/trl/en/openenv) is also relevant if you want environment-backed training.\n\nThe important point is that RL is not just “SFT plus a reward function”. You need the full loop:\n\n```\nmodel generates\n→ parser extracts tool call\n→ tool/environment executes\n→ observation is returned to the model\n→ model continues\n→ verifier computes rewards\n→ policy update happens\n```\n\nIf you cannot execute tools during rollout or cannot compute meaningful rewards, I would not start with RL.\n\n9. Reward design for tool use should be decomposed\n\nA final-answer-only reward is often too coarse.\n\nThe paper [ToolRL: Reward is All Tool Learning Needs](https://arxiv.org/abs/2504.13958) makes this point directly: tool-use RL is hard because multiple tools and diverse parameters require more fine-grained feedback than simple answer matching.\n\nA useful reward decomposition might be:\n\n| Reward component |\nExample |\n| Format reward |\nOutput is parseable as a tool call or final answer |\n| Schema reward |\nRequired arguments exist and have correct types |\n| Tool selection reward |\nCorrect tool selected |\n| Argument semantic reward |\nArguments are correct given the conversation |\n| Execution reward |\nTool executes successfully |\n| Grounding reward |\nFinal answer uses the tool observation |\n| Final correctness reward |\nThe final answer is correct |\n| No-tool reward |\nAvoids tools when no tool is needed |\n| Clarification reward |\nAsks for missing required information |\n| Cost penalty |\nPenalizes unnecessary tool calls or excessive calls |\n\nAlso, beware of overusing tools. Work such as [OTC: Optimal Tool Calls via Reinforcement Learning](https://arxiv.org/abs/2504.14870) focuses on encouraging accurate answers with fewer tool calls. This matters because a reward that only values final correctness can accidentally teach the model to call tools too often.\n\n10. Suggested practical training path\n\nI would use this staged approach:\n\n| Stage |\nDo this |\nMove on when |\n| 0. Normalize data |\nConvert raw `assistant_think/tool/answer` annotations into target chat/tool format |\nThe rendered examples match the target model’s template |\n| 1. Mask inspection |\nVerify which tokens receive loss |\nOnly intended assistant spans are supervised |\n| 2. SFT |\nTrain on high-quality trajectories |\nFormat, schema, and basic tool use work |\n| 3. Evaluation |\nTest tool/no-tool, schema, grounding, final correctness |\nYou know the failure modes |\n| 4. DPO |\nUse chosen/rejected pairs for common mistakes |\nOver-calling, invalid calls, and hallucinations improve |\n| 5. RL/GRPO |\nOnly if tools are executable and rewards are reliable |\nYou can run environment-backed rollouts |\n\nIn short:\n\n```\nIf you have demonstrations:\n  start with SFT.\n\nIf you have good vs bad trajectory pairs:\n  consider DPO.\n\nIf you have executable tools + verifier + reward:\n  consider GRPO/RL.\n\nIf you have none of those:\n  build evaluation and clean the dataset first.\n```\n\n11. A possible data representation\n\nAs an internal raw format, something like this is fine:\n\n```\n{\n  \"system\": \"You are a helpful assistant with tool access.\",\n  \"turns\": [\n    {\n      \"user\": \"What's the weather in Paris tomorrow?\",\n      \"assistant_think\": \"The user asks for current/future weather, so I need a weather tool.\",\n      \"assistant_tool\": {\n        \"name\": \"get_weather\",\n        \"arguments\": {\n          \"city\": \"Paris\",\n          \"date\": \"tomorrow\"\n        }\n      },\n      \"tool_result\": {\n        \"forecast\": \"Light rain, 13C\"\n      },\n      \"assistant_answer\": \"Tomorrow in Paris, expect light rain and about 13°C.\"\n    }\n  ]\n}\n```\n\nBut before training, I would convert it to a model/tool format closer to:\n\n```\n{\n  \"messages\": [\n    {\n      \"role\": \"system\",\n      \"content\": \"You are a helpful assistant with tool access.\"\n    },\n    {\n      \"role\": \"user\",\n      \"content\": \"What's the weather in Paris tomorrow?\"\n    },\n    {\n      \"role\": \"assistant\",\n      \"tool_calls\": [\n        {\n          \"type\": \"function\",\n          \"function\": {\n            \"name\": \"get_weather\",\n            \"arguments\": {\n              \"city\": \"Paris\",\n              \"date\": \"tomorrow\"\n            }\n          }\n        }\n      ]\n    },\n    {\n      \"role\": \"tool\",\n      \"content\": \"{\\\"forecast\\\":\\\"Light rain, 13C\\\"}\"\n    },\n    {\n      \"role\": \"assistant\",\n      \"content\": \"Tomorrow in Paris, expect light rain and about 13°C.\"\n    }\n  ],\n  \"tools\": [\n    {\n      \"type\": \"function\",\n      \"function\": {\n        \"name\": \"get_weather\",\n        \"description\": \"Get a weather forecast for a city and date.\",\n        \"parameters\": {\n          \"type\": \"object\",\n          \"properties\": {\n            \"city\": {\n              \"type\": \"string\"\n            },\n            \"date\": {\n              \"type\": \"string\"\n            }\n          },\n          \"required\": [\"city\", \"date\"]\n        }\n      }\n    }\n  ]\n}\n```\n\nThe exact schema may differ depending on your model, trainer, and serving stack. The key point is not this exact JSON shape. The key point is that the training format should match the model’s tool-calling chat template.\n\n12. Final recommendation\n\nSo my answer would be:\n\n**Yes, start with SFT** if you have correct trajectories.\n**Do not train arbitrary custom roles directly** unless your target model’s template supports them.\n**Convert your annotations into the target tool-call format**, usually `tool_calls`\n\n, `tool`\n\nrole messages, and `tools`\n\nschemas.\n**Mask loss carefully**: user/system/tool observations should generally not be supervised; assistant tool calls and final answers should be.\n**Inspect the labels**, because assistant-only loss depends on the chat template.\n**Add no-tool, clarification, and unavailable-tool cases**, not only positive tool-call examples.\n**Use DPO** if you can create chosen/rejected trajectory pairs.\n**Use GRPO/RL only when you have executable tools and meaningful rewards**.\n**Evaluate more than final accuracy**: measure format validity, schema validity, tool selection, no-tool behavior, clarification behavior, grounding, final correctness, and tool-call cost.\n\nThe practical path is:\n\n```\nSFT first.\nDPO if you can create preference pairs.\nGRPO/RL only if you can run tools during rollout and compute reliable rewards.\n```\n\nUseful references:", "url": "https://wpnews.pro/news/finetuning-a-reasoning-llm-with-supervised-or-reinforcement-learning", "canonical_source": "https://discuss.huggingface.co/t/finetuning-a-reasoning-llm-with-supervised-or-reinforcement-learning/176449", "published_at": "2026-07-12 08:29:43+00:00", "updated_at": "2026-07-12 09:05:27.075817+00:00", "lang": "en", "topics": ["large-language-models", "ai-research", "developer-tools"], "entities": ["Hugging Face", "TRL", "SFTTrainer", "Transformers", "Agent Data Protocol"], "alternates": {"html": "https://wpnews.pro/news/finetuning-a-reasoning-llm-with-supervised-or-reinforcement-learning", "markdown": "https://wpnews.pro/news/finetuning-a-reasoning-llm-with-supervised-or-reinforcement-learning.md", "text": "https://wpnews.pro/news/finetuning-a-reasoning-llm-with-supervised-or-reinforcement-learning.txt", "jsonld": "https://wpnews.pro/news/finetuning-a-reasoning-llm-with-supervised-or-reinforcement-learning.jsonld"}}