{"slug": "multi-turn-agentic-reinforcement-learning-with-tunix-and-gemma-4-on-google-tpus", "title": "Multi-Turn Agentic Reinforcement Learning with Tunix and Gemma 4 on Google TPUs", "summary": "Google's Tunix agentic reinforcement learning pipeline raised the win rate of the Gemma 4 E2B model from 9.4% to 95% on the Plane Strike board game after 100 steps of multi-turn GRPO training on a single Cloud TPU v6e-8 host, cutting the average game from 22.44 strikes to about 16. The walkthrough, published on the Google Developers Blog, shows how to train a multi-turn tool-using LLM agent by subclassing Tunix's BaseTaskEnv for the 6x6 grid game, where the model calls a strike(cell=\"...\") tool, with complete code in a public GitHub repository.", "body_md": "In our previous posts - [Easy FunctionGemma finetuning with Tunix on Google TPUs](https://developers.googleblog.com/easy-functiongemma-finetuning-with-tunix-on-google-tpus/) and [Autonomous LLM post-training with Tunix on TPUs](https://developers.googleblog.com/autonomous-llm-post-training-with-tunix-on-tpus/) - we showed how [Google Tunix](https://github.com/google/tunix) streamlines Supervised Fine-Tuning (SFT) and single-turn Group Relative Policy Optimization (GRPO) on Cloud TPUs. Real-world AI agents, however, rarely finish their job in a single turn. They have to reason over partial observations, invoke tools, observe environment feedback, and adapt their strategy across multiple steps. So in this post, we will walk through how to train a multi-turn, tool-using LLM agent using Tunix's agentic reinforcement learning pipeline on Cloud TPU v6e-8. You can find the complete code in [this GitHub repository](https://github.com/windmaple/planestrike-gemma4-rl).\n\nWe will revisit the Plane Strike board game from an earlier [TensorFlow/JAX reinforcement learning blog](https://blog.tensorflow.org/2022/09/building-reinforcement-learning-agent-with-JAX-and-deploying-it-on-android-with-tensorflow-lite.html), replacing the custom MLP policy with Gemma 4 E2B in thinking mode on a single Cloud TPU v6e-8 host. Plane Strike is played on a 6x6 grid (`A1` to `F6`) where one 8-cell plane is hidden in one of 4 orientations. Each turn, the agent sees the result of its previous strike and the raw visible board (`.` untried, `X` hit, `O` miss), thinks through candidate placements, and calls the `strike(cell=\"...\")` tool to strike one cell. The game ends when all 8 plane cells are hit, or after 23 strikes.\n\nAlthough `google/gemma-4-E2B-it` has decent spatial reasoning and tool calling capabilities, out of the box, it wins only 9.4% of games on held-out boards (averaging 22.44 strikes). After 100 steps of agentic GRPO with Tunix, its win rate reaches 95% while cutting the average game length to ~16 strikes.\n\nWe set up the multi-turn RL pipeline in four modular components:\n\n`BaseTaskEnv` in `BaseTool` and `Gemma4ChatTemplateParser` in `ConversationAgentBase` in `TrajectoryCollectEngine` and `GRPOLearner` in Let's look at the key steps for each piece.\n\n`env.py`\nIn [env.py](https://github.com/windmaple/planestrike-gemma4-rl/blob/main/env.py), we subclass Tunix's `BaseTaskEnv` to implement `PlaneStrikeEnv`. Each environment instance receives a task dictionary with a `seed` that fixes the hidden plane position so all 8 rollouts in a GRPO group play the exact same board:\n\n```\nclass PlaneStrikeEnv(BaseTaskEnv):\n  \"\"\"One Plane Strike game. `entry[\"seed\"]` fixes the hidden plane position.\"\"\"\n\n  def __init__(\n      self,\n      entry: dict[str, Any],\n      group_id: int | None = None,\n      pair_index: int | None = None,\n      max_steps: int = MAX_STEPS,\n      **kwargs,\n  ):\n    entry = {k: _unwrap(v) for k, v in entry.items()}\n    self.seed = int(entry.get(\"seed\", 0))\n    self.split = str(entry.get(\"split\", \"train\"))\n    super().__init__(task=dict(entry), max_steps=max_steps, **kwargs)\n    self.extra_kwargs[\"group_id\"] = group_id\n    self.extra_kwargs[\"pair_index\"] = pair_index\n    self._reset_board()\n```\n\nOn each turn, [`_step_impl`](https://github.com/windmaple/planestrike-gemma4-rl/blob/main/env.py#L150-L179) evaluates the struck cell index against the hidden board, updates the visible grid, computes the step reward (summed into the trajectory-level reward over the episode), and returns `EnvStepResult(observation, reward, done, info)`.\n\nFor standard multi-tool agents, `tunix.rl.agentic` also provides ready-to-use `ToolEnvironment`, `ToolAgent` and `ToolManager` abstractions out of the box so we don't have to write custom message/observation plumbing unless we need custom turn control.\n\n`strike_tool.py`)\nIn Gemma 4, tool schemas are declared in the system prompt inside `<|tool>declaration:...<tool|>`, and a model turn that calls a tool ends with:\n\n```\n<|channel>thought\n...model reasoning...\n<channel|><|tool_call>call:strike{cell:<|\"|>C5<|\"|>}<tool_call|><|tool_response>\n```\n\n`<|tool_response>` is one of Gemma 4's native stop tokens: the model pauses there and waits for `response:strike{value:<|\"|>...<|\"|>}<tool_response|>`, then continues generating its next thought and tool call within the same assistant turn.\n\nIn [s`trike_tool.py`](https://github.com/windmaple/planestrike-gemma4-rl/blob/main/strike_tool.py), we define [`StrikeTool(BaseTool)`](https://github.com/windmaple/planestrike-gemma4-rl/blob/main/strike_tool.py#L28-L52) and subclass Tunix's `Gemma4ChatTemplateParser` to append the tool declaration to the system prompt and render `\"tool\"` messages as in-turn responses:\n\n```\nclass Gemma4ToolChatTemplateParser(parser_lib.Gemma4ChatTemplateParser):\n  \"\"\"Gemma4ChatTemplateParser plus native tool declarations and in-turn tool\n  responses.\"\"\"\n\n  def __init__(self, tokenizer, tools: list[base_tool.BaseTool], enable_thinking: bool = True):\n    super().__init__(tokenizer, enable_thinking=enable_thinking)\n    self.tool_declarations = render_tool_declarations(tokenizer, tools)\n    self.tool_response_id = tokenizer.convert_tokens_to_ids(TOOL_RESPONSE_START)\n\n  def _parse_system(self, content: str) -> str:\n    return self.tokens.system_token + content.strip() + self.tool_declarations + self.tokens.eot_token\n\n  def _parse_tool(self, content: str) -> str:\n    return f\"response:{TOOL_NAME}{{value:{_STR}{content}{_STR}}}<tool_response|>\"\n\n  def parse(self, messages, add_generation_prompt: bool = False, is_first_msg: bool = False) -> str:\n    if messages and messages[-1][\"role\"] == \"tool\":\n      add_generation_prompt = False  # The model continues its own turn.\n    return super().parse(messages, add_generation_prompt=add_generation_prompt,\n                         is_first_msg=is_first_msg)\n```\n\n`agent.py`)\nIn [`agent.py`](https://github.com/windmaple/planestrike-gemma4-rl/blob/main/agent.py), `PlaneStrikeAgent(ConversationAgentBase)` translates between environment observations and model messages.\n\nWhen the model finishes a turn, [tool_strike](https://github.com/windmaple/planestrike-gemma4-rl/blob/main/agent.py#L83-L90) ignores everything inside `<|channel>thought...<channel|>` and parses the final answer after `<channel|>`; only a final answer with a single valid `strike` call counts as a valid board coordinate (otherwise `INVALID_ACTION` is returned). If the model stopped at <|tool_response>, the next board observation is appended with `role=\"tool\"` so the model continues its turn:\n\n```\nclass PlaneStrikeAgent(base_agent.ConversationAgentBase):\n\n  def _observation_to_messages(\n      self, observation: Any, reward: float, done: bool, info: dict[str, Any]\n  ) -> None:\n    if not done:\n      role = \"tool\" if self._awaiting_tool_response else \"user\"\n      self._messages.append({\"role\": role, \"content\": str(observation)})\n\n  def update_from_model(self, response: str, **kwargs) -> agent_types.Action:\n    action = tool_strike(response, self.tool)\n    answer = final_answer(response)\n    self.tool_calls_per_turn.append(\n        0 if answer is None else len(strike_tool.parse_tool_calls(answer)))\n    self._awaiting_tool_response = response.rstrip().endswith(strike_tool.TOOL_RESPONSE_START)\n    self._messages.append({\"role\": \"assistant\", \"content\": response})\n    self._trajectory.steps.append(self.cur_step)\n    self.cur_step.action = action\n    self.cur_step.model_response = response\n    return agent_types.Action(action=action)\n```\n\n`train_planestrike.py`)\nIn [`train_planestrike.py`](https://github.com/windmaple/planestrike-gemma4-rl/blob/main/train_planestrike.py), we configure Tunix's asynchronous `vllm` rollout engine, `RLEngine`, and `GRPOLearner`:\n\n```\nrollout_config = base_rollout.RolloutConfig(\n    max_prompt_length=args.max_prompt_length,\n    max_tokens_to_generate=args.max_response_length,\n    kv_cache_size=args.max_prompt_length + args.max_response_length + 256,\n    temperature=args.temperature,\n    top_p=1.0,\n    top_k=0,\n    return_logprobs=True,\n    eos_tokens=[tokenizer.eos_token_id, tokenizer.convert_tokens_to_ids(\"<turn|>\"),\n                tokenizer.convert_tokens_to_ids(strike_tool.TOOL_RESPONSE_START)],\n    rollout_vllm_model_version=MODEL_VERSION,\n    rollout_vllm_hbm_utilization=args.vllm_hbm_utilization,\n    rollout_vllm_tpu_backend_type=\"jax\",\n    rollout_vllm_server_mode=True,\n    rollout_vllm_kwargs={\n        \"enable_prefix_caching\": True,\n        \"dtype\": \"bfloat16\",\n        \"hf_overrides\": {\n            \"final_logit_softcapping\": 30.0,\n            \"text_config\": {\"final_logit_softcapping\": 30.0},\n            \"architectures\": [\"Gemma4ForCausalLM\"],\n        },\n    },\n    rollout_vllm_sampling_kwargs={\"skip_special_tokens\": False},\n)\n```\n\nMulti-turn agentic RL is often bottlenecked by variable-length rollout generation, so we set `rollout_vllm_server_mode` to true to enable continuous asynchronous multi-turn rollouts via [`vllm-tpu`](https://github.com/vllm-project/tpu-inference) and `enable_prefix_caching to true` to true to cache the growing multi-turn conversation history. Since we are using a v6e-8 single host, the actor model, the reference model, and the rollout workers share the 8 chips.\n\nWe then pass our actor, tokenizer, `ClusterConfig`, `GRPOConfig` (`loss_algo=\"gspo-token\", advantage_estimator=\"rloo\", exact_token_continuity=True` to preserve exact token IDs across multi-turn rollouts without retokenization drift), agent, and environment into `RLEngine` and `PlaneStrikeGRPOLearner`:\n\n```\nrl_engine = rl_engine_lib.RLEngine(\n    actor=actor, reference=reference, tokenizer=tokenizer, cluster_config=cluster_config)\n\nlearner = PlaneStrikeGRPOLearner(\n    rl_engine=rl_engine,\n    agent_class=agent_lib.PlaneStrikeAgent,\n    agent_kwargs={},\n    env_class=env_lib.PlaneStrikeEnv,\n    env_kwargs={\"max_steps\": args.max_steps_per_game},\n    algo_config=grpo_config,\n    chat_parser=chat_parser,\n    metric_fns=[metric_fn],\n)\nlearner.train(train_dataset, eval_dataset=eval_dataset)\n```\n\nAll of these enable us to launch the 100-step training job with:\n\n```\npython train_planestrike.py \\\n  --run_dir runs/grpo_6x6_tool_thinking_100 \\\n  --learning_rate 2e-6 \\\n  --lr_end 2e-7\n```\n\nOver 100 steps (8 boards x 8 games per step), the agent learns both reliable tool calling and spatial deduction:\n\n`game/train/tool_call_rate` quickly reaches ~100%, while `game/train/invalid` drops to ~0 and `game/train/repeats` falls from ~0.55 to ~0.15. Because all strikes in a game currently share a single trajectory-level reward, further improvements can be achieved with finer per-turn credit assignment.\nWith Tunix's `tunix.rl.agentic` abstractions, `vllm-tpu`, and `GRPOLearner`, multi-turn, tool-using agentic RL on Cloud TPUs is easy and efficient. Check out the [planestrike-gemma4-rl GitHub repository](https://github.com/windmaple/planestrike-gemma4-rl) and read the [Tunix documentation](https://tunix.readthedocs.io/en/latest/) to train your own agentic model with Tunix and Google TPUs.", "url": "https://wpnews.pro/news/multi-turn-agentic-reinforcement-learning-with-tunix-and-gemma-4-on-google-tpus", "canonical_source": "https://developers.googleblog.com/multi-turn-agentic-reinforcement-learning-with-tunix-and-gemma-4-on-google-tpus/", "published_at": "2026-11-19 17:11:49.478302+00:00", "updated_at": "2026-11-19 17:11:52.082573+00:00", "lang": "en", "topics": ["ai-agents", "large-language-models", "ai-research", "machine-learning", "ai-tools"], "entities": ["Google", "Tunix", "Gemma 4 E2B", "Cloud TPU v6e-8", "Plane Strike", "Google Developers Blog", "Group Relative Policy Optimization"], "also_reported_by": [], "takeaway": "Training Gemma 4 E2B with 100 steps of multi-turn agentic GRPO in Tunix on one Cloud TPU v6e-8 host lifted Plane Strike win rate from 9.4% to 95% and cut average game length from 22.44 to about 16 strikes.", "recommendation": {"advice": "Fix the hidden plane position with a per-task seed so all 8 rollouts in a GRPO group play the same board, then sum step rewards into a trajectory-level reward for multi-turn training.", "playbook": "agent-architecture", "playbook_url": "https://wpnews.pro/playbooks/agent-architecture"}, "alternates": {"html": "https://wpnews.pro/news/multi-turn-agentic-reinforcement-learning-with-tunix-and-gemma-4-on-google-tpus", "markdown": "https://wpnews.pro/news/multi-turn-agentic-reinforcement-learning-with-tunix-and-gemma-4-on-google-tpus.md", "text": "https://wpnews.pro/news/multi-turn-agentic-reinforcement-learning-with-tunix-and-gemma-4-on-google-tpus.txt", "jsonld": "https://wpnews.pro/news/multi-turn-agentic-reinforcement-learning-with-tunix-and-gemma-4-on-google-tpus.jsonld"}}