# Multi-Turn Agentic Reinforcement Learning with Tunix and Gemma 4 on Google TPUs

> Source: <https://developers.googleblog.com/multi-turn-agentic-reinforcement-learning-with-tunix-and-gemma-4-on-google-tpus/>
> Published: 2026-11-19 17:11:49.478302+00:00

**Recommendation:** Fix the hidden plane position with a per-task seed so all 8 rollouts in a GRPO group play the same board, then sum step rewards into a trajectory-level reward for multi-turn training.
More in the playbook "Designing agent systems": <https://wpnews.pro/playbooks/agent-architecture>

In our previous posts - [Easy FunctionGemma finetuning with Tunix on Google TPUs](https://developers.googleblog.com/easy-functiongemma-finetuning-with-tunix-on-google-tpus/) and [Autonomous LLM post-training with Tunix on TPUs](https://developers.googleblog.com/autonomous-llm-post-training-with-tunix-on-tpus/) - we showed how [Google Tunix](https://github.com/google/tunix) streamlines Supervised Fine-Tuning (SFT) and single-turn Group Relative Policy Optimization (GRPO) on Cloud TPUs. Real-world AI agents, however, rarely finish their job in a single turn. They have to reason over partial observations, invoke tools, observe environment feedback, and adapt their strategy across multiple steps. So in this post, we will walk through how to train a multi-turn, tool-using LLM agent using Tunix's agentic reinforcement learning pipeline on Cloud TPU v6e-8. You can find the complete code in [this GitHub repository](https://github.com/windmaple/planestrike-gemma4-rl).

We will revisit the Plane Strike board game from an earlier [TensorFlow/JAX reinforcement learning blog](https://blog.tensorflow.org/2022/09/building-reinforcement-learning-agent-with-JAX-and-deploying-it-on-android-with-tensorflow-lite.html), replacing the custom MLP policy with Gemma 4 E2B in thinking mode on a single Cloud TPU v6e-8 host. Plane Strike is played on a 6x6 grid (`A1` to `F6`) where one 8-cell plane is hidden in one of 4 orientations. Each turn, the agent sees the result of its previous strike and the raw visible board (`.` untried, `X` hit, `O` miss), thinks through candidate placements, and calls the `strike(cell="...")` tool to strike one cell. The game ends when all 8 plane cells are hit, or after 23 strikes.

Although `google/gemma-4-E2B-it` has decent spatial reasoning and tool calling capabilities, out of the box, it wins only 9.4% of games on held-out boards (averaging 22.44 strikes). After 100 steps of agentic GRPO with Tunix, its win rate reaches 95% while cutting the average game length to ~16 strikes.

We set up the multi-turn RL pipeline in four modular components:

`BaseTaskEnv` in `BaseTool` and `Gemma4ChatTemplateParser` in `ConversationAgentBase` in `TrajectoryCollectEngine` and `GRPOLearner` in Let's look at the key steps for each piece.

`env.py`
In [env.py](https://github.com/windmaple/planestrike-gemma4-rl/blob/main/env.py), we subclass Tunix's `BaseTaskEnv` to implement `PlaneStrikeEnv`. Each environment instance receives a task dictionary with a `seed` that fixes the hidden plane position so all 8 rollouts in a GRPO group play the exact same board:

```
class PlaneStrikeEnv(BaseTaskEnv):
  """One Plane Strike game. `entry["seed"]` fixes the hidden plane position."""

  def __init__(
      self,
      entry: dict[str, Any],
      group_id: int | None = None,
      pair_index: int | None = None,
      max_steps: int = MAX_STEPS,
      **kwargs,
  ):
    entry = {k: _unwrap(v) for k, v in entry.items()}
    self.seed = int(entry.get("seed", 0))
    self.split = str(entry.get("split", "train"))
    super().__init__(task=dict(entry), max_steps=max_steps, **kwargs)
    self.extra_kwargs["group_id"] = group_id
    self.extra_kwargs["pair_index"] = pair_index
    self._reset_board()
```

On each turn, [`_step_impl`](https://github.com/windmaple/planestrike-gemma4-rl/blob/main/env.py#L150-L179) evaluates the struck cell index against the hidden board, updates the visible grid, computes the step reward (summed into the trajectory-level reward over the episode), and returns `EnvStepResult(observation, reward, done, info)`.

For standard multi-tool agents, `tunix.rl.agentic` also provides ready-to-use `ToolEnvironment`, `ToolAgent` and `ToolManager` abstractions out of the box so we don't have to write custom message/observation plumbing unless we need custom turn control.

`strike_tool.py`)
In Gemma 4, tool schemas are declared in the system prompt inside `<|tool>declaration:...<tool|>`, and a model turn that calls a tool ends with:

```
<|channel>thought
...model reasoning...
<channel|><|tool_call>call:strike{cell:<|"|>C5<|"|>}<tool_call|><|tool_response>
```

`<|tool_response>` is one of Gemma 4's native stop tokens: the model pauses there and waits for `response:strike{value:<|"|>...<|"|>}<tool_response|>`, then continues generating its next thought and tool call within the same assistant turn.

In [s`trike_tool.py`](https://github.com/windmaple/planestrike-gemma4-rl/blob/main/strike_tool.py), we define [`StrikeTool(BaseTool)`](https://github.com/windmaple/planestrike-gemma4-rl/blob/main/strike_tool.py#L28-L52) and subclass Tunix's `Gemma4ChatTemplateParser` to append the tool declaration to the system prompt and render `"tool"` messages as in-turn responses:

```
class Gemma4ToolChatTemplateParser(parser_lib.Gemma4ChatTemplateParser):
  """Gemma4ChatTemplateParser plus native tool declarations and in-turn tool
  responses."""

  def __init__(self, tokenizer, tools: list[base_tool.BaseTool], enable_thinking: bool = True):
    super().__init__(tokenizer, enable_thinking=enable_thinking)
    self.tool_declarations = render_tool_declarations(tokenizer, tools)
    self.tool_response_id = tokenizer.convert_tokens_to_ids(TOOL_RESPONSE_START)

  def _parse_system(self, content: str) -> str:
    return self.tokens.system_token + content.strip() + self.tool_declarations + self.tokens.eot_token

  def _parse_tool(self, content: str) -> str:
    return f"response:{TOOL_NAME}{{value:{_STR}{content}{_STR}}}<tool_response|>"

  def parse(self, messages, add_generation_prompt: bool = False, is_first_msg: bool = False) -> str:
    if messages and messages[-1]["role"] == "tool":
      add_generation_prompt = False  # The model continues its own turn.
    return super().parse(messages, add_generation_prompt=add_generation_prompt,
                         is_first_msg=is_first_msg)
```

`agent.py`)
In [`agent.py`](https://github.com/windmaple/planestrike-gemma4-rl/blob/main/agent.py), `PlaneStrikeAgent(ConversationAgentBase)` translates between environment observations and model messages.

When the model finishes a turn, [tool_strike](https://github.com/windmaple/planestrike-gemma4-rl/blob/main/agent.py#L83-L90) ignores everything inside `<|channel>thought...<channel|>` and parses the final answer after `<channel|>`; only a final answer with a single valid `strike` call counts as a valid board coordinate (otherwise `INVALID_ACTION` is returned). If the model stopped at <|tool_response>, the next board observation is appended with `role="tool"` so the model continues its turn:

```
class PlaneStrikeAgent(base_agent.ConversationAgentBase):

  def _observation_to_messages(
      self, observation: Any, reward: float, done: bool, info: dict[str, Any]
  ) -> None:
    if not done:
      role = "tool" if self._awaiting_tool_response else "user"
      self._messages.append({"role": role, "content": str(observation)})

  def update_from_model(self, response: str, **kwargs) -> agent_types.Action:
    action = tool_strike(response, self.tool)
    answer = final_answer(response)
    self.tool_calls_per_turn.append(
        0 if answer is None else len(strike_tool.parse_tool_calls(answer)))
    self._awaiting_tool_response = response.rstrip().endswith(strike_tool.TOOL_RESPONSE_START)
    self._messages.append({"role": "assistant", "content": response})
    self._trajectory.steps.append(self.cur_step)
    self.cur_step.action = action
    self.cur_step.model_response = response
    return agent_types.Action(action=action)
```

`train_planestrike.py`)
In [`train_planestrike.py`](https://github.com/windmaple/planestrike-gemma4-rl/blob/main/train_planestrike.py), we configure Tunix's asynchronous `vllm` rollout engine, `RLEngine`, and `GRPOLearner`:

```
rollout_config = base_rollout.RolloutConfig(
    max_prompt_length=args.max_prompt_length,
    max_tokens_to_generate=args.max_response_length,
    kv_cache_size=args.max_prompt_length + args.max_response_length + 256,
    temperature=args.temperature,
    top_p=1.0,
    top_k=0,
    return_logprobs=True,
    eos_tokens=[tokenizer.eos_token_id, tokenizer.convert_tokens_to_ids("<turn|>"),
                tokenizer.convert_tokens_to_ids(strike_tool.TOOL_RESPONSE_START)],
    rollout_vllm_model_version=MODEL_VERSION,
    rollout_vllm_hbm_utilization=args.vllm_hbm_utilization,
    rollout_vllm_tpu_backend_type="jax",
    rollout_vllm_server_mode=True,
    rollout_vllm_kwargs={
        "enable_prefix_caching": True,
        "dtype": "bfloat16",
        "hf_overrides": {
            "final_logit_softcapping": 30.0,
            "text_config": {"final_logit_softcapping": 30.0},
            "architectures": ["Gemma4ForCausalLM"],
        },
    },
    rollout_vllm_sampling_kwargs={"skip_special_tokens": False},
)
```

Multi-turn agentic RL is often bottlenecked by variable-length rollout generation, so we set `rollout_vllm_server_mode` to true to enable continuous asynchronous multi-turn rollouts via [`vllm-tpu`](https://github.com/vllm-project/tpu-inference) and `enable_prefix_caching to true` to true to cache the growing multi-turn conversation history. Since we are using a v6e-8 single host, the actor model, the reference model, and the rollout workers share the 8 chips.

We then pass our actor, tokenizer, `ClusterConfig`, `GRPOConfig` (`loss_algo="gspo-token", advantage_estimator="rloo", exact_token_continuity=True` to preserve exact token IDs across multi-turn rollouts without retokenization drift), agent, and environment into `RLEngine` and `PlaneStrikeGRPOLearner`:

```
rl_engine = rl_engine_lib.RLEngine(
    actor=actor, reference=reference, tokenizer=tokenizer, cluster_config=cluster_config)

learner = PlaneStrikeGRPOLearner(
    rl_engine=rl_engine,
    agent_class=agent_lib.PlaneStrikeAgent,
    agent_kwargs={},
    env_class=env_lib.PlaneStrikeEnv,
    env_kwargs={"max_steps": args.max_steps_per_game},
    algo_config=grpo_config,
    chat_parser=chat_parser,
    metric_fns=[metric_fn],
)
learner.train(train_dataset, eval_dataset=eval_dataset)
```

All of these enable us to launch the 100-step training job with:

```
python train_planestrike.py \
  --run_dir runs/grpo_6x6_tool_thinking_100 \
  --learning_rate 2e-6 \
  --lr_end 2e-7
```

Over 100 steps (8 boards x 8 games per step), the agent learns both reliable tool calling and spatial deduction:

`game/train/tool_call_rate` quickly reaches ~100%, while `game/train/invalid` drops to ~0 and `game/train/repeats` falls from ~0.55 to ~0.15. Because all strikes in a game currently share a single trajectory-level reward, further improvements can be achieved with finer per-turn credit assignment.
With Tunix's `tunix.rl.agentic` abstractions, `vllm-tpu`, and `GRPOLearner`, multi-turn, tool-using agentic RL on Cloud TPUs is easy and efficient. Check out the [planestrike-gemma4-rl GitHub repository](https://github.com/windmaple/planestrike-gemma4-rl) and read the [Tunix documentation](https://tunix.readthedocs.io/en/latest/) to train your own agentic model with Tunix and Google TPUs.
