# What compute do you need for RL training of LLM agents?

> Source: <https://dev.to/viswaretas_kotra/what-compute-do-you-need-for-rl-training-of-llm-agents-2mb>
> Published: 2026-11-04 19:14:54+00:00

RL training for an LLM agent needs two kinds of compute. GPUs generate the agent's actions and update its weights, and a fleet of isolated CPU sandboxes runs those actions so they can be scored. On Nodus, the GPUs come from a TrainingJob or a Job, and the sandboxes are Nodus Sandboxes that start with no network access and stop at a dollar cap. In our worked example below, the sandboxes are about a tenth of the bill; the expensive part is GPUs waiting while tool calls finish.

That last point decides most design choices, so this post walks through the parts of an agent RL setup, what each one costs, and when the managed path is enough.

An agent RL run is a loop of three jobs: generate, act, and learn. The policy model writes a reply or a tool call. An environment executes it, returns the result, and the episode continues for several turns. At the end, a grader turns the episode into a reward, and the trainer updates the model.

GRPO is a common choice for agent RL. It samples a group of episodes per task and rewards the ones that beat their group's average. It needs no critic model, which keeps GPU memory down. The [DeepSeekMath paper](https://arxiv.org/abs/2402.03300) introduced it, and our post on [what GRPO is and how to run it](https://www.nodus-compute.ai/blog/what-is-grpo-training/) covers the method itself.

The three jobs want different hardware:

| Part | What it does | Hardware it needs | 
|---|---|---|
| Policy and trainer | Samples actions, computes the loss, updates weights | GPUs sized to the model and method | 
| Environment | Runs code, shell commands or tool calls for each episode | Isolated CPU containers, one per live episode | 
| Grader | Scores the finished episode, holds the answers | A separate process the model cannot read | 

The environment is the part that is new compared with ordinary RL on text. A coding agent needs a real filesystem, a shell and a test runner for every episode in flight. That is why agent RL is as much a sandbox problem as a GPU problem.

The trainer's GPU follows from the model and method, the same as any fine-tune. With a LoRA adapter on a small model, one GPU is enough. The `nodus/grpo-lora` runtime's presets put Qwen3-8B with QLoRA on one L40S, RTX A6000 or A100 80GB, and the L40S is listed at $0.793/hr on the [Nodus pricing page](https://www.nodus-compute.ai/pricing/).

Full-parameter training is another matter. In our [PPO vs DPO vs GRPO comparison](https://www.nodus-compute.ai/blog/ppo-vs-dpo-vs-grpo/) we worked out that full GRPO on an 8B model needs about 131 GB at 16 bytes per parameter. That means several 80 GB GPUs before you count activations.

The group size also multiplies generation work. The Nodus `rl.train` recipe samples 8 replies for each of 4 tasks per step, capped at 256 tokens each. That is up to 8,192 generated tokens per step for single-turn tasks. A multi-turn agent episode repeats that generation on every turn, so plan generation capacity per turn, not per task.

Plan on one sandbox per episode that is running at the same moment. With 16 tasks per step and a group size of 8, that is 128 episodes in flight, so 128 sandboxes. Fewer sandboxes means episodes queue, and the GPUs wait for them.

On Nodus, a 1 vCPU, 2 GiB Sandbox is listed at $0.0179/hr on the pricing page. Keeping 128 of them running costs 128 x $0.0179 = $2.29 per hour. Sandboxes run on CPU machines Nodus operates, not on the GPU machines used for your jobs, so they never take GPU memory from the trainer.

Each one is isolated in ways that matter for code a model wrote. The [Sandbox isolation page](https://www.nodus-compute.ai/docs/concepts/sandboxes-isolation/) lists the layers: a user-space kernel that filters system calls, a default-deny firewall with cloud metadata addresses always blocked, a 256-process limit and a non-root user. The `Ephemeral` continuity mode starts every run empty, which fits a fresh sandbox per task.

Because the GPUs bill for every minute a step takes, including the minutes they spend waiting on tool calls. The worked example below shows the size of the effect.

Take an illustrative run on one 8 x H100 node with 128 sandboxes at 1 vCPU and 2 GiB. It trains for 300 steps at 3 minutes of wall-clock time per step, most of it spent waiting on tests and shell commands. These inputs are our assumptions for illustration, not a benchmark. The rates are from the Nodus pricing page, where the H100 SXM is listed at $2.600/hr. Prices checked November 4, 2026.

The run lasts 300 x 3 minutes = 15 hours. The GPUs cost 8 x $2.60 x 15 = $312.00. The sandboxes cost 128 x $0.0179 x 15 = $34.37. The total is $346.37, and the sandboxes are 9.9% of it.

*In this example the sandboxes are under a tenth of the bill; the GPUs are the rest.*

Now cut the step time in half, to 1.5 minutes, by running tool calls in parallel or overlapping generation with execution. The run drops to 7.5 hours. GPUs cost $156.00 and sandboxes $17.18, for $173.18 in total.

*Halving the step time saves $156.00 of GPU time, more than four times the entire $34.37 sandbox bill.*

From this arithmetic we get a decision rule. Never starve rollouts of sandboxes to save money, because a sandbox costs cents per hour while every idle minute costs the whole GPU node. Spend your engineering effort on step time: parallel episodes, fast environment resets and generation that does not sit idle during tool calls.

The managed path fits when your environment's tools are plain Python that runs offline. Nodus Environments are versioned task sets with a grader, and the [Environments guide](https://www.nodus-compute.ai/docs/guides/environments/) shows `nodus/grpo-lora` training on multi-turn games and tool-calling environments. The tools run inside the grader, and the trainer never sees the expected answers.

A tool-calling run is a short manifest, copied from that guide:

```
spec:
  runtime: nodus/grpo-lora
  environment: {name: tool-test@0.1.1, trainTasks: 21, heldOutTasks: 43, seed: 42}
  parameters:
    maxTurns: 3
    maxCompletionLength: 256
```

Each episode can call tools for up to `maxTurns` rounds, and the reward covers the whole episode. Program graders run in grading Sandboxes with no network access, billed to the TrainingJob and counted against its `maxCostUSD`. Before you launch, the recipe's `preview()` returns a cost estimate from the runtime's published per-step times.

Grading results are also built to keep training honest. A grader that crashes returns `InfrastructureFailure`, which carries no reward and does not count toward the pass rate. An outage never trains the model to believe a correct action was wrong.

You need your own trainer when tools need the network, a browser, an MCP server or another model as judge. The managed grader has no network, so the Environments guide says those environments fail the build. For them, run an open-source trainer such as [verl](https://github.com/volcengine/verl) or [SkyRL](https://github.com/NovaSky-AI/SkyRL) on GPUs, and give each episode a Nodus Sandbox through the Python SDK.

This sketch shows the sandbox side of one episode, using calls from the [Python Sandboxes guide](https://www.nodus-compute.ai/docs/guides/python/sandboxes/). The `policy` and `score` functions are placeholders for your own generation and reward code:

``` python
import nodus

def run_episode(task, policy, score, max_turns=6):
    sb = nodus.Sandbox.create(cpu=1, memory="2Gi", idle_timeout="5m", max_cost=1)
    try:
        with sb.open("/workspace/task.json", "w") as f:
            f.write(task.setup_json)
        history = [task.prompt]
        for _ in range(max_turns):
            command = policy(history)            # your model proposes a shell command
            if command is None:
                break
            p = sb.exec("bash", "-c", command)
            history.append(p.stdout.read())
            p.wait()
        return score(sb, task)                   # e.g. run the hidden tests
    finally:
        sb.terminate()
```

The egress default is deny, so the agent cannot reach the internet unless you open it with `network=nodus.Egress.open()`. Keep it closed unless a tool needs it, because code in a sandbox can read every secret you put there.

Be clear about the limits before you plan around them. Nodus has no managed PPO runtime, so a learned reward model with a value head means running your own trainer. Allow-list egress, where only named hosts are reachable, is available in Sandbox manifests but not yet in the Python SDK. Our worked example uses assumed step times; your environment's real reset and tool latency decide the numbers, so time a few episodes before you size a run.

Training recipes, TrainingJobs and GPU Sandbox state snapshots are Beta; Environments are generally available.

Yes, for a LoRA adapter on a model up to about 8B parameters. The smallest `nodus/grpo-lora` presets, Qwen3-0.6B and Qwen3-1.7B with LoRA, fit on one RTX 4090, listed at $0.290/hr on the pricing page. That is a cheap place to debug an environment before you scale up.

Only if the agent's tools need a GPU, such as an agent that writes and benchmarks CUDA kernels. Shell and test-running tools work fine on CPU sandboxes, which cost cents per hour. Nodus GPU Sandboxes run on dedicated on-demand machines and are never preempted; our [GPU sandboxes post](https://www.nodus-compute.ai/blog/gpu-sandboxes-for-ai-agents/) compares the options.

Keep the expected answers outside the model's reach. In Nodus grading, code from a completion runs as a separate user that cannot read the answers, and the trainer only sees prompts and metadata. With your own trainer, run hidden tests from a location the agent's sandbox never mounts.

Many of them, yes. The Environments guide covers `verifiers` packages, [OpenEnv](https://github.com/meta-pytorch/OpenEnv) environments and verl datasets with their reward functions unchanged, as long as grading needs no network.

*Originally published on the [Nodus blog](https://www.nodus-compute.ai/blog/rl-training-for-llm-agents/).*