{"slug": "multi-turn-reinforcement-learning-on-amazon-sagemaker-ai-a-hands-on-guide", "title": "Multi-Turn Reinforcement Learning on Amazon SageMaker AI: A hands-on guide", "summary": "Amazon Web Services published a hands-on guide showing that Multi-Turn Reinforcement Learning (MTRL) on Amazon SageMaker AI raised gpt-oss-20b's Wordle solve rate from 61% to 84% on held-out words at default reasoning effort, and from 15% to 69% at the low effort it trained at, for about $339 in training tokens. In the pipeline, SageMaker AI sends each training row to an agent hosted on Amazon Bedrock AgentCore, which calls the policy model through SageMaker's OpenAI-compatible endpoint, plays up to six guesses via a guess_word tool, and returns one episode-level reward that Group Relative Policy Optimization (GRPO) uses to update a LoRA adapter. AWS says the same pipeline applies to business-specific multi-step tasks, and that customer datasets, agents and trained models stay in the customer's AWS account and are not used to train base models.", "body_md": "## [Artificial Intelligence](https://aws.amazon.com/blogs/machine-learning/)\n\n# Multi-Turn Reinforcement Learning on Amazon SageMaker AI: A hands-on guide\n\nMost fine-tuning teaches a model to produce a better answer to a single prompt. Agentic tasks are different. An agent that files an expense report or triages a support ticket takes a sequence of actions, and each step depends on what came before. There’s rarely one correct sequence to imitate, and the only reliable feedback is whether the whole sequence succeeded. Supervised fine-tuning has no labels for that, and single-turn reinforcement learning cannot credit a decision made three steps before the outcome.\n\nMulti-Turn Reinforcement Learning (MTRL) on Amazon SageMaker AI trains models on exactly that signal. SageMaker AI sends prompts to an agent you host on Amazon Bedrock AgentCore, or on your own infrastructure, such as Amazon Elastic Kubernetes Service (Amazon EKS), Amazon Elastic Compute Cloud (Amazon EC2), or AWS Fargate, through an [AWS Lambda forwarder](https://docs.aws.amazon.com/sagemaker/latest/dg/model-customize-mtrl-agent.html). The example here uses AgentCore. The agent calls the policy model being trained, acts in your environment over as many turns as the task needs, and reports one reward when the episode ends. You don’t specify the right trajectory, only how to score the result.\n\nThis post uses [Wordle](https://en.wikipedia.org/wiki/Wordle), a game in which a player has six tries to guess a hidden five-letter word, as the example. After each guess, every letter turns green if it is in the right position, yellow if it’s in the word but in a different position, and gray if it isn’t in the word. The game suits MTRL because the rules are few, each guess depends on the feedback from the last, and the outcome is scored objectively. Wordle is a teaching example, and frontier models already play it well. In practice, you would use MTRL to make a small, low-cost model good at a multi-step task specific to your business, such as an internal workflow that public models were never trained on. The same pipeline applies to that setting.\n\nIn our run, MTRL took gpt-oss-20b from a 61 to an 84 percent solve rate on held-out words at its default reasoning effort, and from 15 to 69 percent at the low effort it trained at, for about $339 in training tokens. In this post, we walk through the pipeline end-to-end: deploying the agent and reward to AgentCore, preparing the dataset, training, monitoring in MLflow, and evaluating at serving settings. We also share the reward-design mistakes and training failures we hit. Your datasets, agent, and trained model stay in your AWS account and [are not used to train the base models](https://aws.amazon.com/sagemaker/ai/jumpstart/). All code is in the accompanying [aws-samples repository](https://github.com/aws-samples/sample-multi-turn-rl-wordle).\n\n## Solution overview\n\nThe system has two halves. SageMaker AI owns the training job and the policy model. AgentCore hosts the agent, the Wordle environment, and the reward function. The following diagram shows how they connect.\n\nFor every rollout, the MTRL service invokes the AgentCore runtime, a capability of Amazon Bedrock AgentCore, with one training row, which holds one secret word. A Strands agent in the runtime calls the policy model through the SageMaker Job Runtime’s OpenAI-compatible endpoint. The model plays up to six guesses through one tool, `guess_word`, and after each guess the environment returns per-letter feedback and a summary of what is known. When the game ends, the agent returns one reward. [Group Relative Policy Optimization (GRPO)](https://arxiv.org/abs/2402.03300) uses the spread of rewards across several rollouts of the same word to update a Low-Rank Adaptation ([LoRA](https://arxiv.org/abs/2106.09685)) adapter on the policy model.\n\nThree design points matter for what follows:\n\n- **The agent and reward live in one place.** A single Python file deployed to AgentCore holds the environment, the tool, and the scoring function.\n- **The dataset carries only the secret word.** The word lists bundled with the agent are built from the open-source[SCOWL](http://wordlist.aspell.net/) word lists and are used only for scoring.\n- **The reward is outcome-dominant.** A solve pays 1.05–1.55 depending on speed, a loss pays -0.5, and never making a valid guess pays -1.5. A small shaping term rewards good strategy within each tier.\n\n### Supported models and AWS Regions\n\nMTRL is serverless: you pick a base model and the service provisions the training and sampling infrastructure. At the time of writing, it supports the following base models, each trained with a LoRA adapter:\n\n| **Base model** | **AWS Regions** | \n| Amazon Nova 2 Lite | US East (N. Virginia) (us-east-1), US West (Oregon) (us-west-2) | \n| `GPT-OSS-20B` | US East (N. Virginia) (us-east-1), US West (Oregon) (us-west-2) | \n| Gemma 4 31B (instruction-tuned) | US West (Oregon) (us-west-2) | \n| Qwen 3.6 27B | US West (Oregon) (us-west-2) | \n\nWe chose `GPT-OSS-20B` because it has the lowest per-token price of the four and its reasoning effort is controllable per request. See the [MTRL documentation](https://docs.aws.amazon.com/sagemaker/latest/dg/model-customize-mtrl.html) for the current list.\n\n### Pricing\n\nYou don’t choose instances or size a cluster. You pay per million tokens in three dimensions, priced by base model:\n\n- **Prefill:** input tokens the policy model processes. These accumulate in a multi-turn task, because every turn re-reads the history.\n- **Sample:** tokens the policy model generates, including reasoning tokens.\n- **Train:** tokens processed in the backward pass that updates the LoRA adapter.\n\nThe 100-step run consumed 491.6 million prefill, 159.7 million sample, and 643.3 million train tokens across 105,486 rollouts in 12 hours 40 minutes. At `GPT-OSS-20B` list prices in the US East (N. Virginia) Region (us-east-1) at the time of writing, that is about $339:\n\n| **Dimension** | **Tokens** | **Price per 1M** | **Cost** | \n| Prefill | 491,616,806 | $0.12 | $58.99 | \n| Sample | 159,737,098 | $0.30 | $47.92 | \n| Train | 643,308,554 | $0.36 | $231.59 | \n| Total | **1,294,662,458** |  | **$338.50** | \n\nThe train dimension dominates and scales with trajectory length, so shorter games and less verbose reasoning cut the bill. The AgentCore runtime bills separately for the vCPU and memory the agent uses. For this run, that was about $79, from the runtime’s Amazon CloudWatch usage metrics at us-east-1 list prices. `DescribeJob` reports the token counts under `BillableTokenUsage`. See [Amazon SageMaker AI pricing](https://aws.amazon.com/sagemaker/ai/pricing/) for current rates.\n\n## Prerequisites\n\nTo follow this walkthrough, you need the following. See also the [MTRL prerequisites](https://docs.aws.amazon.com/sagemaker/latest/dg/model-customize-mtrl-prereqs.html).\n\n- Python 3.10 or later, uv, Node 20 or later, and the AgentCore CLI (`npm install -g @aws/agentcore` ). See[Get started with Amazon Bedrock AgentCore](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/agentcore-get-started-cli.html) for installation and the IAM permissions the CLI needs.\n- botocore 1.43.37 or later, on the machine that launches the job. This is the first version that includes the `AgentRFT`` CreateJob` operation.\n- Two AWS Identity and Access Management (IAM) roles, described in Step 1.\n- An Amazon Simple Storage Service (Amazon S3) bucket in the same Region as the agent. The training job must also run in that Region, because `CreateJob` rejects a cross-Region runtime ARN.\n\n## Step 1: Create the IAM roles\n\nMTRL uses two roles, and one of them has a trust policy that the general SageMaker AI documentation doesn’t mention.\n\n**Agent runtime role.** `agentcore deploy` creates this role for you. It trusts `bedrock-agentcore.amazonaws.com` and needs the `AmazonSageMakerJobRuntimeAccess` managed policy, which lets the agent call the policy model and report rewards. The CDK stack in the sample repository attaches this policy automatically.\n\n**Training job role.** You create this one. It must trust `job.sagemaker.amazonaws.com` with both `sts:AssumeRole` and `sts:TagSession`, and it needs the `AmazonSageMakerJobFullAccess` managed policy.\n\n**Note:** The trust principal is `job.sagemaker.amazonaws.com`, not `sagemaker.amazonaws.com`. With the generic principal, `CreateJob` fails with “Could not assume role,” and the message names the correct principal only after retries.\n\n## Step 2: Deploy the agent to Amazon Bedrock AgentCore\n\nClone the [sample repository](https://github.com/aws-samples/sample-multi-turn-rl-wordle), set your account and Region in Wordle/agentcore/`aws-targets.json`, and deploy:\n\nThe CLI builds the agent’s dependencies for the runtime’s Linux ARM64 platform, creates the execution role with `AmazonSageMakerJobRuntimeAccess` attached, and deploys through AWS CloudFormation. We recommend it over uploading a code package on the console, where it’s common to get the roughly 700 MB of compiled dependencies or the policy attachment wrong. Note the runtime ARN in the output for the training job.\n\nThe runtime reads two environment variables from `agentcore.json`. `REASONING_EFFORT=low` sets `reasoning_effort` on every request to the policy model. In our evaluation jobs, low effort used 63 to 83 percent fewer sample tokens per game than medium. `REASONING_PROMPT_HINT=true` also adds “Reasoning: low” to the system prompt, which is how gpt-oss reads the setting in its Harmony chat format. Neither variable gives the model any hint about the secret word.\n\nThe agent is one file, `main.py`. Simplified, its entrypoint looks like the following. For your own task, you change the task parsing, the tools, and the reward:\n\n**Important:** The `sagemaker_rft_handler` decorator does not start an HTTP server by itself. Wrap it with `BedrockAgentCoreApp` and `@app.entrypoint`, with the entrypoint decorator outermost. Without this, every invocation times out after 30 seconds.\n\n## Step 3: Verify the runtime\n\nBefore spending GPU time, confirm the whole chain works with a payload shaped like the one the training service sends. The repository includes one as Wordle/`payload.json`:\n\nThe fake job ARN is expected to fail, and the way it fails is the proof. Put your own account ID into it first: for a job ARN in another account, the service returns a 403 before it checks whether the job exists, which looks like a broken IAM setup. From the Wordle directory, run:\n\nIn the runtime’s Amazon CloudWatch logs, you see the agent parse the prompt, authenticate to the Job Runtime endpoint, and receive a 400 error reading “Could not find job.” That failure confirms that parsing, authentication, and error reporting work. The command itself prints {“reward”: -1.5}, because the agent never reached the policy model. A 403 reading “not authorized to access this resource” means the job ARN still names another account. A 30-second timeout instead means the HTTP server is not starting (see the Important note in Step 2).\n\n## Step 4: Prepare the dataset\n\nThe MTRL dataset has no target responses, transcripts, or labels. Each row is one task for the agent to attempt. Here are the first three rows of the training file:\n\nThe service reads the `prompt` column (or the first column) and passes the string to your agent verbatim, without parsing it. We made the string a small JSON object with three fields:\n\n- **prompt:** the instruction the agent shows the policy model, identical on every row.\n- **answer:** the secret word. Only the environment and the reward function use it. The policy model never sees it.\n- **id:** a row ID for tracing a rollout in the logs back to its source row.\n\nThe sample repository’s `make_dataset.py` draws 640 training words (five full batches of 128) and 128 validation words from the answer list with no overlap, so the validation solve rate measures generalization. The service also accepts Apache Parquet, JSON arrays, and CSV. Upload both files to Amazon S3:\n\nBecause the service never inspects the `prompt` string, a production task can pack a full message list, tool configuration, or reward specification into it. See the [MTRL documentation](https://docs.aws.amazon.com/sagemaker/latest/dg/model-customize-mtrl-assets.html) for examples, including how to protect sensitive content in the string.\n\n## Step 5: Launch the training job\n\nThe launcher uses the `MultiTurnRLTrainer` class from the `sagemaker-train` SDK. Point it at the runtime ARN from Step 2, the role from Step 1, the datasets from Step 4, and an MLflow app for metrics. The MLflow app is required, so if you don’t have one, create it first with `aws sagemaker create-mlflow-app`:\n\nThe launcher ov", "url": "https://wpnews.pro/news/multi-turn-reinforcement-learning-on-amazon-sagemaker-ai-a-hands-on-guide", "canonical_source": "https://aws.amazon.com/blogs/machine-learning/multi-turn-reinforcement-learning-on-amazon-sagemaker-ai-a-hands-on-guide/", "published_at": "2026-10-23 17:48:54+00:00", "updated_at": "2026-10-23 18:14:08.609565+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "ai-agents", "ai-tools", "ai-infrastructure"], "entities": ["Amazon Web Services", "Amazon SageMaker AI", "Amazon Bedrock AgentCore", "gpt-oss-20b", "Group Relative Policy Optimization", "Low-Rank Adaptation", "AWS Lambda", "Strands"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/multi-turn-reinforcement-learning-on-amazon-sagemaker-ai-a-hands-on-guide", "markdown": "https://wpnews.pro/news/multi-turn-reinforcement-learning-on-amazon-sagemaker-ai-a-hands-on-guide.md", "text": "https://wpnews.pro/news/multi-turn-reinforcement-learning-on-amazon-sagemaker-ai-a-hands-on-guide.txt", "jsonld": "https://wpnews.pro/news/multi-turn-reinforcement-learning-on-amazon-sagemaker-ai-a-hands-on-guide.jsonld"}}