SageMaker Multi-Turn RL Walkthrough: gpt-oss-20b Goes From 61% to 84% on Wordle for About $339

Amazon published a hands-on guide on October 23, 2026 to Multi-Turn Reinforcement Learning on Amazon SageMaker AI, in which gpt-oss-20b raised its Wordle solve rate from 61% to 84% on held-out words at default reasoning effort and from 15% to 69% at the low effort it was trained at. The AWS run used Group Relative Policy Optimization to update a LoRA adapter over 100 steps and 105,486 rollouts in 12 hours 40 minutes, at a cost of about $338.50 in tokens plus roughly $79 for Amazon Bedrock AgentCore runtime. AWS frames Wordle as a teaching example, with the pipeline's value being that users score whole multi-step episodes rather than specifying correct trajectories, letting a small model learn business-specific agent tasks.

Amazon published a hands-on guide on October 23, 2026 to Multi-Turn Reinforcement Learning MTRL on Amazon SageMaker AI , and the headline number is concrete. In AWS's run https://aws.amazon.com/blogs/machine-learning/multi-turn-reinforcement-learning-on-amazon-sagemaker-ai-a-hands-on-guide/?ref=explainx , gpt-oss-20b went from a 61 to an 84 percent solve rate on held-out Wordle words at its default reasoning effort, and from 15 to 69 percent at the low effort it was trained at. The training bill was about $339 in tokens, plus roughly $79 for the agent runtime. Wordle is a teaching example, and AWS says frontier models already play it well. The point is the pipeline: you describe how to score a whole task, not what the correct steps are, and SageMaker handles the training infrastructure. This post walks through that pipeline, the costs, and what to be careful about. It builds on our earlier look at SageMaker Training Sessions and writing your own RL loop https://explainx.ai/blog/amazon-sagemaker-training-session-apis-write-your-own-training-loop-grpo-no-gpus-2026 . Quick reference | Item | What AWS describes | |---|---| | Feature | Multi-Turn RL MTRL on SageMaker AI, serverless | | Training signal | One reward at the end of each multi-step episode | | Agent host | Amazon Bedrock AgentCore in the example; EKS, EC2 or Fargate through a Lambda forwarder also possible | | Algorithm | Group Relative Policy Optimization GRPO updating a LoRA adapter | | Example task | Wordle, six guesses, one tool called guess word | | Model in the example | gpt-oss-20b openai-reasoning-gpt-oss-20b | | Training run | 100 steps, 105,486 rollouts, 12 h 40 min | | Cost | About $338.50 tokens plus about $79 AgentCore runtime | | Result | 61% to 84% default effort ; 15% to 69% low effort | Why multi-turn RL, and what problem it addresses Most fine-tuning teaches a model to produce a better answer to one prompt. Agents are different. An agent that files an expense report or triages a ticket takes a sequence of actions, each depending on the last. AWS argues there is rarely one correct sequence to imitate, and the only reliable feedback is whether the whole sequence succeeded. Supervised fine-tuning has no labels for that, and single-turn RL cannot credit a decision made three steps before the outcome. MTRL trains on exactly that outcome signal. You do not specify the right trajectory, only how to score the result. AWS's stated use case is making a small, low-cost model good at a multi-step task specific to your business, such as an internal workflow public models were never trained on. This fits a wider pattern in our coverage: outcome rewards are also how labs train monitoring models, as in Anthropic's RL-trained alignment auditor https://explainx.ai/blog/anthropic-training-alignment-auditors-reinforcement-learning-haiku-matches-opus-4-6-2026 . How the system is split SageMaker AI training job connected to the AgentCore-hosted agent, Wordle environment, and reward function Image: AWS https://aws.amazon.com/blogs/machine-learning/multi-turn-reinforcement-learning-on-amazon-sagemaker-ai-a-hands-on-guide/?ref=explainx The design has two halves, per AWS: - SageMaker AI owns the training job and the policy model. - AgentCore hosts the agent, the Wordle environment and the reward function. For each rollout the MTRL service invokes the AgentCore runtime with one training row holding one secret word. A Strands agent in the runtime calls the policy model through the SageMaker Job Runtime's OpenAI-compatible endpoint. The model plays up to six guesses through one tool, guess word . After each guess the environment returns per-letter feedback and a summary of what is known. When the game ends, the agent returns a single reward. GRPO uses the spread of rewards across several rollouts of the same word to update the LoRA adapter. Three design points AWS highlights: 1. The agent and reward live in one place. One Python file deployed to AgentCore holds the environment, tool and scoring function. 2. The dataset carries only the secret word. The bundled word lists, built from the open-source SCOWL lists, are used only for scoring; the policy model never sees the answer. 3. The reward is outcome-dominant. A solve pays 1.05 to 1.55 depending on speed, a loss pays -0.5, and never making a valid guess pays -1.5. A small shaping term rewards good strategy within each tier. Which models and regions MTRL provisions training and sampling infrastructure for you. At the time of AWS's writing it supports four base models, each trained with a LoRA adapter: | Base model | Regions | |---|---| | Amazon Nova 2 Lite | us-east-1, us-west-2 | | GPT-OSS-20B | us-east-1, us-west-2 | | Gemma 4 31B instruction-tuned | us-west-2 | | Qwen 3.6 27B | us-west-2 | AWS chose GPT-OSS-20B because it has the lowest per-token price of the four and its reasoning effort can be set per request. What it costs You pay per million tokens in three dimensions, priced by base model: prefill input tokens, which accumulate in multi-turn tasks because every turn re-reads the history , sample generated tokens, including reasoning and train tokens processed in the backward pass . | Dimension | Tokens | Price per 1M | Cost | |---|---|---|---| | Prefill | 491,616,806 | $0.12 | $58.99 | | Sample | 159,737,098 | $0.30 | $47.92 | | Train | 643,308,554 | $0.36 | $231.59 | | Total | 1,294,662,458 | | $338.50 | Train tokens dominate and scale with trajectory length, so shorter games and less verbose reasoning cut the bill. The AgentCore runtime is billed separately for vCPU and memory, about $79 for this run according to CloudWatch usage metrics at us-east-1 list prices. DescribeJob reports token counts under BillableTokenUsage . These are AWS's list prices at the time of writing; check the SageMaker pricing page before planning a run. A small shape reaching for a green token at the end of a path, showing the single end-of-episode reward in multi-turn reinforcement learning The pipeline, step by step Step 1: two IAM roles The agent runtime role is created for you by agentcore deploy and needs the AmazonSageMakerJobRuntimeAccess managed policy. The training job role you create yourself. AWS flags a gotcha: it must trust job.sagemaker.amazonaws.com , not sagemaker.amazonaws.com , with both sts:AssumeRole and sts:TagSession , and carry AmazonSageMakerJobFullAccess . With the generic principal, CreateJob fails with "Could not assume role," and the message names the right principal only after retries. Step 2: deploy the agent Clone the aws-samples repository, set the account and Region in aws-targets.json , then run agentcore validate and agentcore deploy . The CLI builds dependencies for the runtime's Linux ARM64 platform and deploys through CloudFormation. AWS recommends it over uploading a code package in the console, where about 700 MB of compiled dependencies make mistakes common. The agent is a single main.py . The entrypoint decorator order matters: wrap the sagemaker rft handler with BedrockAgentCoreApp and @app.entrypoint , with the entrypoint outermost. Without it, AWS says every invocation times out after 30 seconds. Two environment variables control reasoning: REASONING EFFORT=low and REASONING PROMPT HINT=true . In AWS's evaluation jobs, low effort used 63 to 83 percent fewer sample tokens per game than medium. Step 3: verify before spending GPU time Invoke the runtime with a payload shaped like the training service's, using a fake job ARN. The expected result is a 400 "Could not find job" in CloudWatch and a printed reward of -1.5, which proves parsing, authentication and error reporting work. A 403 means the ARN names another account, and a 30-second timeout means the HTTP server never started. Step 4: dataset Each row is one task, with no target responses or labels. The service passes the prompt string to your agent verbatim. AWS made it a small JSON object with prompt , answer and id . The sample script draws 640 training words five batches of 128 and 128 non-overlapping validation words, so validation measures generalization. Parquet, JSON arrays and CSV are also accepted. A production task could pack a full message list, tool configuration or reward specification into the same string. Step 5: launch The launcher uses the MultiTurnRLTrainer class from the sagemaker-train SDK and requires an MLflow app for metrics. It sets sampling max tokens to 8192 the service maximum, because at 4096 gpt-oss often ran out of tokens before its first tool call , max epochs to 20 and max steps to 100. With 640 rows at the default global batch size of 128, that is 5 steps per epoch, and with the default group size of 8, each step plays 1,024 games. AWS also warns not to redeploy the agent while a job runs, since the service fails the whole job. Step 6: monitor in MLflow The metric to watch is val/reward/pass at 1 . AWS reports training reward climbing to about +0.57 within 15 steps and holding, while the validation solve rate jumped from 2 to 31 percent in the first 10 steps and then moved between 35 and 48 percent. With 128 validation words, AWS says those swings are mostly noise. It did not test whether more steps help at serving settings. AWS also shows one trained-model game, with the word PASTE solved in four guesses. The model keeps green letters in place, avoids reusing gray letters and uses yellow letters to rule out positions, behaviors the training data never demonstrated. Step 7: evaluate at serving settings An evaluation job plays your prompts through the same runtime and reward, on the base model or the trained package, and logs eval/reward/pass at 1 . Unlike training validation, you can set the temperature. AWS evaluated both models at low and medium effort and at temperature 0 and 1.0, with each 128-word evaluation taking four to seven minutes. It notes that at temperature 0 both models often end the game without a solve, so the in-training validation at temperature 0 understates the real gain. The default quota allows one evaluation job at a time. How to read the numbers Bar charts comparing base and trained gpt-oss-20b solve rate and mean reward across four decoding settings Image: AWS https://aws.amazon.com/blogs/machine-learning/multi-turn-reinforcement-learning-on-amazon-sagemaker-ai-a-hands-on-guide/?ref=explainx - One run, one task. The 61 to 84 percent figure is from a single training run on 128 held-out words. A small validation set means a few points of noise either way; AWS itself says the validation swings are mostly noise. - A game with an objective score. Wordle has few rules and a clear win condition. Business workflows often have fuzzier success criteria, and designing the reward is the hard part. AWS notes the reward was outcome-dominant with a small shaping term. - Settings matter more than headlines. The gap between 15 to 69 percent at low effort and 61 to 84 percent at default effort shows how much reasoning effort and temperature move results. - Costs add up with long episodes. Prefill re-reads the whole history each turn, and train tokens were over two thirds of the token bill. - A vendor guide. The numbers are AWS's own and come with AWS list prices. Nobody outside has reproduced them as far as we can tell. What this means for what you build or pay If you have an agent workflow with a clear pass or fail outcome, MTRL turns that outcome into a training signal without you labeling steps or managing GPUs. The practical decisions are the reward function, the dataset of tasks, and the cost envelope: budget for roughly the token mix above, and for the agent runtime on top. Test the reward on a handful of rollouts in the verification step before you pay for hundreds of thousands. There is also a security angle. Your agent runs on AgentCore during training and calls tools; a bad reward or an over-permissive tool lets the optimizer find shortcuts. Test your reward against ways to cheat it. For agent hijack risks on the same platform, see our write-up of the AgentCorruption research on Bedrock AgentCore https://explainx.ai/blog/agentcorruption-zenity-aws-bedrock-agentcore-hijack-every-agent-2026 , and for tool-selection choices see Apple's study on how agents should find tools https://explainx.ai/blog/apple-plan-or-retrieve-first-tool-discovery-architectures-toolbench-2026 . What to watch next - Whether AWS adds more base models or regions beyond the four listed. - Independent reproductions on tasks harder than Wordle. - Quota changes for evaluation jobs, currently one at a time by default. - How the reward-design lessons AWS shares in the rest of its guide hold up on real business tasks. Related reading Figures are from AWS's October 23, 2026 post and its list prices at the time; prices, models and regions may change.