{"slug": "deploying-multi-turn-rl-infrastructure-for-amazon-nova-on-amazon-sagemaker", "title": "Deploying Multi-Turn RL Infrastructure for Amazon Nova on Amazon SageMaker HyperPod", "summary": "Amazon Web Services announced a new multi-turn reinforcement learning infrastructure for Amazon Nova on Amazon SageMaker HyperPod, enabling training of enterprise agents on multi-step workflows. The event-driven pipeline automatically provisions compute, routes rewards, and runs training when data is uploaded to Amazon S3, using a two-phase architecture with long-lived and ephemeral resources.", "body_md": "[Artificial Intelligence](https://aws.amazon.com/blogs/machine-learning/)\n\n# Deploying Multi-Turn RL Infrastructure for Amazon Nova on Amazon SageMaker HyperPod\n\nWhen you build enterprise agents that execute multi-step workflows, you face a fundamental training challenge. These agents query databases, call APIs, cross-reference results, and recover from mid-process failures. The quality of any single action depends on what happens several steps later.\n\nStandard [reinforcement learning from human feedback (RLHF)](https://aws.amazon.com/what-is/reinforcement-learning-from-human-feedback/) optimizes single responses in isolation. This approach falls short for multi-step workflows where an agent that validates data before proceeding prevents a cascade of downstream errors. Multi-turn reinforcement learning (RL) addresses this gap by optimizing over entire interaction sequences. Your agents learn tool orchestration, error recovery, and multi-step reasoning through trial and error. Supervised fine-tuning (SFT), [retrieval-augmented generation (RAG)](https://aws.amazon.com/what-is/retrieval-augmented-generation/), and continued pre-training are complementary techniques, but they typically do not teach these sequential decision-making capabilities on their own.\n\n[Amazon SageMaker AI](https://aws.amazon.com/sagemaker/ai/) also offers multi-turn RL as a fully managed, serverless capability, bringing this technique to SageMaker training jobs with no infrastructure to manage. When you need full control over the training stack: your own agent environment, custom orchestration, or specific instance configurations. For these cases, the multi-turn RL infrastructure for Amazon Nova on Amazon SageMaker HyperPod gives you the compute, orchestration, and reward-routing layers to train agents on these complex workflows. Amazon Nova delivers frontier intelligence and industry-leading price performance, and Amazon Nova Forge extends this with multi-turn RL training capabilities.\n\nIn this post, you deploy a two-phase infrastructure for multi-turn RL using Amazon Nova Forge on Amazon SageMaker HyperPod. By the end, you have an event-driven pipeline that starts training when you upload data to [Amazon Simple Storage Service (Amazon S3)](https://aws.amazon.com/s3/). The training job teaches the model to play Wordle, a placeholder for your own RL task.\n\n## Solution overview\n\nThe solution is an event-driven pipeline: you upload a dataset to Amazon S3, and the infrastructure provisions compute, routes rewards, and runs multi-turn RL training automatically. Three layers do the work. A SageMaker HyperPod cluster generates responses and applies GRPO (Group Relative Policy Optimization) weight updates. ECS on AWS Fargate runs your reward environment. The Nova Forge SDK routes messages between the model and that environment while tracking conversation state across turns. AWS Step Functions orchestrates the run, triggered by Amazon EventBridge when data lands in S3.\n\nThe architecture is split into two phases: a one-time AWS Cloud Development Kit (AWS CDK) deployment provisions the long-lived foundation (VPC, EKS/HyperPod, ECS, S3, IAM, and the pipeline), while each training run spins up its own ephemeral resources. This keeps GPU compute from sitting idle between runs and lets you iterate without redeploying.\n\nThe following diagram shows how these components interact during a training run.\n\nThe pipeline orchestrates a multi-turn conversation loop across three compute surfaces:\n\nTraining primary, worker pods, and vLLM generation replicas run on P5 instances. The model generates responses. Training pods perform GRPO weight updates using reward signals.**Amazon SageMaker HyperPod (EKS)**:** ECS on Fargate**: Reward workers run your environment (for example, Wordle or a custom Bring Your Own Orchestrator (BYOO) environment). They receive model responses via SQS, score them against your rubric, and return reward signals.: The SDK’s proxy layer routes messages between the model and the reward environment, tracking conversation state across turns.[Amazon Nova Forge](https://aws.amazon.com/nova/forge/)\n\n## Prerequisites\n\nBefore you deploy, make sure you have the following:\n\n**Amazon Nova Forge subscription**: You need this subscription to access the Nova Forge SDK and model training APIs.** SageMaker HyperPod instance quota**: Minimum of 10 ×`ml.p5.48xlarge`\n\nfor`generation_replicas: 4`\n\n. Request 12–14 for production workloads.`ml.p5en.48xlarge`\n\nis also supported.**Bootstrapped CDK environment**: Run`cdk bootstrap`\n\nbefore your first deploy.**Python 3.12+**: The CDK app and AWS Lambda runtime require this version.** AWS CDK v2**: Install with`npm install -g aws-cdk`\n\n. This synthesizes and deploys the AWS CloudFormation stack.**AWS Command Line Interface (AWS CLI) v2**: Configured with credentials that have permissions to create VPCs, EKS clusters, SageMaker HyperPod clusters, IAM roles, and Step Functions.**Docker**: You use Docker to build the Lambda container images that package the Nova Forge SDK.\n\nImportant: This infrastructure costs approximately $786–$1,180 per hour when running (10–12\n\n`ml.p5.48xlarge`\n\ninstances). Review the Cost breakdown section and plan to destroy the stack when you are not actively training.\n\n## Deploy the infrastructure\n\n### Clone and install\n\nStart by cloning the sample repository and installing the Python dependencies for the AWS CDK app. Run these commands from your local machine or development environment:\n\n### Configure\n\nSet all parameters in `cdk.json`\n\nunder the `context`\n\nkey. Two parameters are **required** before your first deployment:\n\nParameter |\nDescription |\n`project_tag` |\nUnique prefix for all resource names (for example, `my-nova-rl` “. Becomes part of your EKS cluster name, HyperPod cluster name, and S3 bucket tags. |\n`sdk_resource_prefix` |\nPrefix for SDK-created resources (for example, `nrl-myproject` ). The Nova Forge SDK uses this when naming its CloudFormation stacks, Lambda functions, and SQS queues. |\n\nKey infrastructure parameters:\n\nParameter |\nDefault |\nDescription |\n`instance_type` |\n`ml.p5.48xlarge` |\nHyperPod instance type |\n`instance_count` |\n`10` |\nNumber of instances in the Restricted Instance Group |\n`nova_model` |\n`NOVA_LITE_2` |\nModel to train: `NOVA_MICRO` , `NOVA_LITE` , `NOVA_LITE_2` , or `NOVA_PRO` |\n`vf_env_id` |\n`wordle` |\nBuilt-in reward environment for validation |\n`use_custom_env` / `custom_env_id` |\n— | Set `use_custom_env` to “true” and specify the directory name under `custom-environments/` for BYOO |\n`reward_cpu` / `reward_memory` |\n2048 CPU, 4096 MiB | Fargate task sizing |\n`eks_kubernetes_version` |\n1.32 | EKS cluster version |\n\nTraining parameters. These flow through the pipeline event to the training job:\n\nParameter |\nDefault |\nDescription |\n`training_method` |\n— | `RFT_MULTITURN_FULL` or `RFT_MULTITURN_LORA` |\n`max_steps` |\n`10` |\nNumber of training steps |\n`generation_replicas` |\n`4` |\nvLLM generation replicas |\n| global_batch_size | `64` |\nSamples per training step |\n\nYou can override any parameter at deploy time:\n\n### Deploy\n\n### Two-phase deployment model\n\nThe infrastructure follows a two-phase deployment model that separates long-lived foundational resources from ephemeral per-run resources. This separation reduces costs by creating expensive compute resources only when you need them, speeds up iteration by avoiding full redeployment for each training run, and simplifies management by giving each layer an independent lifecycle.\n\n**Phase 1: AWS CDK deploy (one-time):** When you run\n\n`cdk deploy`\n\n, you provision the foundational infrastructure:- An\n[Amazon Virtual Private Cloud (Amazon VPC)](https://docs.aws.amazon.com/vpc/latest/userguide/what-is-amazon-vpc.html)with private subnets and VPC endpoints. - An\n[Amazon Elastic Kubernetes Service (Amazon EKS)](https://docs.aws.amazon.com/eks/latest/userguide/what-is-eks.html)cluster with a HyperPod Restricted Instance Group (RIG). - An\n[Amazon Elastic Container Service (Amazon ECS)](https://docs.aws.amazon.com/ecs/)cluster on AWS Fargate for reward workers. - An\n[Amazon Simple Storage Service (Amazon S3)](https://docs.aws.amazon.com/s3/)bucket for training data and checkpoints. [AWS Identity and Access Management (IAM) roles](https://docs.aws.amazon.com/iam/).[An AWS Step Functions pipeline](https://docs.aws.amazon.com/step-functions/).[Amazon EventBridge rules](https://docs.aws.amazon.com/eventbridge/).\n\nThe deployment takes approximately 30-40 minutes. The majority of time is spent on EKS cluster creation (~15 minutes), followed by SageMaker HyperPod Helm chart installation via AWS CodeBuild (~5 minutes), SageMaker HyperPod cluster provisioning (~15-25 minutes, depending on P5 capacity), and Lambda container image builds (~5 minutes).\n\n**Phase 2: Runtime (per training run):** When you upload a `.jsonl`\n\nfile to the S3 bucket’s `training-data/`\n\nprefix, EventBridge triggers a Step Functions pipeline that creates the runtime resources for that training run. The [Nova Forge SDK](https://github.com/aws/nova-forge-sdk) deploys its own [AWS CloudFormation](https://aws.amazon.com/cloudformation/) stack containing [AWS Lambda](https://aws.amazon.com/lambda/) functions (conversation proxy), [Amazon Simple Queue Service (Amazon SQS)](https://aws.amazon.com/sqs/) FIFO queues (message routing between model and environment), an [Amazon DynamoDB](https://aws.amazon.com/dynamodb/) table (conversation state tracking), and ECS Fargate tasks (reward workers). The SDK manages the lifecycle of these resources.\n\n## Trigger a training run\n\nAfter you deploy the infrastructure, you start a training run with a single S3 upload. EventBridge watches the bucket’s `training-data/`\n\nprefix and starts the Step Functions pipeline automatically.\n\nA successful training run shows increasing reward scores in the Amazon CloudWatch metrics over successive steps. For the Wordle environment, expect the model to converge within 50-100 steps, with average reward improving from near-zero to 0.6-0.8 as the model learns to narrow guesses based on feedback.\n\n### Prepare your training data\n\nThe `.jsonl`\n\nfile uses a metadata-based format: each line contains a prompt and the ground-truth answer that the reward environment uses for scoring:\n\nThe `id`\n\nfield must be unique across all records. The model sees `metadata.prompt`\n\nat the start of each conversation, and the reward environment uses `metadata.answer`\n\nto score the model’s responses across turns. Each training example becomes a multi-turn conversation: the model generates a guess, receives feedback from the reward environment, and iterates until it solves the puzzle or exhausts its turns.\n\n### Upload and trigger automatically\n\nWhen a file lands in the `training-data/`\n\nprefix, EventBridge detects the PutObject event and invokes the `S3TriggerFn`\n\nLambda function. This function extracts the file path, merges it with the training parameters from `cdk.json`\n\n, and passes the resulting event data to Step Functions, which executes the pipeline through five stages. You can watch progress in the Step Functions console, where each step shows its input, output, duration, and retry count.\n\n### Trigger manually for ad-hoc runs\n\nTo iterate on parameters without re-uploading data, use the setup script to bypass EventBridge and invoke the pipeline directly:\n\nThis helps during development: point at an existing dataset and experiment with different step counts, batch sizes, or the training method (full fine-tuning versus LoRA) without modifying `cdk.json`\n\nor redeploying.\n\n## Monitor and debug\n\nYou can monitor and troubleshoot every layer of this multi-stage pipeline across SageMaker HyperPod, ECS, Lambda, and SQS. The infrastructure provides observability at every layer so you can isolate issues quickly.\n\n### Pipeline monitoring\n\nThe Step Functions console is the primary dashboard. Each execution shows step-by-step progress with timing, input and output data, and retry history. Every step writes to its own Amazon CloudWatch Log Group.\n\nDuring normal operation, each Step Functions step completes within its expected time window: infrastructure setup in 2-3 minutes, reward worker deployment in 1-2 minutes, data validation in under 1 minute, and training submission in 3-5 minutes. SQS queues should show messages flowing steadily with no sustained backlog.\n\n### Failure alerting\n\nAn Amazon Simple Notification Service (Amazon SNS) topic, encrypted with AWS Key Management Service (AWS KMS), publishes a notification when a Step Functions execution fails. Subscribe to receive alerts by email, Slack, or PagerDuty:\n\n### Trigger failures\n\nIf an S3 upload does not start the pipeline, check the dead-letter queue. Failed EventBridge-to-Lambda invocations are stored here with the original S3 event data and error details:\n\nCommon causes include malformed file names (the trigger expects a `.jsonl`\n\nextension), IAM permission drift, and Lambda concurrency limits.\n\n### Training stall debugging\n\nIf the pipeline starts but training stalls, the root cause is usually in the message-routing layer between the model and the reward environment. Check SQS queue health using the SDK built-in diagnostic:\n\nA growing backlog in the request queue with an empty response queue indicates that reward workers are not processing. A growing backlog in the response queue indicates that the model is not consuming rewards. Either pattern tells you which component to investigate.\n\n### HyperPod health\n\nVerify that all training and generation pods are running on the SageMaker HyperPod cluster:\n\nAll pods should show `Running`\n\nor `Completed`\n\nstatus. Pods stuck in `Pending`\n\nindicate insufficient instance capacity in the Restricted Instance Group. Pods in `CrashLoopBackOff`\n\n(a Kubernetes status indicating the container is repeatedly crashing and restarting) indicate container-level errors. Check the pod logs with `kubectl logs <pod-name> -n kubeflow`\n\nfor details.\n\n## Clean up and manage cost\n\nThe cleanup script stops active resources in this order: Step Functions executions, ECS tasks, the SDK CloudFormation stack, and the CDK stack.\n\nTo avoid charges, run the following command to destroy the entire stack:\n\nTo destroy the infrastructure while keeping your training data in S3 (storage charges still apply):\n\nImportant:Destroy the stack when you are not actively training. SageMaker HyperPod instances and NAT Gateway idle costs accumulate quickly.\n\n### Cost breakdown\n\nResource |\nHourly Cost |\nNotes |\n| SageMaker HyperPod 8 × ml.p5.48xlarge | ~$786/hr | Minimum configuration (10 instances, 8 used for compute) |\n| SageMaker HyperPod 12 × ml.p5.48xlarge | ~$1,180/hr | Production configuration (12 instances with headroom) |\n| EKS control plane | $0.10/hr | |\n| NAT Gateway | ~$0.045/hr | |\n| ECS Fargate reward workers | ~$0.15-0.30/hr | + data transfer |\n| S3, Lambda, CodeBuild, SQS, DynamoDB | Negligible |\n\nFor current Amazon Nova Forge pricing, see the [Amazon Nova pricing page](https://aws.amazon.com/ai/generative-ai/nova/).\n\n### Optimize for cost\n\nYou can reduce idle costs with two strategies:\n\n**Scale to zero:** Provision a single-instance cluster with automatic scaling policies that scale down to zero during periods of inactivity, eliminating idle compute costs.**Instance-type scheduling:** Use lower-cost instance types during setup and off-peak phases, then transition to higher-performance instances when training workloads begin.\n\n## Conclusion\n\nYou now have a production-ready, event-driven infrastructure for multi-turn reinforcement learning with Amazon Nova Forge on Amazon SageMaker HyperPod. Upload a dataset to Amazon S3, and the pipeline handles provisioning, training orchestration, and reward routing automatically. To adapt this to your use case, replace the Wordle environment with your own API-calling agent or enterprise workflow.\n\nNext steps:\n\n- Clone the sample repository:\n[aws-samples/nova-multi-turn-rl-infra](https://github.com/aws-samples/nova-multi-turn-rl-infra). - Read the Amazon SageMaker HyperPod documentation.\n- Read about Amazon Nova capabilities and pricing.\n- Customize your pipeline with the AWS Step Functions developer guide.", "url": "https://wpnews.pro/news/deploying-multi-turn-rl-infrastructure-for-amazon-nova-on-amazon-sagemaker", "canonical_source": "https://aws.amazon.com/blogs/machine-learning/deploying-multi-turn-rl-infrastructure-for-amazon-nova-on-amazon-sagemaker-hyperpod/", "published_at": "2026-07-06 16:58:13+00:00", "updated_at": "2026-07-07 01:01:21.533390+00:00", "lang": "en", "topics": ["machine-learning", "ai-infrastructure", "ai-agents", "generative-ai"], "entities": ["Amazon Web Services", "Amazon Nova", "Amazon SageMaker HyperPod", "Amazon S3", "ECS on AWS Fargate", "AWS Step Functions", "Amazon EventBridge", "AWS CDK"], "alternates": {"html": "https://wpnews.pro/news/deploying-multi-turn-rl-infrastructure-for-amazon-nova-on-amazon-sagemaker", "markdown": "https://wpnews.pro/news/deploying-multi-turn-rl-infrastructure-for-amazon-nova-on-amazon-sagemaker.md", "text": "https://wpnews.pro/news/deploying-multi-turn-rl-infrastructure-for-amazon-nova-on-amazon-sagemaker.txt", "jsonld": "https://wpnews.pro/news/deploying-multi-turn-rl-infrastructure-for-amazon-nova-on-amazon-sagemaker.jsonld"}}