Route coding agent requests to the right model on Amazon Bedrock with NVIDIA NeMo Switchyard

Artificial Intelligence #

Coding agents such as Claude Code and OpenAI Codex CLI work in tasks. A developer asks for a change, and the agent reads files, edits code, and runs tests until the work is done. One task can take a few model calls or dozens. What the developer pays for is the task, not one call.

The price per token alone doesn’t tell you what a whole task costs. The total depends on how many turns a model needs, how much history each turn replays, and how often a task has to be retried or never finishes. You pay for every attempt, including attempts on tasks that never finish. So when you route a coding agent’s requests between models, judge the routing by cost per solved task: the total cost of your runs divided by the number of runs whose tests pass.

NVIDIA NeMo Switchyard is an open source library that helps an AI agent choose which model handles each request. You define the models that it can use and the strategy that it follows. Its LiteLLM integration supports two strategies. Stage routing reads the recent tool activity and results in the request history, and then picks one of two candidate models. Random routing picks from the configured models, with optional weights, for traffic splits and A/B comparisons. Another option you can explore is LiteLLM, who also recently shipped a native built-in Switchyard stage router as an auto-router strategy.

In this post, you use the AWS Cloud Development Kit (AWS CDK) to deploy LiteLLM, an open source gateway for large language models (LLMs), with the Switchyard routing plugin in your AWS account. LiteLLM calls the models on Amazon Bedrock. You then use the sample’s benchmark runners to measure cost per solved task for a routed model group and for single-model baseline groups, so that you can evaluate routing against your own workloads.

Note: The sample installs switchyard-server 0.3.0 from crates.io and runs the published LiteLLM v1.102.0 image. NVIDIA documents switchyard-server 0.3.0 as a demo and evaluation component, and its configuration can change between releases. This sample uses the experimental Switchyard LiteLLM integration and its configuration and APIs may change between releases. This is sample code, for non-production usage. Work with your security and legal teams to meet your organizational security, regulatory, and compliance requirements before deployment. Deploy the sample in a non-production account and test it before relying on it.

Solution overview #

Figure 1 shows the architecture of the solution. LiteLLM and the Switchyard plugin run in one container image on Amazon Elastic Container Service (Amazon ECS) on AWS Fargate, in private subnets.

A request flows through these steps. The numbers match the diagram.

1. The coding agent sends an OpenAI-compatible request with the model field set to switchyard , the LiteLLM model group that Switchyard routes, and the shared gateway key in theAuthorization header. The endpoint is your hostname over HTTPS, with anAWS Certificate Manager (ACM) certificate on the load balancer, and only the IPv4 CIDR range that you allow can reach it. With a hosted zone inAmazon Route 53 , Route 53 resolves the hostname to the load balancer.
2. AWS WAF filters the request with IP reputation, known bad input, and per-IP rate-limit rules. TheApplication Load Balancer forwards only/v1/chat/completions ,/v1/models , and/health/liveliness to the ECS task in a private subnet, and answers every other path with 404.
3. The task runs one container: LiteLLM with the embedded Switchyard plugin. LiteLLM validates the gateway key and calls the plugin to pick a model. Stage chooses between a capable model and a lower-cost efficient model. It reads the tool activity and results in the request history, so the agent must send the full conversation history. Random chooses from a weighted pool. Neither strategy calls a judge model.
4. LiteLLM calls the selected model through the Amazon Bedrock Converse orConverseStream API, with the task’sAWS Identity and Access Management (IAM) role for credentials. The call goes through an Amazon Bedrock Runtime interface endpoint in theAmazon Virtual Private Cloud (Amazon VPC) , so inference traffic stays on the AWS network.
5. The response or token stream returns through LiteLLM and the load balancer. The x-litellm-model-name response header names the model that answered. The response body’smodel field staysswitchyard .

LiteLLM listens on port 4000 in the container and is the load balancer’s target, and the task’s security group admits only the load balancer. The path allowlist keeps the LiteLLM admin UI, API docs, and key management routes unreachable. Amazon Elastic Container Registry (Amazon ECR) holds the gateway image. AWS Secrets Manager generates the secret behind the gateway key, and the container adds the sk- prefix when it starts. The task definition passes the profile files in LITELLM_CONFIG_TEXT and SWITCHYARD_CONFIG_TEXT, and the startup script writes them to /tmp before it starts LiteLLM. Logs go to one Amazon CloudWatch log group, /<stack name>/litellm, which keeps them for 30 days, and Container Insights collects metrics. The VPC spans two Availability Zones, and one NAT gateway provides outbound access for image pulls and other AWS services.

The plugin supports Stage routing, which needs exactly two candidates, and Random routing, which needs at least two and splits traffic by weight for A/B tests. Classifier-backed strategies, such as Task and Composite routing, need intermediate model calls that this integration doesn’t support. See the Switchyard routing overview for the full catalog.

Prerequisites #

To follow this walkthrough, you need the following:

• A non-production AWS account with Amazon Bedrock access to Claude Opus 5.5 and NVIDIA Nemotron 3 Super 120B. The pair has been tested in US East (N. Virginia). For other AWS Regions, check Regional availability by models and tool-calling support.
• The AWS Command Line Interface (AWS CLI) v2 with configured credentials and an explicit AWS Region, in an account and Region that are bootstrapped for the AWS CDK .
• Node.js 20 or later, npm, Docker with BuildKit, and Python 3.11 or later.
• A hostname that you own, with either a public Route 53 hosted zone for its domain or an ACM certificate for it in the deployment Region. A short plain-HTTP test needs neither.

Walkthrough #

You define the model group, configure Stage, deploy and test the gateway, and measure cost. Start by cloning the sample repository:

Step 1: Define the model group in LiteLLM

A LiteLLM model group is one public model name with one or more deployments behind it. The sample keeps its configuration in deployment profiles under profiles/. Each profile pairs litellm.yaml, which defines the models and gateway settings, with switchyard.toml, which controls how Switchyard picks a model. The following excerpt of profiles/stage/litellm.yaml defines the routed switchyard group, two single-model baseline groups, and the plugin registration:

Both candidates share model_name: switchyard, so your client sends "model": "switchyard" and Switchyard picks a candidate. List the capable model first and the efficient model second, because Stage reads the roles from that order.

The claude-opus-5-5 and nemotron-super-3-120b groups are single-model baselines. Each one always answers with its own model, so you can compare the routed group’s cost and pass rate with each model alone in the same deployment. The Stage plugin alone rejects a group without two candidates, so the image registers a small wrapper, switchyard_gateway.configured.ROUTING_PLUGIN. It applies Switchyard to groups with two or more unique candidates and passes single-model groups through. The configuration registers the wrapper twice: to pick the model and to apply the request changes that Switchyard records.

Every model uses bedrock/converse/<id>, so every candidate goes through the Amazon Bedrock Converse API, and validation rejects any other route. The LiteLLM request_timeout of 600 seconds matches the load balancer’s 10-minute idle timeout.

The file holds no credentials. LiteLLM reads the gateway key from the environment, and the task role supplies the AWS credentials. Never add credentials such as aws_access_key_id, aws_secret_access_key, or api_key under litellm_params: the stack copies the whole litellm.yaml into the task definition and the AWS CloudFormation template in plain text.

The stack derives the task role’s Amazon Bedrock permissions from this file. For a global cross-Region inference profile, the foundation-model resource uses a * Region and a trailing *, so it also matches other model IDs with the same prefix. Review the derived policy before you deploy.

The global. prefix marks a global cross-Region inference profile, and Amazon Bedrock might process those requests in supported commercial AWS Regions worldwide. If processing location matters, use a geographic profile or an in-Region model ID instead. For Nemotron 3 Super, the profile uses the in-Region model ID nvidia.nemotron-super-3-120b, which you can switch for any model for testing.

Step 2: Configure Stage routing

Switchyard reads its algorithm and settings from switchyard.toml. The default profile, profiles/stage/switchyard.toml, configures Stage:

The settings work as follows:

• picker = "efficient_first" : Stage uses the efficient model when the signals in the history are inconclusive. A new request has no tool results yet, so it goes to Nemotron 3 Super.
• confidence_threshold : the confidence cutoff for the execution signals that Stage scores, from 0 through 1.
• recent_window = 3 : Stage examines the three most recent tool results.
• only_on_wrong_signal_escalation = true : the Switchyard default. It controls when Stage adds an optional escalation note, and this profile sets no note.

Stage also accepts optional handoff notes and per-tier system prompts (escalation_note, deescalation_note, capable_system_prompt, and efficient_system_prompt). The sample profile sets none of them.

For an A/B test, the Random profile in profiles/random/ uses the same models with algorithm = "random" and weights = [0.5, 0.5], one weight per unique candidate in declaration order. To deploy it, add -c routingProfile=random to the cdk diff and cdk deploy commands in Step 3. The stack validates both files together when it synthesizes, and the preflight script runs the same checks. To check a profile change on its own, without AWS calls, run npm run validate:profile -- stage. Invalid settings stop LiteLLM when it starts, and unsupported message shapes fail before inference.

Optional: Run the gateway locally

You can try the gateway on your machine before you deploy anything. Docker Compose builds the same image and reads the same profile files that the stack deploys to Amazon ECS. Run from the repository root: