Route Coding Agents on Bedrock with LiteLLM and NVIDIA NeMo Switchyard

AWS published an end-to-end sample on November 13, 2026 that routes coding agent traffic on Amazon Bedrock through NVIDIA NeMo Switchyard inside LiteLLM and benchmarks cost per solved task rather than token list price. The sample, released as MIT-0 code in the aws-samples repository, runs LiteLLM on Fargate behind an ALB with AWS WAF rate limiting at 2,000 requests per 5 minutes per IP and uses the switchyard-server 0.3.0 plugin to route between Claude Opus 5.5 and Nemotron 3 Super 120B on Bedrock Converse. Stage routing defaults to algorithm "stage" with picker "efficient_first", a 0.5 confidence threshold, a 3-result recent window, and only_on_wrong_signal_escalation enabled, requiring exactly two candidates listed capable first and efficient second.

AWS published an end-to-end sample on November 13, 2026 that routes coding agent traffic on Amazon Bedrock with NVIDIA NeMo Switchyard inside LiteLLM, then benchmarks cost per solved task instead of token list price. The post extends the same week’s Bedrock cost-per-task guide https://explainx.ai/blog/aws-bedrock-cost-per-task-not-per-token-agent-roi-baselines-holdouts-part-2-2026 with runnable infrastructure: Fargate, private subnets, WAF, and an OpenAI-compatible endpoint your agent already understands. Primary source: Route coding agent requests to the right model on Amazon Bedrock with NVIDIA NeMo Switchyard https://aws.amazon.com/blogs/machine-learning/route-coding-agent-requests-to-the-right-model-on-amazon-bedrock-with-nvidia-nemo-switchyard/?ref=explainx . Sample repo: aws-samples/sample-route-coding-agent-requests-Amazon-Bedrock-nemo-switchyard https://github.com/aws-samples/sample-route-coding-agent-requests-Amazon-Bedrock-nemo-switchyard?ref=explainx MIT-0, sample code for non-production evaluation . TL;DR | Piece | Detail | |---|---| | Gateway | LiteLLM on Fargate , port 4000 behind ALB + ACM | | Router | NeMo Switchyard plugin switchyard-server 0.3.0 | | Models tested | Claude Opus 5.5 + Nemotron 3 Super 120B on Bedrock Converse | | Routing modes | Stage history-based or Random weighted A/B | | Client model name | Always "switchyard" ; response header x-litellm-model-name shows pick | | Metric | Cost per solved task via included benchmark runners | | Guardrails | allowedCidr required; WAF rate limit; paths limited to chat API | Why cost per solved task matters here Coding agents such as Claude Code and OpenAI Codex CLI loop: read files, edit, run tests, retry. AWS's Part 2 post https://explainx.ai/blog/aws-bedrock-cost-per-task-not-per-token-agent-roi-baselines-holdouts-part-2-2026 already argued list price per token hides verbosity, tokenizer differences, and extra turns. The Switchyard sample makes that argument measurable : deploy routing, run the same agent harness against routed and single-model LiteLLM groups, divide dollars by green tests. Failed attempts still spend money. Routing that cuts retries can beat a cheaper per-token model — the same lesson as the illustrative 65% savings table in Part 2. Architecture AWS describes 1. Agent sends OpenAI-compatible POST /v1/chat/completions with "model": "switchyard" and a gateway key. 2. Route 53 resolves your hostname to the ALB or you CNAME to LoadBalancerDnsName with your own cert . 3. AWS WAF applies IP reputation, known-bad-inputs, and 2,000 requests / 5 min / IP before traffic hits the task. 4. The ALB forwards only /v1/chat/completions , /v1/models , and /health/liveliness ; other paths return 404 to block the LiteLLM admin UI. 5. LiteLLM validates the key, invokes Switchyard, calls Bedrock Converse or ConverseStream through a VPC interface endpoint . 6. Responses stream back; x-litellm-model-name exposes which backend answered while JSON model stays switchyard . Secrets Manager holds the gateway secret; the task prefixes sk- at startup. Profiles live in profiles/stage/litellm.yaml and switchyard.toml , injected via environment variables at container launch. Model routing gateway selecting efficient vs capable Bedrock coding models Stage routing defaults From profiles/stage/switchyard.toml in the sample: - algorithm = "stage" - picker = "efficient first" — inconclusive signals stay on the efficient candidate Nemotron for a new task with no tool history - confidence threshold = 0.5 - recent window = 3 tool results inspected - only on wrong signal escalation = true Stage requires exactly two candidates in the LiteLLM group, listed capable first, efficient second in YAML. The smoke test checks that plain requests pick Nemotron and that a recorded failure history escalates to Opus. Random routing routingProfile=random uses algorithm = "random" with weights for controlled A/B tests. AWS notes Task and Composite Switchyard strategies need extra judge calls this LiteLLM integration does not support yet. Deploy and test outline Prereqs from AWS: non-prod account with Bedrock access to global Claude Opus 5.5 and Nemotron 3 Super 120B in a tested Region post uses us-east-1 , Node 20+, Docker BuildKit, Python 3.11+, CDK bootstrapped. High-level commands from the blog: git clone https://github.com/aws-samples/sample-route-coding-agent-requests-Amazon-Bedrock-nemo-switchyard.git cd sample-route-coding-agent-requests-Amazon-Bedrock-nemo-switchyard npm ci scripts/preflight.sh npx cdk deploy -c allowedCidr="$ALLOWED CIDR" -c domainName="$GATEWAY DOMAIN" -c hostedZone="$HOSTED ZONE" Local trial uses ROUTING PROFILE=stage docker compose up — still bills Bedrock. Point your agent's OpenAI base URL at the gateway, API key at the secret, model at switchyard . Stage needs full conversation history with tool messages on each request. For enterprise identity patterns, pair this with Bedrock spend caps per IAM principal https://explainx.ai/blog/track-and-cap-amazon-bedrock-spend-per-iam-principal-cur-2-0-aws-budgets-deny-policy-guide-2026 and Codex plus AgentCore Gateway SSO https://explainx.ai/blog/openai-codex-amazon-bedrock-sso-agentcore-gateway-setup-guide-2026 . Benchmark tasks AWS ships The sample repo includes bench/run.py for single-task coding loops and bench/multiturn.py for longer agentic conversations. AWS documents four starter tasks: | Task slug | Problem shape | |---|---| | slugify | Fix a short URL-slug builder | | roman | Integer to Roman numerals | | calc | Expression parser without eval , with precedence and functions | | lru ttl | Thread-safe LRU cache with TTL and counters | A solved task means unit tests pass within the default six model turns raise to 12 for calc . The runner saves transcripts under bench/results/ and prints RUNS, SOLVED, ERRORS, COST/TASK, COST/SOLVED . Cost per solved task equals estimated spend on all runs in a model group divided by runs that ended green. AWS warns that a group with lower cost per attempt but fewer solves can lose on cost per solved task because failed attempts still bill. Pricing comes from a PRICES table inside bench/run.py , keyed off x-litellm-model-name and token counts. Missing price rows mark runs unpriced rather than zero — a honesty detail FinOps teams should copy. bench/multiturn.py adds scripted seven-turn chats mixing simple checks with design, debug, and coding prompts, with --max-conv-cost default $1 capping each conversation. That better mimics agents that chatter before editing. Security warnings AWS repeats The blog is explicit about runner risk: - Model-written Python executes on your machine during benchmarks. - Test subprocesses lack AWS creds in env, but are not OS sandboxes . - Generated code can read files your user can read and use the network. - Use disposable dev machines without long-lived AWS keys on disk. - Protect bench/results/ — they store prompts, completions, and patches. That mirrors why enterprises route agents through AgentCore isolated runtimes https://explainx.ai/blog/aws-bedrock-agentcore-runtime-next-generation-november-2026 instead of laptops. Gateway hardening in the sample: - Secrets Manager master key with sk- prefix at runtime only. - WAF rate limits and reputation lists; body rules omitted so agent payloads are not falsely blocked. - ALB path allowlist blocks LiteLLM admin UI routes. - allowedCidr required — stack refuses open CIDR synthesis. AWS nudges teams toward HTTPS only ; plain HTTP opt-in logs a warning every synth because keys and source cross the wire unencrypted. IAM and residency details The sample derives Bedrock invoke policies from litellm.yaml model strings. Global inference profiles use wildcard regions; AWS notes data may process outside your home Region. Swap to geographic profiles if contracts require it. Interface VPC endpoints keep inference traffic off the public internet even when the ALB faces your office IP. How to read benchmark output without fooling yourself AWS lists interpretation rules worth quoting for any agent eval: 1. Compare solve counts before celebrating low cost per task . 2. Break out hard tasks separately — Stage only escalates after failure signals, so it may burn cash replaying failed histories. 3. Increase --repeats until solve-rate gaps stabilize; consider Fisher's exact test on small samples. 4. Match --max-turns to production agent limits. 5. Watch max tokens truncation — truncated replies cheapen measured cost but poison later turns. 6. Re-benchmark when you change candidates, thresholds, or profiles . Those bullets are vendor advice, but they are also the difference between a demo win and a production routing policy. Switchyard beyond this sample We first explained Switchyard in NVIDIA's eight-node DGX Spark cluster post https://explainx.ai/blog/nvidia-dgx-spark-eight-node-cluster-1tb-glm-5-3-753b-roce-decode-4-43x-switchyard-2026 , where a classifier routed between MiniCPM and Qwen replicas. The AWS sample moves the same library into enterprise Bedrock behind LiteLLM — closer to how most teams already centralize model keys. LiteLLM also ships its own Switchyard stage router, which AWS mentions as an alternative auto-router strategy. Evaluate both if you already run LiteLLM without CDK. Safety and compliance notes AWS repeats: sample only , work with security and legal, deploy in a non-production account , and treat switchyard-server 0.3.0 as demo/eval quality. Prompts, tool output, and source code cross the gateway; allowedPlainHttp=true is for short tests only. Global inference profiles may process outside your home Region. If data residency matters, swap model IDs per AWS guidance in the post. Related reading When to pick Random routing instead of Stage AWS's Random profile exists for A/B tests , not production savings. Use it when you need a controlled 50/50 split or other weights to compare solve rates without history-based escalation. Stage fits production agent loops where early turns are cheap reads and late turns follow tool failures. If your agent emits sparse tool history — for example, single-shot refactors — Stage may never escalate because only on wrong signal escalation waits for failure signals. In that regime, measure whether you should start on the capable model or adjust confidence threshold downward with care. Cost accounting tie-in for platform teams Pair Switchyard measurements with CUR 2.0 Bedrock tagging https://explainx.ai/blog/track-and-cap-amazon-bedrock-spend-per-iam-principal-cur-2-0-aws-budgets-deny-policy-guide-2026 so gateway spend rolls up per team. Tag the CloudFormation stack, map x-litellm-model-name values to cost centers, and reconcile bench estimates weekly until PRICES matches your negotiated rates. For coding agents already on Codex plus Bedrock , the SSO AgentCore gateway guide https://explainx.ai/blog/openai-codex-amazon-bedrock-sso-agentcore-gateway-setup-guide-2026 shows identity patterns; Switchyard sits downstream of identity, at the model pick layer. Cleanup and ongoing operations AWS ends the walkthrough with cdk destroy , noting the master key secret is deleted without recovery and the bootstrap ECR image may remain. Keep CloudWatch log retention in mind if compliance requires longer inference audit trails than the sample's default 30-day log group retention. Rotate gateway keys after any plain-HTTP trial. If you change profiles/stage/switchyard.toml , rerun npm run validate:profile before deploy so LiteLLM does not fail at container start — a failure mode that is easy to miss when only the CDK stack changes. AWS blog steps, version pins, and repository layout are accurate as of November 13, 2026.