# Route Coding Agents on Bedrock with LiteLLM and NVIDIA NeMo Switchyard

> Source: <https://explainx.ai/blog/aws-bedrock-litellm-nemo-switchyard-coding-agent-routing-2026?utm_source=rss&utm_medium=feed&utm_campaign=blog>
> Published: 2026-11-13 00:00:00+00:00

**AWS published an end-to-end sample on November 13, 2026 that routes coding agent traffic on Amazon Bedrock with NVIDIA NeMo Switchyard inside LiteLLM, then benchmarks cost per solved task instead of token list price.** The post extends the same week’s [Bedrock cost-per-task guide](https://explainx.ai/blog/aws-bedrock-cost-per-task-not-per-token-agent-roi-baselines-holdouts-part-2-2026) with runnable infrastructure: Fargate, private subnets, WAF, and an OpenAI-compatible endpoint your agent already understands.

Primary source: [Route coding agent requests to the right model on Amazon Bedrock with NVIDIA NeMo Switchyard](https://aws.amazon.com/blogs/machine-learning/route-coding-agent-requests-to-the-right-model-on-amazon-bedrock-with-nvidia-nemo-switchyard/?ref=explainx). Sample repo: [aws-samples/sample-route-coding-agent-requests-Amazon-Bedrock-nemo-switchyard](https://github.com/aws-samples/sample-route-coding-agent-requests-Amazon-Bedrock-nemo-switchyard?ref=explainx) (MIT-0, sample code for non-production evaluation).

## TL;DR

| Piece | Detail | 
|---|---|
| Gateway | **LiteLLM** on**Fargate** , port 4000 behind ALB +**ACM** | 
| Router | **NeMo Switchyard** plugin (`switchyard-server`**0.3.0** ) | 
| Models tested | **Claude Opus 5.5** +**Nemotron 3 Super 120B** on Bedrock**Converse** | 
| Routing modes | **Stage** (history-based) or**Random** (weighted A/B) | 
| Client model name | Always `"switchyard"` ; response header`x-litellm-model-name` shows pick | 
| Metric | **Cost per solved task** via included benchmark runners | 
| Guardrails | **allowedCidr** required; WAF rate limit; paths limited to chat API | 

## Why cost per solved task matters here

Coding agents such as Claude Code and OpenAI Codex CLI loop: read files, edit, run tests, retry. [AWS's Part 2 post](https://explainx.ai/blog/aws-bedrock-cost-per-task-not-per-token-agent-roi-baselines-holdouts-part-2-2026) already argued list **price per token** hides verbosity, tokenizer differences, and extra turns. The Switchyard sample makes that argument **measurable**: deploy routing, run the same agent harness against **routed** and **single-model** LiteLLM groups, divide dollars by green tests.

Failed attempts still spend money. Routing that cuts retries can beat a cheaper per-token model — the same lesson as the illustrative 65% savings table in Part 2.

## Architecture AWS describes

1. Agent sends OpenAI-compatible `POST /v1/chat/completions` with`"model": "switchyard"` and a gateway key.
2. **Route 53** resolves your hostname to the ALB (or you CNAME to`LoadBalancerDnsName` with your own cert).
3. **AWS WAF** applies IP reputation, known-bad-inputs, and**2,000 requests / 5 min / IP** before traffic hits the task.
4. The ALB forwards only `/v1/chat/completions` ,`/v1/models` , and`/health/liveliness` ; other paths return**404** to block the LiteLLM admin UI.
5. LiteLLM validates the key, invokes Switchyard, calls **Bedrock Converse or ConverseStream** through a**VPC interface endpoint** .
6. Responses stream back; **`x-litellm-model-name`** exposes which backend answered while JSON` model` stays`switchyard` .

Secrets Manager holds the gateway secret; the task prefixes `sk-` at startup. Profiles live in `profiles/stage/litellm.yaml` and `switchyard.toml`, injected via environment variables at container launch.

Model routing gateway selecting efficient vs capable Bedrock coding models

## Stage routing defaults

From `profiles/stage/switchyard.toml` in the sample:

- `algorithm = "stage"`
- `picker = "efficient_first"` — inconclusive signals stay on the**efficient** candidate (Nemotron for a new task with no tool history)
- `confidence_threshold = 0.5`
- `recent_window = 3` tool results inspected
- `only_on_wrong_signal_escalation = true`

**Stage requires exactly two candidates** in the LiteLLM group, listed **capable first, efficient second** in YAML. The smoke test checks that plain requests pick Nemotron and that a recorded failure history escalates to Opus.

**Random routing** (`routingProfile=random`) uses `algorithm = "random"` with weights for controlled A/B tests.

AWS notes **Task** and **Composite** Switchyard strategies need extra judge calls this LiteLLM integration does not support yet.

## Deploy and test (outline)

Prereqs from AWS: non-prod account with Bedrock access to **global Claude Opus 5.5** and **Nemotron 3 Super 120B** in a tested Region (post uses **us-east-1**), Node 20+, Docker BuildKit, Python 3.11+, CDK bootstrapped.

High-level commands from the blog:

```
git clone https://github.com/aws-samples/sample-route-coding-agent-requests-Amazon-Bedrock-nemo-switchyard.git
cd sample-route-coding-agent-requests-Amazon-Bedrock-nemo-switchyard
npm ci
scripts/preflight.sh
npx cdk deploy -c allowedCidr="$ALLOWED_CIDR" -c domainName="$GATEWAY_DOMAIN" -c hostedZone="$HOSTED_ZONE"
```

Local trial uses `ROUTING_PROFILE=stage docker compose up` — still bills Bedrock.

Point your agent's OpenAI base URL at the gateway, API key at the secret, model at **`switchyard`**. Stage needs **full conversation history** with tool messages on each request.

For enterprise identity patterns, pair this with [Bedrock spend caps per IAM principal](https://explainx.ai/blog/track-and-cap-amazon-bedrock-spend-per-iam-principal-cur-2-0-aws-budgets-deny-policy-guide-2026) and [Codex plus AgentCore Gateway SSO](https://explainx.ai/blog/openai-codex-amazon-bedrock-sso-agentcore-gateway-setup-guide-2026).

## Benchmark tasks AWS ships

The sample repo includes `bench/run.py` for single-task coding loops and `bench/multiturn.py` for longer agentic conversations. AWS documents four starter tasks:

| Task slug | Problem shape | 
|---|---|
| `slugify` | Fix a short URL-slug builder | 
| `roman` | Integer to Roman numerals | 
| `calc` | Expression parser without `eval` , with precedence and functions | 
| `lru_ttl` | Thread-safe LRU cache with TTL and counters | 

A solved task means **unit tests pass** within the default **six model turns** (raise to 12 for `calc`). The runner saves transcripts under `bench/results/` and prints **RUNS, SOLVED, ERRORS, COST/TASK, COST/SOLVED**.

**Cost per solved task** equals estimated spend on all runs in a model group divided by runs that ended green. AWS warns that a group with lower cost per attempt but fewer solves can lose on **cost per solved task** because failed attempts still bill.

Pricing comes from a `PRICES` table inside `bench/run.py`, keyed off `x-litellm-model-name` and token counts. Missing price rows mark runs **unpriced** rather than zero — a honesty detail FinOps teams should copy.

`bench/multiturn.py` adds scripted seven-turn chats mixing simple checks with design, debug, and coding prompts, with `--max-conv-cost` (default **$1**) capping each conversation. That better mimics agents that chatter before editing.

## Security warnings AWS repeats

The blog is explicit about runner risk:

- Model-written Python executes **on your machine** during benchmarks.
- Test subprocesses lack AWS creds in env, but are **not OS sandboxes** .
- Generated code can read files your user can read and use the network.
- Use disposable dev machines without long-lived AWS keys on disk.
- Protect `bench/results/` — they store prompts, completions, and patches.

That mirrors why enterprises route agents through [AgentCore isolated runtimes](https://explainx.ai/blog/aws-bedrock-agentcore-runtime-next-generation-november-2026) instead of laptops.

Gateway hardening in the sample:

- **Secrets Manager** master key with`sk-` prefix at runtime only.
- **WAF** rate limits and reputation lists; body rules omitted so agent payloads are not falsely blocked.
- **ALB path allowlist** blocks LiteLLM admin UI routes.
- **`allowedCidr` required** — stack refuses open CIDR synthesis.

AWS nudges teams toward **HTTPS only**; plain HTTP opt-in logs a warning every synth because keys and source cross the wire unencrypted.

## IAM and residency details

The sample derives Bedrock invoke policies from `litellm.yaml` model strings. Global inference profiles use wildcard regions; AWS notes data may process outside your home Region. Swap to geographic profiles if contracts require it.

Interface VPC endpoints keep **inference traffic** off the public internet even when the ALB faces your office IP.

## How to read benchmark output without fooling yourself

AWS lists interpretation rules worth quoting for any agent eval:

1. Compare **solve counts** before celebrating low**cost per task** .
2. Break out **hard tasks** separately — Stage only escalates after failure signals, so it may burn cash replaying failed histories.
3. Increase **`--repeats`** until solve-rate gaps stabilize; consider Fisher's exact test on small samples.
4. Match **`--max-turns`** to production agent limits.
5. Watch **`max_tokens` truncation** — truncated replies cheapen measured cost but poison later turns.
6. Re-benchmark when you change **candidates, thresholds, or profiles** .

Those bullets are vendor advice, but they are also the difference between a demo win and a production routing policy.

## Switchyard beyond this sample

We first explained Switchyard in [NVIDIA's eight-node DGX Spark cluster post](https://explainx.ai/blog/nvidia-dgx-spark-eight-node-cluster-1tb-glm-5-3-753b-roce-decode-4-43x-switchyard-2026), where a classifier routed between MiniCPM and Qwen replicas. The AWS sample moves the same library **into enterprise Bedrock** behind LiteLLM — closer to how most teams already centralize model keys.

LiteLLM also ships its own Switchyard stage router, which AWS mentions as an alternative auto-router strategy. Evaluate both if you already run LiteLLM without CDK.

## Safety and compliance notes

AWS repeats: **sample only**, work with security and legal, deploy in a **non-production account**, and treat `switchyard-server 0.3.0` as **demo/eval** quality. Prompts, tool output, and source code cross the gateway; `allowedPlainHttp=true` is for short tests only.

Global inference profiles may process outside your home Region. If data residency matters, swap model IDs per AWS guidance in the post.

## Related reading

## When to pick Random routing instead of Stage

AWS's **Random** profile exists for **A/B tests**, not production savings. Use it when you need a controlled 50/50 split (or other weights) to compare solve rates without history-based escalation. Stage fits **production agent loops** where early turns are cheap reads and late turns follow tool failures.

If your agent emits sparse tool history — for example, single-shot refactors — Stage may never escalate because `only_on_wrong_signal_escalation` waits for failure signals. In that regime, measure whether you should start on the capable model or adjust `confidence_threshold` downward with care.

## Cost accounting tie-in for platform teams

Pair Switchyard measurements with [CUR 2.0 Bedrock tagging](https://explainx.ai/blog/track-and-cap-amazon-bedrock-spend-per-iam-principal-cur-2-0-aws-budgets-deny-policy-guide-2026) so gateway spend rolls up per team. Tag the CloudFormation stack, map `x-litellm-model-name` values to cost centers, and reconcile bench estimates weekly until `PRICES` matches your negotiated rates.

For coding agents already on **Codex plus Bedrock**, the [SSO AgentCore gateway guide](https://explainx.ai/blog/openai-codex-amazon-bedrock-sso-agentcore-gateway-setup-guide-2026) shows identity patterns; Switchyard sits **downstream** of identity, at the model pick layer.

## Cleanup and ongoing operations

AWS ends the walkthrough with `cdk destroy`, noting the master key secret is deleted without recovery and the bootstrap ECR image may remain. Keep CloudWatch log retention in mind if compliance requires longer inference audit trails than the sample's default 30-day log group retention. Rotate gateway keys after any plain-HTTP trial. If you change `profiles/stage/switchyard.toml`, rerun `npm run validate:profile` before deploy so LiteLLM does not fail at container start — a failure mode that is easy to miss when only the CDK stack changes.

*AWS blog steps, version pins, and repository layout are accurate as of November 13, 2026.*