{"slug": "meet-switchyard-a-rust-proxy-and-library-that-routes-and-translates-llm-traffic", "title": "Meet Switchyard: A Rust Proxy and Library That Routes and Translates LLM Traffic Across OpenAI and Anthropic APIs", "summary": "NVIDIA released Switchyard, a Rust proxy and library that routes and translates LLM traffic across OpenAI and Anthropic APIs, under Apache 2.0, but labels it pre-alpha and experimental, not for production use. The tool supports three inbound formats (OpenAI Chat Completions, OpenAI Responses, Anthropic Messages) and offers routing algorithms including passthrough, random, llm_classifier, and stage_router, with Prometheus metrics via OpenTelemetry.", "body_md": "Teams running coding agents hit the same wall. Claude Code speaks the Anthropic Messages API, Codex CLI speaks OpenAI, and the model a team actually wants to serve sits behind vLLM, [NVIDIA NIM](https://developer.nvidia.com/nim), or Ollama. Rewriting the agent is not an option, so the translation layer has to live somewhere else.\n\n[Switchyard](https://github.com/NVIDIA-NeMo/Switchyard) is NVIDIA’s answer: a Rust proxy and library for LLM traffic that routes requests across providers, translates between OpenAI and Anthropic formats, records operational metrics, and exposes typed, composable routing algorithms. It is released under [Apache 2.0](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/LICENSE) with documentation at [docs.nvidia.com/nemo/switchyard](https://docs.nvidia.com/nemo/switchyard/home).\n\n**Is it deployable?** Yes, but for evaluation only. The binary installs from crates.io and the launcher from PyPI, and it self-hosts anywhere, but NVIDIA labels Switchyard [pre-alpha and experimental](https://github.com/NVIDIA-NeMo/Switchyard#maturity), warns it is not for production use, and expects the API and algorithms to change significantly before v1.0.\n\n**What Switchyard does**\n\nClients keep their native API. Switchyard decodes the inbound request into provider-neutral Rust types, runs a routing algorithm to pick a backend, re-encodes the request in that backend’s own wire format, calls it, and translates the response, including streaming events, back into the shape the client expects.\n\nThe [server](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/crates/switchyard-server/README.md) accepts three inbound formats: OpenAI Chat Completions, OpenAI Responses, and Anthropic Messages. Any of the three can address any route, and each configured LLM client selects one upstream format of its own. That decoupling is the point: the agent’s API and the backend’s API no longer have to match.\n\n**Three ways to run it**\n\nThe **launcher path** targets coding agents. Install the published tool with `uv tool install --python 3.12 \"nemo-switchyard[cli]\"`\n\n, then run `switchyard launch claude`\n\n, `switchyard launch codex`\n\n, or `switchyard launch openclaw`\n\nagainst a packaged deployment or your own TOML file.\n\nThe **server path** installs the standalone proxy with `cargo install --locked switchyard-server`\n\n, validates a config with `--dry-run`\n\n, and serves on a host and port you choose.\n\nThe **library path** uses `switchyard-libsy`\n\n, which embeds the routing algorithms in a Rust application without owning an HTTP stack. It never calls a model itself; the algorithm decides which target to use and hands every model call back to the caller.\n\n**Routing algorithms**\n\nA [route](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/docs/routing_algorithms/overview.md) is one client-visible model ID plus the algorithm behind it. **The server supports:**\n\nsends every request to one target.`passthrough`\n\nsplits traffic across targets using optional relative weights, with an optional seed that reproduces the selection sequence. This is the A/B and cost-experiment path.`random`\n\ncalls a classifier target for a capability verdict, then routes to a weak or strong target.`llm_classifier`\n\n`base_threshold`\n\nis required;`min_confidence`\n\n,`capability_elevated_floor`\n\n, and`session_affinity`\n\ntune it, and anything the judge cannot decide falls through to the strong target. Setting`mode = \"escalation\"`\n\nruns every turn on the weak tier first and lets a judge decide whether to rerun it on the strong tier.scores tool-result and agent-progress signals from recent turns to pick a capable or efficient target, avoiding an extra classifier call on most turns.`stage_router`\n\nStrong, weak, capable, and efficient are roles inside a route, not fixed properties of a model. The same upstream model can serve different roles in different routes.\n\n**Observability**\n\n`GET /metrics`\n\nreturns Prometheus text from the server’s process-wide OpenTelemetry provider. The families cover requests, errors, model-call latency, full-turn latency, prompt, completion, cached, cache-creation, and reasoning tokens, and upstream HTTP attempts by outcome and code. A `tier`\n\nlabel carries `strong`\n\nor `weak`\n\nfor distinguishable classifier decisions, and classifier calls are excluded from those families.\n\nThe more interesting metric is `switchyard_routing_overhead_ms`\n\n, which reports the algorithm’s run time minus the call that served the request. Classifier calls are not subtracted, so an LLM-classifier route reports its classification time here while `passthrough`\n\nand `random`\n\nreport the sub-millisecond cost of picking a target. Buckets start at 0.1 ms. Separately, `--routing-log-file`\n\nappends a JSON record per completed response, and `GET /v1/routing/session-stats`\n\nreturns per-session call and token totals from that log.\n\n**Configuration**\n\nA TOML deployment has three layers: `llm_clients`\n\ndefine base URL, wire format, credential environment variable, and retry policy; `targets`\n\nbind one upstream model ID to a client; `routes`\n\nexpose one client-visible model ID and its algorithm. Secrets never sit in the file, since `api_key_env`\n\nonly names an environment variable. `max_retries`\n\ndefaults to 2 and applies to transport failures, timeouts, HTTP 408/429, and 5xx responses.\n\n**Key Takeaways**\n\n- Switchyard is an Apache-2.0 Rust proxy and library that routes and translates LLM traffic.\n- It bridges OpenAI Chat, OpenAI Responses, and Anthropic Messages in both directions, including streams.\n- Four route types ship: passthrough, random, LLM-classifier, and signal-driven stage router.\n- Prometheus metrics isolate routing overhead from model-call latency, per model and tier.\n- It is pre-alpha and explicitly not for production, so treat it as an evaluation tool.\n\nCheck out the [ GitHub Repo](https://github.com/NVIDIA-NeMo/Switchyard) and\n\n[. Also, feel free to follow us on](https://docs.nvidia.com/nemo/switchyard/home)\n\n**Documentation****and don’t forget to join our**[Twitter](https://x.com/intent/follow?screen_name=marktechpost)\n\n**and Subscribe to**\n\n[150k+ML SubReddit](https://www.reddit.com/r/machinelearningnews/)**. Wait! are you on telegram?**\n\n[our Newsletter](https://magic.beehiiv.com/v1/f5e63dd4-5653-4f09-83e2-321a8b1ba526?email={{email}})\n\n[now you can join us on telegram as well.](https://t.me/machinelearningresearchnews)Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? [Connect with us](https://forms.gle/wbash1wF6efRj8G58)\n\nMichal Sutter is a data science professional with a Master of Science in Data Science from the University of Padova. With a solid foundation in statistical analysis, machine learning, and data engineering, Michal excels at transforming complex datasets into actionable insights.", "url": "https://wpnews.pro/news/meet-switchyard-a-rust-proxy-and-library-that-routes-and-translates-llm-traffic", "canonical_source": "https://www.marktechpost.com/2026/09/02/nvidia-releases-switchyard-rust-proxy-llm-traffic-openai-anthropic-api-translation/", "published_at": "2026-09-02 18:05:52+00:00", "updated_at": "2026-09-02 18:25:24.821657+00:00", "lang": "en", "topics": ["ai-infrastructure", "developer-tools", "ai-tools"], "entities": ["NVIDIA", "Switchyard", "OpenAI", "Anthropic", "Apache 2.0", "Claude Code", "Codex CLI", "NVIDIA NIM"], "alternates": {"html": "https://wpnews.pro/news/meet-switchyard-a-rust-proxy-and-library-that-routes-and-translates-llm-traffic", "markdown": "https://wpnews.pro/news/meet-switchyard-a-rust-proxy-and-library-that-routes-and-translates-llm-traffic.md", "text": "https://wpnews.pro/news/meet-switchyard-a-rust-proxy-and-library-that-routes-and-translates-llm-traffic.txt", "jsonld": "https://wpnews.pro/news/meet-switchyard-a-rust-proxy-and-library-that-routes-and-translates-llm-traffic.jsonld"}}