{"slug": "millwright-self-hosting-your-own-llm-router", "title": "Millwright: Self-Hosting Your Own LLM Router", "summary": "Millwright, a Rust-based router for self-hosting LLM traffic, enables developers to manage model failovers and cost tracking without relying on third-party gateways. The tool supports Anthropic, Amazon Bedrock, and OpenAI-compatible APIs, offering smart routing policies, circuit breakers, and cost visibility via JSON or Markdown reports. Its small footprint allows deployment as a single binary, Docker container, or with SQLite or PostgreSQL backend.", "body_md": "# Millwright: Self-Hosting Your Own LLM Router\n\nRelying on third-party gateways for LLM routing introduces a layer of opacity and potential vendor lock-in that our team wanted to avoid. We've been looking for a way to manage model failovers and cost tracking without sending our traffic through another company's black box, which is why Millwright caught my eye. It's a Rust-based router designed to sit between your AI agents and the actual providers.\n\nThe architecture is lean—basically a single binary—and focuses on reducing the \"tax\" of using frontier models by implementing smart routing policies.\n\n## Core Capabilities\n\n**Provider Support:** Works with Anthropic, Amazon Bedrock, and any OpenAI-compatible API.**Routing Logic:** You can assign models to roles (cheap, mid, frontier) and the router automatically picks the cheapest healthy route.**Reliability Layers:** Includes circuit breakers, bounded failover, and concurrency limits to stop a single hanging request from killing the whole agent workflow.**Cost Visibility:** Tracks spend per team and analyzes model economics via JSON or Markdown reports.**Cache Affinity:** Uses session lanes to manage cache without bottlenecking concurrent traffic.\n\n## Deployment Options\n\nSince it's written in Rust, the footprint is tiny. You can deploy it via:\n\n1. A single binary\n\n2. Docker\n\n3. Backend storage using SQLite or PostgreSQL\n\nFor those of us building internal AI workflows, the biggest win here is the interactive configuration. You can set up providers and pricing without having to hardcode secrets into a complex config file. It effectively turns into a local control plane for your LLM spend and reliability.\n\nIf you are managing multiple agents and want to avoid the unpredictability of hosted routers, this is a solid path for a self-hosted deployment.\n\n[Next My Cybersecurity Career Path: Myths vs Reality →](/en/threads/2303/)\n\n## All Replies （3）\n\nP\n\nThis is super helpful! I've been digging into cache-aware concurrency lately and it's honestly a game changer for performance once you get the hang of it. Thanks for sharing!\n\n0\n\nS\n\nOh look, another on-device router. Just what the world needed! Seriously though, I'm actually building one called role-model—check it out here: https://github.com/try-works/role-model\n\n0\n\nA\n\nI've never actually used an LLM myself, so I'm curious what the point of a router is here. Wouldn't a standard library handle this just fine?\n\n0", "url": "https://wpnews.pro/news/millwright-self-hosting-your-own-llm-router", "canonical_source": "https://promptcube3.com/en/threads/2318/", "published_at": "2026-07-23 13:02:57+00:00", "updated_at": "2026-07-23 21:08:34.624356+00:00", "lang": "en", "topics": ["ai-infrastructure", "developer-tools", "ai-tools"], "entities": ["Millwright", "Anthropic", "Amazon Bedrock", "OpenAI"], "alternates": {"html": "https://wpnews.pro/news/millwright-self-hosting-your-own-llm-router", "markdown": "https://wpnews.pro/news/millwright-self-hosting-your-own-llm-router.md", "text": "https://wpnews.pro/news/millwright-self-hosting-your-own-llm-router.txt", "jsonld": "https://wpnews.pro/news/millwright-self-hosting-your-own-llm-router.jsonld"}}