Relying on third-party gateways for LLM routing introduces a layer of opacity and potential vendor lock-in that our team wanted to avoid. We've been looking for a way to manage model failovers and cost tracking without sending our traffic through another company's black box, which is why Millwright caught my eye. It's a Rust-based router designed to sit between your AI agents and the actual providers.
The architecture is lean—basically a single binary—and focuses on reducing the "tax" of using frontier models by implementing smart routing policies.
Core Capabilities #
Provider Support: Works with Anthropic, Amazon Bedrock, and any OpenAI-compatible API.Routing Logic: You can assign models to roles (cheap, mid, frontier) and the router automatically picks the cheapest healthy route.Reliability Layers: Includes circuit breakers, bounded failover, and concurrency limits to stop a single hanging request from killing the whole agent workflow.Cost Visibility: Tracks spend per team and analyzes model economics via JSON or Markdown reports.Cache Affinity: Uses session lanes to manage cache without bottlenecking concurrent traffic.
Deployment Options #
Since it's written in Rust, the footprint is tiny. You can deploy it via:
-
A single binary
-
Docker
-
Backend storage using SQLite or PostgreSQL
For those of us building internal AI workflows, the biggest win here is the interactive configuration. You can set up providers and pricing without having to hardcode secrets into a complex config file. It effectively turns into a local control plane for your LLM spend and reliability.
If you are managing multiple agents and want to avoid the unpredictability of hosted routers, this is a solid path for a self-hosted deployment.
[Next My Cybersecurity Career Path: Myths vs Reality →](/en/threads/2303/)
All Replies （3） #
P
This is super helpful! I've been digging into cache-aware concurrency lately and it's honestly a game changer for performance once you get the hang of it. Thanks for sharing!
0
S
Oh look, another on-device router. Just what the world needed! Seriously though, I'm actually building one called role-model—check it out here: https://github.com/try-works/role-model
0
A
I've never actually used an LLM myself, so I'm curious what the point of a router is here. Wouldn't a standard library handle this just fine?
0