LLMRix Model Router — an open-source multi-model routing and orchestration framework for Java. LLMRix Inc. has released LLMRix Model Router, an open-source multi-model routing and orchestration framework for Java that manages provider differences, model selection, failover, quotas, costs, and observability. The framework sits between AI applications and model services, enabling dynamic routing decisions and request forwarding while keeping business code decoupled from specific providers. When AI applications move from demo to production, the trouble usually isn't with prompts—it's with the model calls themselves. Which model should handle this request? What happens when the primary model is rate-limited? How do you control costs? Can you switch providers mid-stream when a streaming response drops? And when running multiple instances, how do you keep quota and health state consistent? LLMRix Model Router packages all these concerns into an open-source routing runtime for Java. Over the past couple of years, the barrier to integrating AI models has dropped significantly. With a single SDK and a few lines of code, an app can call OpenAI, DeepSeek, or any other compatible interface. But once you enter production, a different set of problems emerges: upstream rate limits, timeouts, regional outages, and models that differ widely in capabilities, pricing, context windows, and response speed. When business code is directly coupled to a specific provider, swapping models means rewriting interfaces, exception handling, monitoring, and configuration all at once. Simple round-robin or reverse proxies can't make these decisions either. They don't know whether a request needs tool calling, which models to exclude for image inputs, or whether a candidate has already exceeded its cost budget or is in a cooldown period. LLMRix Model Router https://github.com/llmrix-inc/llmrix-router sits at this boundary. It's a multi-model routing and orchestration framework for Java that extracts provider differences, model selection, failover, quotas, costs, and observability out of business code. Positioned between AI applications and model services, it handles runtime decision-making and request forwarding. A chat SDK answers how to call; a Router also decides who to call, when to retry, and how to reconcile state. The project's architectural tradeoff is clear: routing strategies can change, but the correctness of request execution must not . To achieve this, LLMRix divides the system into layers with well-defined boundaries: llmrix-model-open . llmrix-model-router-core handles model targets, capability matching, strategy selection, execution budgets, timeouts, retries, health, and lifecycle events. llmrix-model-router-integrations provides adapters for OpenAI, DeepSeek, OpenRouter, Ollama, Redis, Bucket4j, ONNX, Shadow, evaluation, and Fugu.Vector source: llmrix-router-architecture.svg https://raw.githubusercontent.com/llmrix-inc/llmrix-router/main/docs/images/llmrix-router-architecture.svg . Original panoramic diagram: GitHub architecture SVG https://github.com/llmrix-inc/llmrix-router/blob/main/docs/images/llmrix-architecture.svg . The Router Core in the middle sits at the junction of decision and execution. It doesn't bind to any single vendor, nor does it rewrite business requests into provider-private objects—instead, it manages candidates through a unified ModelClient and ModelTarget . When adding a new provider, changes stay concentrated in the Provider SPI and Transport adapters, without spreading to routing strategies or business code. | Maven Artifact | Responsibility | Typical Usage | |---|---|---| llmrix-model-open | Shared model contracts, common exceptions, auth SPI, OpenAI-compatible transport & adapters | Reuse unified types when building clients or custom integrations | llmrix-model-router-core | Router Builder, model targets, strategies, executor, state SPI, quota, health & events | Embed Router in plain Java applications | llmrix-model-router-integrations | Built-in providers, Redis, Bucket4j, ONNX, Shadow, evaluation & Fugu | Use official integrations or advanced routing capabilities | llmrix-model-router-spring-starter | Auto-configuration, YAML properties, HTTP/SSE, auth, Actuator & Micrometer | Build Spring Boot routing services | llmrix-model-orion | Framework-neutral remote Java client | Call a standalone Router from Java services | llmrix-model-orion-spring-starter | Orion auto-configuration & Micrometer adapter | Inject remote model clients in Spring Boot business services | These module boundaries let teams pull in only what they need. Remote business services can depend solely on Orion and the shared model contracts, without dragging in Redis, ONNX, or the entire Router Runtime. The sequence diagram below follows the source code's execution order, covering capability matching, quota control, and failover. It highlights three distinct paths: successful return, retry before the first visible output, and no-replay after streaming has begun. Vector source: llmrix-router-request-decision-sequence.svg https://raw.githubusercontent.com/llmrix-inc/llmrix-router/main/docs/images/llmrix-router-request-decision-sequence.svg . "Whether a model is suitable for this request" and "whether this call will succeed" are handled by two separate components: candidate snapshots and strategies handle the former, while the executor, state store, and exception classification handle the latter. | Capability Domain | Specific Features | Problems Solved | |---|---|---| | Model abstraction | Provider-neutral ModelClient , multimodal request/response, unified exceptions | Business code no longer binds to vendor SDKs | | Capability matching | operations , features , input-modalities , traits | Prevents sending tool, image, or audio requests to unsupported models | | Routing strategies | priority, round-robin, weighted random, least-busy, latency-aware, cost-aware, balanced, cache-aware | Trade off stability, cost, latency, and cache hits according to business goals | | Dynamic decisions | semantic scoring, contextual bandit, customizable RoutingStrategy | Continuously improve model selection using request semantics or feedback data | | Reliability | per-attempt timeout, total budget, retry predicates, failure thresholds, target cooldown, candidate pool continuation | Handles rate limits, timeouts, transient failures, and partial outages | | Streaming safety | first-token timeout, stream idle timeout, pre-first-chunk switching, cancellation propagation, tool request non-replay | Prevents duplicate output and replay of side-effecting tool calls | | Cost governance | input/output/cache/inference token pricing, per-request maxCostUsd , route-level RPM/TPM | Makes cost constraints part of the decision, not a post-hoc statistic | | Quota & concurrency | target-level limits, route-level limits, auth quota partitions, local Caffeine, Redis atomic leases | Controls resource usage per model, per route, and per tenant | | Multimodal | Chat, Responses core subset, Embeddings, Rerank, Audio, Image, Video | Unified handling of text, image, audio, file, and video workflows | | Evaluation & orchestration | Online Shadow, offline Evaluation, Fugu Worker/Thinker/Verifier, ONNX policies | Compare models and organize multi-round collaboration without affecting main traffic | | Observability | Router/Fugu Listener, Micrometer, Observation, Actuator, request ID | Answers "who was selected, how long it took, why it retried, and how much it cost" | | Extensibility | ModelProvider , ProviderAuthenticator , ModelPricingResolver , RouterStateStore | Integrate enterprise proxies, signed auth, internal pricing catalogs, and custom state systems | LLMRix doesn't reduce "model capability" to a single boolean. Instead, configuration declares four separate dimensions: operations : what the model can do, e.g. chat , embeddings , rerank , video-generation ; features : what protocol features it supports, e.g. streaming , tools , structured-output , prompt-cache ; input-modalities : what inputs it accepts, e.g. vision , video , audio , file ; traits : what task types it excels at, e.g. code , reasoning , long-context .The four declarations are independent. They're validated at startup and used to filter candidates per request. Compared to maintaining a single "universal model list," this approach is easier to audit and reduces capability mismatches in production. LLMRix lets applications depend on stable route names rather than upstream model names. For example, business code simply requests general , code , or reasoning . Whether the request ultimately goes to OpenAI, DeepSeek, OpenRouter, or a local Ollama instance is decided by the router based on capability, health, latency, cost, and strategy. Model adjustments therefore live in routing configuration, not in business code. When adding providers, replacing models, or changing selection strategies, upstream applications typically don't need to be rewritten. LLMRix offers three integration modes: When a Java service needs to call a remote Router, it can use the project's lightweight client, Orion. Provider keys stay on the Router side only—clients only see route names and the unified protocol. LLMRix doesn't just work off a flat list of models—it processes requests through a well-bounded execution pipeline. Upon receiving a request, the Router first checks the operation type, tool calling, structured output, input modality, context length, and routing constraints passed by the caller. Models that can't satisfy the conditions are eliminated before execution begins. A chat with images won't be sent to a text-only model; when a request requires reasoning or code traits, candidates that don't declare those traits won't enter the selection phase. The remaining candidates are then passed to a routing strategy for ordering. Built-in strategies include priority, round-robin, weighted random, least-busy, latency-aware, cost-aware, balanced scoring, and prompt-cache affinity. Semantic routing and contextual multi-armed bandit implementations are also provided. Teams can implement RoutingStrategy to incorporate data residency, tenant tier, compliance tags, or internal model scores into selection rules. Before invoking a model, the Router acquires quota and concurrency leases and checks the per-request cost budget. On success, it settles costs based on actual token usage, releases leases, and publishes lifecycle events. On failure, the Router determines—based on exception type—whether to retry, put the target into cooldown, and move on to the next candidate in the pool. Thus, "who to pick" and "how to call" are two separate responsibilities: strategies handle ordering, while the executor handles timeouts, retries, quotas, health, and resource cleanup. Selection algorithms can be swapped; execution rules remain consistent. Failover for ordinary requests is relatively straightforward. Streaming responses are different. Suppose Model A has already output half a sentence to the user when the connection drops. If the system switches to Model B and replays from the beginning, the user sees duplicate content, and tool calls may execute twice. LLMRix's rule is: candidates can be switched before the first data chunk becomes visible to the caller; after output begins, the request is never replayed. The Router manages first-token timeout and stream idle timeout separately, and propagates cancellation signals to any still-running upstream requests. This rule directly affects whether callers see consistent output. It encodes the streaming response boundary into the executor, rather than leaving it to business code to handle on its own. As of version 1.0.2 , LLMRix supports Chat, a core subset of the Responses API, Embeddings, Rerank, audio, image, and video operations. The Spring Boot Starter provides corresponding OpenAI-style endpoints. Both Chat and Responses endpoints support JSON and SSE streaming. Built-in integrations include OpenAI, DeepSeek, OpenRouter, and Ollama. Not every provider supports the same set of operations, and the Router reads each target's operations , features , input-modalities , and traits —it never treats "OpenAI-compatible interfaces" as having identical capabilities. Explicit capability declaration has two practical benefits: configuration errors surface at startup, and routing strategies can filter before requests are sent upstream. | Integration | Key Operations Implemented | Suitable For | |---|---|---| | OpenAI | Chat, Embeddings, Audio, Images, Videos | Using OpenAI's native multimodal interface | | DeepSeek | Chat | General conversation, code, and reasoning routes | | OpenRouter | Chat, Embeddings, Rerank | Accessing multiple models and free models through a single upstream | | Ollama | Chat, Embeddings | Local development, offline environments, and private models | | Custom Provider | Determined by the ModelProvider implementation | Enterprise model platforms, internal proxies, proprietary protocols | "Integration support" doesn't mean all models under that provider support the same operations. Final capabilities depend on the specific model, provider account, and configuration declarations. For example, a particular OpenRouter model might support text chat but not accept image inputs. The Router only filters by target declarations—it doesn't invent capabilities a model doesn't have. | Endpoint | Purpose | |---|---| POST /v1/chat/completions | Synchronous or SSE streaming chat | POST /v1/responses | Core subset of Responses API, with JSON and SSE | POST /v1/embeddings | Text or token-array embeddings | POST /v1/rerank | Query/document reranking | POST /v1/audio/transcriptions | Audio transcription | POST /v1/audio/translations | Audio translation | POST /v1/audio/speech | Text-to-speech | POST /v1/images/generations | Image generation | POST /v1/images/edits | Multipart image editing | POST /v1/videos | Create a video generation task | GET /v1/videos/{video id} | Check video task status | GET /v1/videos/{video id}/content | Download completed video | DELETE /v1/videos/{video id} | Delete a video task | POST /v1/videos/{video id}/remix | Create a remix task from an existing video | GET /v1/models | List available route identifiers | For development and small deployments, you can use in-memory state with no extra infrastructure. Local quota partitions are managed by Caffeine with capacity and idle expiry, preventing unbounded growth from dynamic tenant keys. When the Router scales to multiple instances, you can switch to Redis. Health state, concurrency leases, RPM, TPM, and multi-armed bandit state can all be shared across instances, with updates performed via Redis atomic operations. Redis mode uses a fail-closed strategy: when configuration is wrong or the store is unavailable, the system won't silently fall back to JVM-local state, avoiding the "it looks rate-limited but each machine is counting separately" problem. Local mode is suitable for trials and small services; Redis mode handles shared constraints for horizontal scaling. Vector source: llmrix-router-production-deployment.svg https://raw.githubusercontent.com/llmrix-inc/llmrix-router/main/docs/images/llmrix-router-production-deployment.svg . This production deployment diagram extends the earlier module boundaries with external components from the runtime environment: The main path is "caller → enterprise gateway → Router replica → Provider Adapter → model service," with responses returning along the same connection. Green links represent Router↔Redis state reads and writes, orange links represent secret injection, and purple dashed links represent asynchronous telemetry. None of these three link types enter the model response body. Smaller systems can skip the gateway and Redis, embedding the Router directly in business services or running a single instance. When the Router scales horizontally, shared Redis state should be used, and edge security and public traffic governance should be handed to a gateway. The Router's API key authentication handles the client-to-router boundary; provider keys stay only in the Router deployment environment or a secret management system. When model routing lacks explainability, production troubleshooting is slow. At minimum, you need to know which model was selected, why a retry happened, which candidate went into cooldown, how long the first token took, and where costs were spent. LLMRix defines lifecycle events at the core layer: request started, route selected, attempt started/ended, first token, usage recorded, target cooldown, and request completed. With Spring integration connecting to Micrometer, Observation, and Actuator, you get metrics for request volume, latency, attempt count, first-token latency, candidate availability, in-flight requests, and token usage. The project doesn't deploy Prometheus, Grafana, or an OpenTelemetry Collector, but it provides a stable observability boundary. Java teams with existing monitoring infrastructure can plug in directly without maintaining a separate console. Beyond standard routing, LLMRix provides several capabilities for evaluation and feedback. Semantic routing scores candidates based on request content. Contextual bandits combine selection counts with reward feedback to balance between using the current best model and exploring alternatives. Online Shadow sends side-effect-free requests to shadow models at a sample rate without affecting the main request result. The offline evaluation component aggregates quality, latency, failure, and cost across multiple models on a sample set. Fugu orchestration supports roles like Worker, Thinker, and Verifier iterating within bounded turns, token budgets, and cost budgets. Generation, reflection, and verification all have stop conditions, retry, and fallback mechanisms. ONNX policies can be loaded at runtime, but training and policy rollout remain the responsibility of offline systems. These capabilities aren't the starting point for every team. They show that LLMRix goes beyond static load balancing—it reserves interfaces for evaluable, learnable model decisions. The project requires Java 17 or later Java 21 recommended . Core artifacts are published to Maven Central under the MIT License. Without Spring, just include Core and Integrations: