When AI applications move from demo to production, the trouble usually isn't with prompts—it's with the model calls themselves. Which model should handle this request? What happens when the primary model is rate-limited? How do you control costs? Can you switch providers mid-stream when a streaming response drops? And when running multiple instances, how do you keep quota and health state consistent?
LLMRix Model Router packages all these concerns into an open-source routing runtime for Java.
Over the past couple of years, the barrier to integrating AI models has dropped significantly.
With a single SDK and a few lines of code, an app can call OpenAI, DeepSeek, or any other compatible interface. But once you enter production, a different set of problems emerges: upstream rate limits, timeouts, regional outages, and models that differ widely in capabilities, pricing, context windows, and response speed. When business code is directly coupled to a specific provider, swapping models means rewriting interfaces, exception handling, monitoring, and configuration all at once.
Simple round-robin or reverse proxies can't make these decisions either. They don't know whether a request needs tool calling, which models to exclude for image inputs, or whether a candidate has already exceeded its cost budget or is in a cooldown period.
LLMRix Model Router sits at this boundary. It's a multi-model routing and orchestration framework for Java that extracts provider differences, model selection, failover, quotas, costs, and observability out of business code.
Positioned between AI applications and model services, it handles runtime decision-making and request forwarding. A chat SDK answers how to call; a Router also decides who to call, when to retry, and how to reconcile state.
The project's architectural tradeoff is clear: routing strategies can change, but the correctness of request execution must not. To achieve this, LLMRix divides the system into layers with well-defined boundaries:
llmrix-model-open
.llmrix-model-router-core
handles model targets, capability matching, strategy selection, execution budgets, timeouts, retries, health, and lifecycle events.llmrix-model-router-integrations
provides adapters for OpenAI, DeepSeek, OpenRouter, Ollama, Redis, Bucket4j, ONNX, Shadow, evaluation, and Fugu.Vector source: llmrix-router-architecture.svg. Original panoramic diagram: GitHub architecture SVG.
The Router Core in the middle sits at the junction of decision and execution. It doesn't bind to any single vendor, nor does it rewrite business requests into provider-private objects—instead, it manages candidates through a unified ModelClient
and ModelTarget
. When adding a new provider, changes stay concentrated in the Provider SPI and Transport adapters, without spreading to routing strategies or business code.
| Maven Artifact | Responsibility | Typical Usage |
|---|---|---|
llmrix-model-open |
||
| Shared model contracts, common exceptions, auth SPI, OpenAI-compatible transport & adapters | Reuse unified types when building clients or custom integrations | |
llmrix-model-router-core |
||
| Router Builder, model targets, strategies, executor, state SPI, quota, health & events | Embed Router in plain Java applications | |
llmrix-model-router-integrations |
||
| Built-in providers, Redis, Bucket4j, ONNX, Shadow, evaluation & Fugu | Use official integrations or advanced routing capabilities | |
llmrix-model-router-spring-starter |
||
| Auto-configuration, YAML properties, HTTP/SSE, auth, Actuator & Micrometer | Build Spring Boot routing services | |
llmrix-model-orion |
||
| Framework-neutral remote Java client | Call a standalone Router from Java services | |
llmrix-model-orion-spring-starter |
||
| Orion auto-configuration & Micrometer adapter | Inject remote model clients in Spring Boot business services |
These module boundaries let teams pull in only what they need. Remote business services can depend solely on Orion and the shared model contracts, without dragging in Redis, ONNX, or the entire Router Runtime.
The sequence diagram below follows the source code's execution order, covering capability matching, quota control, and failover. It highlights three distinct paths: successful return, retry before the first visible output, and no-replay after streaming has begun.
Vector source: llmrix-router-request-decision-sequence.svg.
"Whether a model is suitable for this request" and "whether this call will succeed" are handled by two separate components: candidate snapshots and strategies handle the former, while the executor, state store, and exception classification handle the latter.
| Capability Domain | Specific Features | Problems Solved |
|---|---|---|
| Model abstraction | Provider-neutral ModelClient , multimodal request/response, unified exceptions |
|
| Business code no longer binds to vendor SDKs | ||
| Capability matching | ||
operations , features , input-modalities , traits |
||
| Prevents sending tool, image, or audio requests to unsupported models | ||
| Routing strategies | priority, round-robin, weighted random, least-busy, latency-aware, cost-aware, balanced, cache-aware | Trade off stability, cost, latency, and cache hits according to business goals |
| Dynamic decisions | semantic scoring, contextual bandit, customizable RoutingStrategy |
|
| Continuously improve model selection using request semantics or feedback data | ||
| Reliability | per-attempt timeout, total budget, retry predicates, failure thresholds, target cooldown, candidate pool continuation | Handles rate limits, timeouts, transient failures, and partial outages |
| Streaming safety | first-token timeout, stream idle timeout, pre-first-chunk switching, cancellation propagation, tool request non-replay | Prevents duplicate output and replay of side-effecting tool calls |
| Cost governance | input/output/cache/inference token pricing, per-request maxCostUsd , route-level RPM/TPM |
|
| Makes cost constraints part of the decision, not a post-hoc statistic | ||
| Quota & concurrency | target-level limits, route-level limits, auth quota partitions, local Caffeine, Redis atomic leases | Controls resource usage per model, per route, and per tenant |
| Multimodal | Chat, Responses core subset, Embeddings, Rerank, Audio, Image, Video | Unified handling of text, image, audio, file, and video workflows |
| Evaluation & orchestration | Online Shadow, offline Evaluation, Fugu Worker/Thinker/Verifier, ONNX policies | Compare models and organize multi-round collaboration without affecting main traffic |
| Observability | Router/Fugu Listener, Micrometer, Observation, Actuator, request ID | Answers "who was selected, how long it took, why it retried, and how much it cost" |
| Extensibility | ||
ModelProvider , ProviderAuthenticator , ModelPricingResolver , RouterStateStore |
||
| Integrate enterprise proxies, signed auth, internal pricing catalogs, and custom state systems |
LLMRix doesn't reduce "model capability" to a single boolean. Instead, configuration declares four separate dimensions:
operations
: what the model can do, e.g. chat
, embeddings
, rerank
, video-generation
;features
: what protocol features it supports, e.g. streaming
, tools
, structured-output
, prompt-cache
;input-modalities
: what inputs it accepts, e.g. vision
, video
, audio
, file
;traits
: what task types it excels at, e.g. code
, reasoning
, long-context
.The four declarations are independent. They're validated at startup and used to filter candidates per request. Compared to maintaining a single "universal model list," this approach is easier to audit and reduces capability mismatches in production.
LLMRix lets applications depend on stable route names rather than upstream model names.
For example, business code simply requests general
, code
, or reasoning
. Whether the request ultimately goes to OpenAI, DeepSeek, OpenRouter, or a local Ollama instance is decided by the router based on capability, health, latency, cost, and strategy.
Model adjustments therefore live in routing configuration, not in business code. When adding providers, replacing models, or changing selection strategies, upstream applications typically don't need to be rewritten.
LLMRix offers three integration modes:
When a Java service needs to call a remote Router, it can use the project's lightweight client, Orion. Provider keys stay on the Router side only—clients only see route names and the unified protocol.
LLMRix doesn't just work off a flat list of models—it processes requests through a well-bounded execution pipeline.
Upon receiving a request, the Router first checks the operation type, tool calling, structured output, input modality, context length, and routing constraints passed by the caller. Models that can't satisfy the conditions are eliminated before execution begins. A chat with images won't be sent to a text-only model; when a request requires reasoning
or code
traits, candidates that don't declare those traits won't enter the selection phase.
The remaining candidates are then passed to a routing strategy for ordering. Built-in strategies include priority, round-robin, weighted random, least-busy, latency-aware, cost-aware, balanced scoring, and prompt-cache affinity. Semantic routing and contextual multi-armed bandit implementations are also provided. Teams can implement RoutingStrategy
to incorporate data residency, tenant tier, compliance tags, or internal model scores into selection rules.
Before invoking a model, the Router acquires quota and concurrency leases and checks the per-request cost budget. On success, it settles costs based on actual token usage, releases leases, and publishes lifecycle events. On failure, the Router determines—based on exception type—whether to retry, put the target into cooldown, and move on to the next candidate in the pool.
Thus, "who to pick" and "how to call" are two separate responsibilities: strategies handle ordering, while the executor handles timeouts, retries, quotas, health, and resource cleanup. Selection algorithms can be swapped; execution rules remain consistent.
Failover for ordinary requests is relatively straightforward. Streaming responses are different.
Suppose Model A has already output half a sentence to the user when the connection drops. If the system switches to Model B and replays from the beginning, the user sees duplicate content, and tool calls may execute twice.
LLMRix's rule is: candidates can be switched before the first data chunk becomes visible to the caller; after output begins, the request is never replayed. The Router manages first-token timeout and stream idle timeout separately, and propagates cancellation signals to any still-running upstream requests.
This rule directly affects whether callers see consistent output. It encodes the streaming response boundary into the executor, rather than leaving it to business code to handle on its own.
As of version 1.0.2
, LLMRix supports Chat, a core subset of the Responses API, Embeddings, Rerank, audio, image, and video operations. The Spring Boot Starter provides corresponding OpenAI-style endpoints. Both Chat and Responses endpoints support JSON and SSE streaming.
Built-in integrations include OpenAI, DeepSeek, OpenRouter, and Ollama. Not every provider supports the same set of operations, and the Router reads each target's operations
, features
, input-modalities
, and traits
—it never treats "OpenAI-compatible interfaces" as having identical capabilities.
Explicit capability declaration has two practical benefits: configuration errors surface at startup, and routing strategies can filter before requests are sent upstream.
| Integration | Key Operations Implemented | Suitable For |
|---|---|---|
| OpenAI | Chat, Embeddings, Audio, Images, Videos | Using OpenAI's native multimodal interface |
| DeepSeek | Chat | General conversation, code, and reasoning routes |
| OpenRouter | Chat, Embeddings, Rerank | Accessing multiple models and free models through a single upstream |
| Ollama | Chat, Embeddings | Local development, offline environments, and private models |
| Custom Provider | Determined by the ModelProvider implementation |
|
| Enterprise model platforms, internal proxies, proprietary protocols |
"Integration support" doesn't mean all models under that provider support the same operations. Final capabilities depend on the specific model, provider account, and configuration declarations. For example, a particular OpenRouter model might support text chat but not accept image inputs. The Router only filters by target declarations—it doesn't invent capabilities a model doesn't have.
| Endpoint | Purpose |
|---|---|
POST /v1/chat/completions |
|
| Synchronous or SSE streaming chat | |
POST /v1/responses |
|
| Core subset of Responses API, with JSON and SSE | |
POST /v1/embeddings |
|
| Text or token-array embeddings | |
POST /v1/rerank |
|
| Query/document reranking | |
POST /v1/audio/transcriptions |
|
| Audio transcription | |
POST /v1/audio/translations |
|
| Audio translation | |
POST /v1/audio/speech |
|
| Text-to-speech | |
POST /v1/images/generations |
|
| Image generation | |
POST /v1/images/edits |
|
| Multipart image editing | |
POST /v1/videos |
|
| Create a video generation task | |
GET /v1/videos/{video_id} |
|
| Check video task status | |
GET /v1/videos/{video_id}/content |
|
| Download completed video | |
DELETE /v1/videos/{video_id} |
|
| Delete a video task | |
POST /v1/videos/{video_id}/remix |
|
| Create a remix task from an existing video | |
GET /v1/models |
|
| List available route identifiers |
For development and small deployments, you can use in-memory state with no extra infrastructure. Local quota partitions are managed by Caffeine with capacity and idle expiry, preventing unbounded growth from dynamic tenant keys.
When the Router scales to multiple instances, you can switch to Redis. Health state, concurrency leases, RPM, TPM, and multi-armed bandit state can all be shared across instances, with updates performed via Redis atomic operations. Redis mode uses a fail-closed strategy: when configuration is wrong or the store is unavailable, the system won't silently fall back to JVM-local state, avoiding the "it looks rate-limited but each machine is counting separately" problem.
Local mode is suitable for trials and small services; Redis mode handles shared constraints for horizontal scaling.
Vector source: llmrix-router-production-deployment.svg.
This production deployment diagram extends the earlier module boundaries with external components from the runtime environment:
The main path is "caller → enterprise gateway → Router replica → Provider Adapter → model service," with responses returning along the same connection. Green links represent Router↔Redis state reads and writes, orange links represent secret injection, and purple dashed links represent asynchronous telemetry. None of these three link types enter the model response body.
Smaller systems can skip the gateway and Redis, embedding the Router directly in business services or running a single instance. When the Router scales horizontally, shared Redis state should be used, and edge security and public traffic governance should be handed to a gateway. The Router's API key authentication handles the client-to-router boundary; provider keys stay only in the Router deployment environment or a secret management system.
When model routing lacks explainability, production troubleshooting is slow. At minimum, you need to know which model was selected, why a retry happened, which candidate went into cooldown, how long the first token took, and where costs were spent.
LLMRix defines lifecycle events at the core layer: request started, route selected, attempt started/ended, first token, usage recorded, target cooldown, and request completed. With Spring integration connecting to Micrometer, Observation, and Actuator, you get metrics for request volume, latency, attempt count, first-token latency, candidate availability, in-flight requests, and token usage.
The project doesn't deploy Prometheus, Grafana, or an OpenTelemetry Collector, but it provides a stable observability boundary. Java teams with existing monitoring infrastructure can plug in directly without maintaining a separate console.
Beyond standard routing, LLMRix provides several capabilities for evaluation and feedback.
Semantic routing scores candidates based on request content. Contextual bandits combine selection counts with reward feedback to balance between using the current best model and exploring alternatives.
Online Shadow sends side-effect-free requests to shadow models at a sample rate without affecting the main request result. The offline evaluation component aggregates quality, latency, failure, and cost across multiple models on a sample set.
Fugu orchestration supports roles like Worker, Thinker, and Verifier iterating within bounded turns, token budgets, and cost budgets. Generation, reflection, and verification all have stop conditions, retry, and fallback mechanisms. ONNX policies can be loaded at runtime, but training and policy rollout remain the responsibility of offline systems.
These capabilities aren't the starting point for every team. They show that LLMRix goes beyond static load balancing—it reserves interfaces for evaluable, learnable model decisions.
The project requires Java 17 or later (Java 21 recommended). Core artifacts are published to Maven Central under the MIT License.
Without Spring, just include Core and Integrations:
<dependency>
<groupId>com.llmrix.model</groupId>
<artifactId>llmrix-model-router-core</artifactId>
<version>1.0.2</version>
</dependency>
<dependency>
<groupId>com.llmrix.model</groupId>
<artifactId>llmrix-model-router-integrations</artifactId>
<version>1.0.2</version>
</dependency>
Then register models in a route pool:
try (LlmRouter router = LlmRouter.builder()
.integration("openai", provider -> provider
.apiKey(System.getenv("OPENAI_API_KEY"))
.model("gpt-4.1-mini", model -> model
.operations(ModelOperation.CHAT)
.features(ModelFeature.TOOLS)))
.integration("deepseek", provider -> provider
.apiKey(System.getenv("DEEPSEEK_API_KEY"))
.model("deepseek-chat", model -> model
.operations(ModelOperation.CHAT)
.features(ModelFeature.TOOLS)
.traits(ModelTrait.CODE)))
.route("general", route -> route
.strategy("balanced")
.quota(600L, 100_000L)
.models("openai/gpt-4.1-mini", "deepseek/deepseek-chat"))
.build()) {
ChatResponse response = router.chat("Review this Java code");
}
Teams using Spring Boot can use the Starter and YAML configuration instead. Non-Java callers can enable the OpenAI-compatible HTTP API and point their existing SDK's base_url
at the Router. The model
field in requests takes a route name, not a vendor's real model ID.
In a Spring Boot service, routes can be entirely managed by YAML. The example below splits general chat, code analysis, vector search, and reranking into four routes, with shared RPM/TPM quotas for general chat:
llmrix:
model:
router:
enabled: true
default-route: general
routes:
general:
strategy: balanced
quota:
requests-per-minute: 600
tokens-per-minute: 100000
models:
- integration: openai
model: gpt-4.1-mini
- integration: deepseek
model: deepseek-chat
code:
strategy: cost-aware
models:
- integration: deepseek
model: deepseek-chat
- integration: openrouter
model: cohere/north-mini-code:free
embeddings:
strategy: latency-aware
models:
- integration: openai
model: text-embedding-3-small
- integration: openrouter
model: nvidia/nemotron-3-embed-1b:free
rerank:
strategy: priority
models:
- integration: openrouter
model: nvidia/llama-nemotron-rerank-vl-1b-v2:free
execution:
timeout: 30s
max-retries: 1
http:
enabled: true
auth:
mode: api-key
bootstrap-key: ${LLMRIX_MODEL_ROUTER_API_KEY}
integrations:
openai:
provider: openai
api-key: ${OPENAI_API_KEY}
models:
- name: gpt-4.1-mini
operations: [chat]
features: [streaming, tools]
- name: text-embedding-3-small
operations: [embeddings]
deepseek:
provider: deepseek
api-key: ${DEEPSEEK_API_KEY}
models:
- name: deepseek-chat
operations: [chat]
features: [streaming, tools]
traits: [code]
openrouter:
provider: openrouter
api-key: ${OPENROUTER_API_KEY}
models:
- name: cohere/north-mini-code:free
operations: [chat]
traits: [code]
- name: nvidia/nemotron-3-embed-1b:free
operations: [embeddings]
- name: nvidia/llama-nemotron-rerank-vl-1b-v2:free
operations: [rerank]
The models
list in a route is the full candidate pool. The system selects a target within the pool and continues trying on failure—no need to maintain a separate, error-prone fallbacks
list. A provider's base-url
can point to official endpoints, an enterprise proxy, or a private gateway.
Route names express stable business intent; RoutingHints
express per-request temporary constraints. For example, this code review request requires code-capable models while capping cost at 5 cents:
ChatRequest request = ChatRequest.builder()
.userMessage("Find the race condition in this Java code")
.routingHints(RoutingHints.builder()
.require(ModelTrait.CODE)
.maxCostUsd(0.05)
.maxLatency(Duration.ofSeconds(8))
.attribute("tenant", "acme")
.build())
.build();
ChatResponse response = router.chatRoute("code").chat(request);
Hints can require specific operations, tools, or input modalities; they can allow or deny specific targets; they can set max cost, max latency, and pass tenant or business tags. Authenticated requests can also use RoutingHints.AUTH_QUOTA_KEY
to assign independent quota partitions to different callers.
Once the Spring Starter's HTTP endpoints are enabled, Python, Node.js, Go, or any existing OpenAI SDK doesn't need to know about the Router's internals. Just point base_url
at the Router and set the model field to a route name:
export BASE_URL=http://127.0.0.1:8080
export API_KEY=your-llmrix-http-key
curl --no-buffer "$BASE_URL/v1/chat/completions" \
-H "Authorization: Bearer $API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "code",
"messages": [{"role": "user", "content": "Review this pull request"}],
"stream": true,
"temperature": 0
}'
Beyond Chat Completions, the HTTP layer provides a core subset of Responses, Embeddings, Rerank, audio transcription & translation, text-to-speech, image generation & editing, video tasks, and the Models catalog. Controller-level errors use an OpenAI-style structure with error.message
, error.type
, error.code
, and error.param
. Auth requests rejected before reaching the controller may return a more compact error body. Callers should branch on HTTP status and error type, not match against full error text.
When model keys need to be kept centrally, or business services need to scale independently from the Router, use Orion:
OrionModelClient client = OrionModelClient.builder()
.baseUrl("https://router.example.com/v1")
.apiKey(System.getenv("LLMRIX_API_KEY"))
.defaultModel("general")
.defaultEmbeddingModel("embeddings")
.defaultRerankModel("rerank")
.build();
ChatResponse answer = client.chat(ChatRequest.user("Summarize this incident"));
EmbeddingResponse vector = client.embed(EmbeddingRequest.text("Text to index"));
RerankResponse ranked = client.rerank(new RerankRequest(
"refund policy", List.of("Support contact", "Refunds within 30 days")));
Orion supports per-request request IDs and controlled custom headers, as well as async CompletionStage
, Flow.Publisher<ChatChunk>
streaming, and typed models for audio, image, and video. Remote errors and upstream HTTP status are mapped to client exceptions for unified handling on the business side.
Multimodal requests don't require callers to manually pick a provider. The Router excludes unsupported candidates based on each target's declared input modalities:
ChatRequest multimodal = ChatRequest.builder()
.message(Message.user(
new TextPart("Compare this diagram with the attached report."),
new ImagePart("https://example.com/diagram.png", "high"),
new FilePart("https://example.com/report.pdf", "report.pdf")))
.build();
ChatResponse analysis = client.chatModel("multimodal").chat(multimodal);
The corresponding model configuration needs to declare input-modalities: [vision, file]
; audio and video inputs work the same way. This turns "model support status" into verifiable configuration, rather than relying on implicit conventions—and reduces capability errors that only surface after a request reaches upstream.
When troubleshooting a request, you can first inspect route candidates and exclusion reasons, then get runtime metrics through Listeners or Micrometer:
RouteExplanation explanation = router.chatRoute("code")
.explain(ChatRequest.user("Find the race condition"));
System.out.println("selected=" + explanation.selectedTarget());
System.out.println("eligible=" + explanation.eligibleTargets());
System.out.println("excluded=" + explanation.excludedTargets());
Common metrics in Spring Boot include llm.router.requests
, llm.router.attempts
, llm.router.request.duration
, llm.router.first.token
, llm.router.cooldowns
, candidate availability, and in-flight request count. Usage events separately record input, output, cache-read, cache-write, and reasoning tokens, which can be used to aggregate costs by route, provider, and tenant.
You don't have to switch all traffic at once when replacing a model. Online Shadow asynchronously calls shadow models at a sample rate, while the main request is still served by the current production route. The evaluation component records response quality, latency, failures, and cost, and teams can use the online comparison data to decide whether to adjust strategy.
When multi-round generation, thinking, and verification are needed, you can compose multiple ChatModels into a Fugu orchestration:
maxTurns
, token budget, cost budget, and timeout together bound the worst case;These constraints draw resource boundaries for multi-round AI workflows and avoid writing entire orchestration logic as a hard-to-observe block of business code.
| Scenario | Recommended Route Design | Key Capabilities |
|---|---|---|
| Intelligent customer service | Separate routes for general , reasoning , translation |
|
| Stability-first, tenant quotas, failover, usage tracking | ||
| Code assistant | Two-tiered code-fast and code-deep |
|
| Code traits, cost budgeting, tool calling, long context | ||
| Enterprise RAG | Series of embeddings , rerank , answer |
|
| Vectorization, reranking, structured output, traceability | ||
| Multimodal moderation | ||
vision , audio , video split by operation |
||
| Input modality filtering, file safety, binary responses | ||
| Model migration assessment | Main route + Online Shadow | Sampling, isolated timeouts, quality scoring, A/B reports |
| Hybrid cloud & on-prem models | Public models + Ollama/private Provider in candidate pool | Data boundaries, custom auth, health & latency strategies |
| Multi-agent workflows | Fugu or custom Router organizing roles | Multi-round budgets, stop conditions, verification, fallback & event streams |
CHANGELOG.md
and run HTTP/SSE and multimodal regression tests in staging.LLMRix is a better fit than maintaining multiple vendor SDKs directly when:
LLMRix also has clear boundaries. It is not a model hosting platform, a training platform, a secret management system, or a full API gateway. TLS, WAF, external load balancing, Redis HA, secret custody, and container orchestration remain the responsibility of the corresponding infrastructure. When full gateway capabilities are needed, deploy it behind Nginx, APISIX, Higress, or a cloud load balancer.
The project is still young. The repository was created in July 2026, the community is small, and there's no admin console, official container image, or Kubernetes Helm Chart yet. The number of built-in providers is also limited. Complex enterprise environments typically need to integrate internal proxies, authentication, and pricing systems through the SPI.
That said, the repository contains more than just concept docs. The current codebase has 6 releasable artifacts and 43 test classes. On September 3, 2026, running a full mvn test
on the main branch with Java 21 yielded 199 tests run, 0 failures, 0 errors, 0 skipped. Tests cover core routing, quota & health state, protocol mapping, SSE, multimodal controllers, Spring auto-configuration, server startup, and the Orion client. These results indicate basic engineering completeness but don't substitute for production case studies or performance benchmarks.
When an AI application is just getting started, calling a single model directly is usually the fastest approach. As model count grows, traffic increases, or the business becomes more critical, model selection needs to become an independent infrastructure capability.
LLMRix Router provides a stable model boundary. Upstream applications express what they need; the Router decides which models can handle it, which one to pick, how to continue on failure, and records the cost.
If your codebase already has scattered vendor SDKs, retry loops, and model names, start by validating with one non-critical route. After unifying the interface, add health, cost, and quota policies; once Shadow and evaluation have accumulated data, refine the selection approach.
Project: github.com/llmrix-inc/llmrix-router
Further reading: Server Deployment · Orion Client · HTTP API · Maven Central
License: MIT
Current version: 1.0.2
Requirements: Java 17+ (Java 21 recommended); Spring Boot 3.x optional