Part 3 of the series Building and Governing Production AI Infrastructure with Bifrost. Part 2 covered Access Profiles and how reusable policy becomes governed per-user access.
Routing usually starts as a configuration problem. You have a few providers, some models are available through more than one of them, and traffic needs to be distributed in a predictable way.
One provider may be cheaper, another may perform better for a particular workload, and a third may exist mainly as a fallback. At that stage, weighted routing and provider restrictions are often enough.
The problem becomes more interesting once the right destination depends on information that is only available when a request arrives. A research team may need access to a different model from a customer-facing application. Premium traffic may deserve a different path from free-tier traffic. A provider that is normally preferred may need to stop receiving requests once its budget is close to exhaustion. A request that looks like a difficult reasoning task may justify a more capable model, while a simple one does not.
At that point, routing stops being only a question of where traffic should normally go. The gateway has to decide where this particular request should go, based on what it knows at runtime.
That is the role of Routing Rules in Bifrost. They use CEL, the Common Expression Language, to evaluate conditions against request context and, when a rule matches, change the provider or model selected for that request.
The important architectural detail is that Routing Rules run before Bifrost’s normal governance-based provider selection. They can therefore override the route that would otherwise have been selected from a Virtual Key’s provider_configs.
They do not, however, replace the governance around that request. That distinction becomes important later.
Consider a Virtual Key configured to distribute eligible requests 70/30 between two providers. If no Routing Rule matches, Bifrost can use that weighted configuration as usual. If a rule does match, the rule’s target takes precedence and the normal weighted provider selection is skipped for that request.
A simplified version of the flow looks like this:
One detail is easy to miss here: selecting a provider is not the same thing as selecting the API key that will serve the request.
Bifrost separates those decisions. A Routing Rule may choose the provider, after which normal key selection still takes place within that provider. When Adaptive Load Balancing is enabled, Bifrost can use runtime performance signals such as error rate, token-aware latency, and rate-limit state when choosing among the available keys.
A Routing Rule can bypass that key-selection step, but only explicitly. Targets accept an optional key_id; if it is set, the request is pinned to that key. The target must also specify the provider. If key_id is omitted, Bifrost continues with its normal load-balanced key selection.
This separation is useful because “routing” actually covers several different decisions. Governance limits which providers and models the caller is allowed to use.
Routing Rules decide whether a request should take a different route based on runtime context. Load balancing then decides how the chosen provider should actually serve the request.
Treating those as separate layers makes the rest of the system much easier to reason about.
CEL itself is not the most interesting part of Bifrost’s implementation. Most useful rules are short. What matters is the context Bifrost makes available to those expressions.
A rule can inspect basic request properties such as the model, provider, and request type:
request_type == "embedding"
Bifrost documents request types including chat completions, embeddings, batch requests, image generation, moderation, transcription, and translation.
Rules can also inspect request headers and query parameters:
headers["x-tier"] == "premium"
params["region"] == "eu"
Header lookup is case-insensitive. If a header or parameter referenced by a rule is missing, that condition does not bring down the request; it simply does not produce a matching route.
More interestingly, the evaluator can use organizational context that Bifrost already knows about the caller. Variables include identifiers and names for Virtual Keys, teams, and customers:
team_name == "ml-research"
virtual_key_name.startsWith("prod-")
That changes where routing logic needs to live. Without a gateway-level policy, an application often ends up containing its own branching logic: inspect the caller, decide which provider they should use, select a model, and repeat the same logic across every service that talks to an LLM.
With Routing Rules, the application can keep sending ordinary inference requests. It only needs to provide whatever context the gateway requires. The policy governing what that context means can remain centralized.
For a single application, that may look like additional abstraction. Once ten applications share the same gateway, it starts to look much more useful.
The application knows a lot about the request, but it may know very little about the current state of the AI infrastructure behind the gateway. Bifrost exposes some of that state to CEL through three capacity variables:
budget_used,tokens_used, andrequest
budget_used represents budget utilization, tokens_used represents token-rate-limit utilization, and request represents request-rate-limit utilization. These values are percentages and can exceed 100 when a configured limit has already been exhausted.
A rule can therefore intervene before a limit is actually reached:
budget_used > 85
or:
tokens_used > 80
The useful part is how Bifrost decides what budget_used means when several limits apply to the same request.
A route can be covered by more than one configured budget or limit. Bifrost can resolve capacity from provider-and-model-specific configuration, model-only configuration, global provider configuration, and provider configuration associated with the request’s Virtual Key.
When more than one applicable limit exists, the Routing Rules engine uses the highest utilization percentage among them.
This is an important implementation choice. The routing engine reacts to the limit that is currently closest to being exhausted rather than arbitrarily choosing one budget hierarchy to inspect.
If no applicable limit is configured, the corresponding capacity value is 0.0. The absence of a budget therefore does not accidentally look like a heavily consumed one.
This mechanism should not be confused with Adaptive Load Balancing. A CEL rule using budget_used is responding to an explicit governance limit. Adaptive Load Balancing is concerned with operational behavior such as provider or key performance.
In practice, the two can complement each other. A Routing Rule might decide that a particular provider should no longer receive normal traffic because its budget is already 90% consumed. Once another provider has been chosen, load balancing can still decide which key inside that provider is currently the best one to use.
There is another part of the Routing Rules documentation worth paying attention to: what happens when a condition cannot be evaluated as intended.
An expression with invalid CEL syntax is logged and skipped, after which Bifrost continues evaluating subsequent rules. Type mismatches follow similar behavior, and missing request values generally result in the relevant condition not matching rather than in the request itself failing.
That is operationally convenient, but it also means a broken rule may be less obvious than expected. The application can continue working while traffic quietly falls through to another routing rule or to the normal provider-selection path.
An empty CEL expression has the opposite behavior: it always matches.
That is particularly easy to overlook in a first-match-wins system. A blank rule placed early in a scope can capture every request that reaches it, leaving perfectly valid rules underneath it untouched.
If the intention is to create a catch-all, I would prefer to write it explicitly:
true
That makes the rule understandable when someone comes back to the configuration three months later. If the rule should not be active, disabling it is clearer than removing its expression.
A CEL condition is only one part of a Routing Rule. The other important pieces are its scope and priority.
Bifrost evaluates rules through the following hierarchy:
Virtual Key ↓Team ↓Customer ↓Global
The most specific scope is evaluated first. Within one scope, rules are ordered by ascending numeric priority, so priority 0 is evaluated before priority 10. Normal evaluation is first-match-wins.
That hierarchy lets you build defaults without copying the same rules everywhere. A global rule can provide a baseline for the organization, a customer rule can change it for one tenant, a team can apply a narrower policy, and a particular Virtual Key can define an application-specific exception.
It also makes ordering mistakes surprisingly easy.
Every request from the research team satisfies the first rule, so the budget rule never has a chance to run for that traffic. Nothing is technically broken. Both expressions are valid, both rules are enabled, and the second rule can sit visibly in the dashboard. It simply never participates in the routing decision.
This is why priority needs to be designed together with the conditions rather than added afterward. Bifrost’s documentation recommends using low values such as 0–10 for high-priority rules and leaving larger values such as 100+ for broader fallback or catch-all behavior.
Bifrost can also make request complexity available to CEL through complexity_tier.
The Complexity Router classifies supported requests into four tiers:
SIMPLEMEDIUMCOMPLEXREASONING
The implementation is worth understanding because the feature name might suggest that Bifrost sends the prompt to another model in order to decide which model should handle it. It does not.
The documented analyzer runs in-process using configured lexical and structural signals. These include things such as code, reasoning markers, technical terminology, prompt length, and indicators associated with simpler requests. There is no external classification model call sitting in front of the real inference request.
One rule overrides the score entirely: when two or more reasoning keywords appear in the latest user message, the tier is forced to REASONING regardless of the numeric result, which is why the documentation warns against putting broad single words like “explain” or “analyze” into that list.
The resulting value can be used directly in a Routing Rule:
complexity_tier == "REASONING" &&team_name == "research"
That makes it possible to reserve a more capable or expensive route for requests that Bifrost classifies as reasoning-heavy while allowing simpler prompts to follow a cheaper default path.
The failure behavior is also sensible. If Bifrost cannot establish a usable complexity tier, a rule depending on complexity_tier simply does not match. Other Routing Rules that rely on unrelated variables can still be evaluated.
Classification also does not run at all for embeddings, image generation, audio, or any request whose user content mixes text with image or file blocks, so a complexity rule can never fire for that traffic.
There is some nuance for multi-turn conversations. Bifrost documents continuation handling for messages such as “continue” or “try again,” where recent conversation context can contribute to the complexity calculation. A low-information message does not automatically inherit an old complexity classification simply because it belongs to the same conversation.
For that reason, complexity routing looks more appropriate as a targeted policy input than as the only thing deciding where every request goes.
A Routing Rule combines its CEL condition with one or more targets. Targets can set the provider, model, and optionally an API key. Rules can also carry scope, priority, fallbacks, and chaining behavior.
A simplified configuration might look like this:
{ "name": "Research reasoning traffic", "enabled": true, "cel_expression": "complexity_tier == \"REASONING\"", "targets": [ { "provider": "provider-a", "model": "model-a", "weight": 1 } ], "scope": "global", "priority": 10}
Both provider and model are optional on the target. If a field is omitted, the incoming request's value for that field can be preserved. This makes it possible to rewrite only the model, for example, while leaving the provider unchanged.
A rule can also contain several targets whose weights sum to 1:
"targets": [ { "provider": "provider-a", "model": "model-a", "weight": 0.7 }, { "provider": "provider-b", "model": "model-b", "weight": 0.3 }]
When the CEL expression matches, Bifrost chooses among those targets probabilistically according to their weights.
This means CEL does not have to encode traffic splitting itself. The expression answers whether the policy applies; the targets determine what to do with the matching traffic. The same mechanism can therefore be used for controlled rollouts or A/B-style routing without making the CEL condition unnecessarily complicated.
First-match-wins keeps normal rule evaluation predictable, but there are cases where one routing decision needs to feed another. Bifrost supports this with chain_rule.
{ "chain_rule": true}
When a chained rule matches, its target updates the current provider/model state and Bifrost evaluates the scope hierarchy again using that new state.
Conceptually:
model="best-model" │ ▼global chained rule │ ▼provider/model rewritten │ ▼rules evaluated again │ ▼Virtual Key-specific rule │ ▼final route
Evaluation ends when no new rule matches, a terminal rule is reached, or the provider/model state stops changing. That last condition protects the engine from endlessly re-evaluating a chain that is no longer making progress.
An important detail is that capacity variables are resolved again after the provider or model changes. A later rule therefore evaluates budget and rate-limit utilization for the route currently under consideration rather than carrying forward the values from the original request.
That is what makes budget-aware escalation work: one rule downgrades the model when budget is high, and the next rule routes the downgraded model against capacity data for the route it is actually about to use.
This is also where dynamic model aliases become interesting.
Bifrost distinguishes aliases configured at the provider-key level from dynamic aliases implemented through Routing Rules. A client can request a logical name such as:
best-model
and allow Bifrost to determine what that name means for the current Virtual Key, team, customer, or runtime situation. Static provider-key aliasing can then happen later, after the provider key has been selected.
A global chained rule can establish the provider, after which a Virtual Key-scoped terminal rule picks the model, so the same name resolves differently for a premium key than for a standard one.
The practical benefit is that applications do not need to know every infrastructure change behind the name they call. The gateway can change that mapping centrally.
This is probably the most important detail in the routing architecture.
Routing Rules execute before governance provider selection. If a rule matches, the normal weighted provider_configs selection is skipped. Read in isolation, that could sound like a rule has escaped the constraints of the Virtual Key.
It has not.
When a Virtual Key defines allowed providers through its provider configuration, Bifrost carries that permitted-provider set into the request context. The provider-routing documentation describes enforcement at more than one point in the routing process. Components such as the load balancer and model-catalog resolver can restrict their candidate providers against the allowlist, and Bifrost core performs a final provider validation after the routing plugins have run.
The two levels serve different ends: the first makes routing decisions legible in the logs, the second makes the constraint a guarantee that no plugin, and no caller-supplied prefix, can route around.
If the final provider is outside the permitted set, the request is rejected. Fallback providers that are not allowed are filtered out.
The behavior is fail-closed. An empty permitted-provider set does not mean “use whichever provider is available”; it means no provider is allowed.
Supplying the provider explicitly in the request does not bypass that enforcement either. If a Virtual Key is allowed to use only providers A and B, a caller cannot simply request provider C and expect the gateway to accept it.
That gives us a clearer way of describing the relationship between Routing Rules and governance: provider selection can be overridden, but provider authorization remains enforced.
This also connects directly to Access Profiles from the previous article. Access Profiles can provision Virtual Keys with provider access, model restrictions, budgets, rate limits, and other policy. Routing Rules operate later, when an individual request is already moving through the gateway.
Those are related responsibilities, but they solve different problems. Access Profiles define what access should exist. Routing Rules use request-time information to decide how that authorized traffic should be routed.
Once an organization has several providers, models, teams, applications, budgets, and workload types, a single set of static percentages stops being enough.
Some decisions are about permission: which providers and models can this caller use at all?
Others are request-specific: given the caller, headers, current budget utilization, rate-limit state, and perhaps request complexity, which eligible route makes sense right now?
And after that decision is made, there is still the operational question of which concrete provider key should serve the request.
Bifrost keeps those concerns separate enough that the application does not have to own all of them. A team can change its routing policy without every client being redeployed. Premium traffic can follow a different route from ordinary traffic. Requests can be moved away from a provider as a budget or rate limit becomes constrained. Complexity can be used as one signal for deciding whether a more capable model is justified. Global defaults can still coexist with customer, team, and Virtual Key-specific exceptions.
The interesting shift is that the route no longer has to be fully known when the application is written. The client sends the request and enough context to describe it; the gateway can make the infrastructure decision at runtime.
Routing solves only half of the production problem, though. Once Bifrost has chosen a route, the next question is what happens when that route fails. Retries and fallback chains handle that part of the lifecycle: retries can absorb eligible failures on the current provider, while fallbacks allow the request to move elsewhere when the original path cannot complete successfully.
Part 4: Building Resilient LLM Applications with Bifrost: Retries, Fallbacks and Multi-Provider Routing.
Dynamic Model Routing with an LLM Gateway was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.