Designing LLM failover as a route change, not a retry loop

Every request depends on two identifiers: the exact model ID in the body and the routing group on the key. Ordinary keys add ordered fallback groups that take over when a route cannot serve the request.

Modelflare routes each request through a routing group, a set of service provider channels. This article explains how model IDs and groups interact, how group failover works on ordinary keys, which errors stop a request immediately, and how billing follows the group that served the request.

Two identifiers on every request #

1. The model ID in the JSON body. It must match a public catalog ID exactly. Do not substitute a provider marketing name, a date suffix, or a nearby variant.
2. The routing group bound to the API key. It decides which service routes can serve that model.

Both have to agree. A valid JSON body still fails when the key's group has no channel for that model. Create the key in a non-default group that lists the model; there is no single required group. The live catalog, with prices, groups, and protocol labels, is Models & Pricing.

To check which models a key can call:

curl https://modelflare.dev/v1/models \
  -H "Authorization: Bearer YOUR_MODELFLARE_API_KEY"

Use the exact id from the response in the model field.

The routing chain #

An ordinary API key names one primary group and, optionally, fallback groups in priority order. Configure the chain in API Keys when you create or edit a key. Order matters, and the console lets you move entries up and down.

• Primary group. The group named on the key.
• Channel priorities inside it. Every priority in the group is tried before the next group.
• Next fallback group. A route change, not another retry of the same service provider.
• Billed by the group that served. The resolved group is recorded in the consume log.

When a request moves to the next group #

• The group has no channel for the requested model.
• The group is at its rate limit for your account.
• Every channel priority inside the group has been tried and still returned a retryable error.

A group exhausts its own channel priorities before the next group is tried, so a fallback is a route change rather than an extra retry of the same provider.

When it does not #

Failover covers route and connection problems only. The request stops immediately on:

• client errors, such as malformed JSON, an invalid parameter, or context length exceeded;
• a cancelled client connection;
• a request pinned to one specific channel;
• a streamed response that has already sent real body bytes to the client.

Heartbeats alone do not prevent a switch, but once content has been written, the request is never replayed on another route. A client that streams should handle a mid-stream failure itself.

Status codes to expect #

Image and video requests sent to the wrong protocol return 400 with unsupported_endpoint and name the required API.

Billing after a switch #

After a failover, the request is billed at the multiplier of the group that served it, not the primary group. The consume log records which group that was, so a route change is visible after the fact.

Campaign groups require a fallback #

Limited-time routes are temporary by design. When a key's primary group is a campaign route, the key cannot be saved without at least one concrete fallback group, so it keeps working after that route ends.

Smart API Keys #

A Smart API Key does not take a manual fallback list. It walks every group the account can use, ordered by its routing strategy (balanced, stability first, or low price first). Choose an ordinary key when you want to control the order yourself and a Smart API Key when you want the gateway to resolve it. See Smart API Keys for details.

Further reading: Models and groups, Group failover, and Errors. For higher limits or volume arrangements, contact business@modelflare.dev.