Designing LLM failover as a route change, not a retry loop

Modelflare published a technical guide explaining that its LLM failover works as a route change between routing groups rather than a retry loop against the same service provider. Each request carries two identifiers — the exact model ID in the JSON body and the routing group bound to the API key — and an ordinary key can name one primary group plus ordered fallback groups; a request moves to the next group when the current group has no channel for the model, is at its rate limit, or exhausts every channel priority with a retryable error. Modelflare said failover stops immediately on client errors such as malformed JSON or context length exceeded, on a cancelled client connection, on requests pinned to one channel, and on streamed responses that have already sent body bytes, and that a request is billed at the multiplier of the group that served it, with the resolved group recorded in the consume log.

Model groups and failover: how Modelflare routes a request Every request depends on two identifiers: the exact model ID in the body and the routing group on the key. Ordinary keys add ordered fallback groups that take over when a route cannot serve the request. Modelflare https://modelflare.dev/?utm source=huggingface&utm medium=space&utm campaign=model-groups-and-failover routes each request through a routing group, a set of service provider channels. This article explains how model IDs and groups interact, how group failover works on ordinary keys, which errors stop a request immediately, and how billing follows the group that served the request. Two identifiers on every request 1. The model ID in the JSON body. It must match a public catalog ID exactly. Do not substitute a provider marketing name, a date suffix, or a nearby variant. 2. The routing group bound to the API key. It decides which service routes can serve that model. Both have to agree. A valid JSON body still fails when the key's group has no channel for that model. Create the key in a non-default group that lists the model; there is no single required group. The live catalog, with prices, groups, and protocol labels, is Models & Pricing https://modelflare.dev/pricing?utm source=huggingface&utm medium=space&utm campaign=model-groups-and-failover . To check which models a key can call: curl https://modelflare.dev/v1/models \ -H "Authorization: Bearer YOUR MODELFLARE API KEY" Use the exact id from the response in the model field. The routing chain An ordinary API key names one primary group and, optionally, fallback groups in priority order. Configure the chain in API Keys https://modelflare.dev/keys?utm source=huggingface&utm medium=space&utm campaign=model-groups-and-failover when you create or edit a key. Order matters, and the console lets you move entries up and down. - Primary group. The group named on the key. - Channel priorities inside it. Every priority in the group is tried before the next group. - Next fallback group. A route change, not another retry of the same service provider. - Billed by the group that served. The resolved group is recorded in the consume log. When a request moves to the next group - The group has no channel for the requested model. - The group is at its rate limit for your account. - Every channel priority inside the group has been tried and still returned a retryable error. A group exhausts its own channel priorities before the next group is tried, so a fallback is a route change rather than an extra retry of the same provider. When it does not Failover covers route and connection problems only. The request stops immediately on: - client errors, such as malformed JSON, an invalid parameter, or context length exceeded; - a cancelled client connection; - a request pinned to one specific channel; - a streamed response that has already sent real body bytes to the client. Heartbeats alone do not prevent a switch, but once content has been written, the request is never replayed on another route. A client that streams should handle a mid-stream failure itself. Status codes to expect | Status | Typical cause | |---|---| | 400 | Invalid JSON, unsupported field, or wrong endpoint for the model | | 401 | Missing or invalid API key | | 403 | The key cannot use this model or group | | 404 | Unknown path or missing task | | 429 | Rate limit | | 499 | Client cancelled the request | | 5xx | Service provider or gateway failure | Image and video requests sent to the wrong protocol return 400 with unsupported endpoint and name the required API. Billing after a switch After a failover, the request is billed at the multiplier of the group that served it, not the primary group. The consume log records which group that was, so a route change is visible after the fact. Campaign groups require a fallback Limited-time routes are temporary by design. When a key's primary group is a campaign route, the key cannot be saved without at least one concrete fallback group, so it keeps working after that route ends. Smart API Keys A Smart API Key does not take a manual fallback list. It walks every group the account can use, ordered by its routing strategy balanced, stability first, or low price first . Choose an ordinary key when you want to control the order yourself and a Smart API Key when you want the gateway to resolve it. See Smart API Keys smart-api-key.html for details. Further reading: Models and groups https://docs.modelflare.dev/guides/models-and-groups/?utm source=huggingface&utm medium=space&utm campaign=model-groups-and-failover , Group failover https://docs.modelflare.dev/guides/features/group-failover/?utm source=huggingface&utm medium=space&utm campaign=model-groups-and-failover , and Errors https://docs.modelflare.dev/guides/errors/?utm source=huggingface&utm medium=space&utm campaign=model-groups-and-failover . For higher limits or volume arrangements, contact business@modelflare.dev mailto:business@modelflare.dev .