{"slug": "build-observable-ai-model-fallbacks-with-telnyx-ai-gateway", "title": "Build Observable AI Model Fallbacks with Telnyx AI Gateway", "summary": "Telnyx published a Python sample, gateway-fallbacks, that moves AI model retry and fallback logic into its Telnyx AI Gateway rather than client-side SDK loops. The sample provisions a primary model (Kimi-K3) and a fallback (GLM-5.3-Flash) through a token group, forces a one-second timeout on the primary to trigger failover, and exposes the served model and attempt count via response headers such as x-ltg-served-model and x-ltg-attempts. It also disables the OpenAI client's own retries so gateway attempts, ledger records, and billing stay aligned around a single retry policy.", "body_md": "An AI model fallback is only useful if your application can answer two questions afterward:\n\nA client-side retry loop can hide both answers. The application eventually receives a successful response, but the path to that response is difficult to reconstruct. SDK retries can also stack on top of gateway retries, creating extra requests and misleading attempt counts.\n\nThe [`gateway-fallbacks`](https://github.com/team-telnyx/telnyx-code-examples/tree/main/gateway-fallbacks) Python sample moves that resilience policy into Telnyx AI Gateway. It provisions a primary and fallback model, forces a timeout on the primary, and proves the resulting failover twice:\n\nThe example separates management, inference, and observation:\n\n```\nManagement plane                         Inference plane\napi.telnyx.com/v2/llm_token_gateway     llm.telnyx.com/v1\nTelnyx account API key                  scoped token key\n\nPOST /token_groups                      GET /models\nPOST /token_keys                        POST /chat/completions\nGET  /spend/events                             |\nGET  /usage/summary                            v\n        |                               gateway retries and fallbacks\n        +----------------------+----------------+\n                               |\n                               v\nObservation: x-ltg-served-model, x-ltg-attempts,\n             requested_model, model, attempts, cache_hit\n```\n\nThe management plane owns token groups, token keys, budgets, and usage reporting. It uses the Telnyx account API key.\n\nThe inference plane is OpenAI-compatible. It uses only the token key issued for the group, so application code does not need the account credential.\n\nThe sample provisions two allowed model aliases and one fallback relationship:\n\n```\nallowed_models = [\"Kimi-K3\", \"GLM-5.3-Flash\"]\nfallbacks = {\n    \"Kimi-K3\": [\"GLM-5.3-Flash\"],\n}\nretries = {\n    \"max_attempts\": 2,\n    \"retry_delay_ms\": 250,\n    \"backoff\": \"exponential\",\n}\n\ngroup = client.llm_token_gateway.token_groups.create(\n    name=\"gateway-fallbacks-demo\",\n    allowed_models=allowed_models,\n    fallbacks=fallbacks,\n    retries=retries,\n    max_budget=10,\n    budget_duration=\"1d\",\n    idempotency_key=f\"gateway-fallbacks-group-{uuid.uuid4()}\",\n).data\n```\n\nThe group constrains which models its key can call and defines how failures move through the chain. The sample validates the same policy limits locally before making the API request:\n\n`max_attempts` must be between one and five`allowed_models`\n`max_attempts × (1 + longest chain)` cannot exceed eight\nThe gateway also enforces these rules. `policy.py` deliberately sends an invalid `max_attempts=9` update and expects an HTTP `400` without mutating the group.\n\nThe gateway owns retries and model fallbacks, so the OpenAI client disables its own retry behavior:\n\n```\nllm = OpenAI(\n    base_url=\"https://llm.telnyx.com/v1\",\n    api_key=token_key,\n    max_retries=0,\n)\n```\n\nThis matters because an SDK retry is a new request layered on top of the gateway's attempts. Disabling it keeps the response headers, ledger records, and billing aligned around one retry policy.\n\nThe sample distinguishes failures where no model work occurred from failures where processing may already have started.\n\n**Definite no-work failures** such as pre-dispatch errors, `429`, and `529` can retry the same model.\n\n**Uncertain failures** such as timeouts, `408`, `500`, `502`, `503`, and `504` move to the next fallback because the original request may already have reached the provider.\n\nA context-length rejection is terminal because sending the same oversized context to another attempt does not fix the input.\n\nWaiting for a provider incident is not a useful test plan. The sample creates a controlled race instead.\n\n`inference.py` sends a streaming request to `Kimi-K3` with a one-second first-event deadline:\n\n```\nresult = chat_completion(\n    token_key,\n    \"Kimi-K3\",\n    build_forced_failure_prompt(),\n    extra_headers={\n        \"x-ltg-request-timeout\": \"1000\",\n        \"x-ltg-max-attempts\": \"1\",\n    },\n    stream=True,\n    max_tokens=64,\n)\n```\n\nThe generated input is intentionally heavy, approximately 22,000 tokens, and includes a unique nonce so a repeated run cannot rely on provider KV caching to shorten prefill. The goal is for the primary's first event to miss the one-second deadline while `GLM-5.3-Flash` completes as the fallback.\n\nProvider latency changes over time, so this is not presented as an absolute guarantee. The script tries up to three times and reports honestly if the primary answers within the deadline or if both attempts time out.\n\nThe client reads two headers from the raw response:\n\n```\nserved_model = raw.headers.get(\"x-ltg-served-model\")\nattempts = raw.headers.get(\"x-ltg-attempts\")\n```\n\nA successful failover should show:\n\n```\nrequested model:   Kimi-K3\nserved model:      GLM-5.3-Flash\nx-ltg-attempts:    2 or more\n```\n\nThe SDK response model is useful, but the gateway headers make the routing decision explicit on the live path.\n\nThe next step reads the gateway spend events for the issued token key:\n\n```\nevents = client.llm_token_gateway.spend.retrieve_events(\n    start_date=start_date,\n    end_date=end_date,\n    token_key_id=token_key_id,\n    page_number=1,\n    page_size=50,\n)\n\nfor event in events.data:\n    print(\n        event.requested_model,\n        \"->\",\n        event.model,\n        \"attempts:\",\n        event.attempts,\n        \"status:\",\n        event.status,\n        \"cache_hit:\",\n        event.cache_hit,\n    )\n```\n\nWhen `requested_model` and `model` differ, the ledger provides the durable counterpart to the response header. The sample uses a half-open UTC date range and snapshot pagination so the report remains reproducible as the event set changes.\n\n`GET /usage/summary` adds totals and per-model rollups when the account has the `llm_token_gateway.usage.read` permission. A `403` from that endpoint is treated as a nonfatal warning because the spend-events ledger still contains the request-level proof.\n\nThe gateway supports request-level overlays for attempts, delay, backoff, and timeout:\n\n```\nextra_headers = {\n    \"x-ltg-max-attempts\": \"1\",\n    \"x-ltg-retry-delay\": \"1000\",\n    \"x-ltg-backoff\": \"exponential\",\n}\n```\n\nThese controls adjust one request without rewriting the token-group policy.\n\nThe sample also updates `cache_ttl` to 60 seconds and sends the same prompt twice. Gateway caching is best effort, so the script does not invent a cache hit: it reports `x-ltg-attempts: 0` when the repeat is served from cache and otherwise directs you to the ledger's `cache_hit` field.\n\nEvery update and delete follows the mutation-safety pattern:\n\n`If-Match`\n`Idempotency-Key`\nA stale ETag returns `412`, preventing one process from overwriting another process's newer configuration.\n\nClone the repository and install the pinned dependencies:\n\n```\ngit clone https://github.com/team-telnyx/telnyx-code-examples.git\ncd telnyx-code-examples/gateway-fallbacks\n\npython -m venv .venv\nsource .venv/bin/activate\npip install -r requirements.txt\n\ncp .env.example .env\n```\n\nAdd your Telnyx account API key to `.env`:\n\n```\nTELNYX_API_KEY=your_telnyx_api_key_here\nPRIMARY_MODEL=Kimi-K3\nFALLBACK_MODEL=GLM-5.3-Flash\n```\n\nRun the local smoke tests before making network requests:\n\n```\npython -m pytest smoke_test.py -v\n```\n\nThen run the five-step demonstration:\n\n```\n# Creates the group and key. The key secret is printed once.\npython provision.py\n\nexport GATEWAY_TOKEN_KEY=\"<paste the secret>\"\n\npython inference.py\npython ledger.py\npython policy.py\npython teardown.py\n```\n\nThe live demonstration incurs real inference spend. The sample configures a maximum budget of USD 10 over one day, but you should review the policy before running it in your account.\n\n`provision.py` stores only the group and key IDs in `.gateway_state.json`; it never writes the token secret to disk. `teardown.py` revokes the token key, deletes the group, verifies that the revoked key receives HTTP `401`, and removes the local state file.\n\nThe useful pattern is not limited to these two models:\n\nThat turns model failover from an assumption into something operators can inspect during the request and audit afterward.", "url": "https://wpnews.pro/news/build-observable-ai-model-fallbacks-with-telnyx-ai-gateway", "canonical_source": "https://dev.to/sonam_gupta_dba00ec9473e5/build-observable-ai-model-fallbacks-with-telnyx-ai-gateway-3ken", "published_at": "2026-10-20 02:08:30+00:00", "updated_at": "2026-10-20 02:17:03.052512+00:00", "lang": "en", "topics": ["ai-infrastructure", "ai-agents", "developer-tools", "ai-tools", "large-language-models"], "entities": ["Telnyx", "Kimi-K3", "GLM-5.3-Flash", "OpenAI", "Telnyx AI Gateway"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/build-observable-ai-model-fallbacks-with-telnyx-ai-gateway", "markdown": "https://wpnews.pro/news/build-observable-ai-model-fallbacks-with-telnyx-ai-gateway.md", "text": "https://wpnews.pro/news/build-observable-ai-model-fallbacks-with-telnyx-ai-gateway.txt", "jsonld": "https://wpnews.pro/news/build-observable-ai-model-fallbacks-with-telnyx-ai-gateway.jsonld"}}