{"slug": "hybridinfer-local-first-llm-router-that-falls-back-to-cloud-on-failure", "title": "HybridInfer: Local-first LLM router that falls back to cloud on failure", "summary": "HybridInfer, a local-first LLM router that falls back to a remote OpenAI-compatible endpoint when local inference stalls or fails, has been released as a Python package (pip install hybridinfer) with an OpenAI-compatible server, CLI, and streaming support. The router, a port of the failure-aware runtime-health controller from the HybridInfer research system, uses first-token-commit semantics to handle streaming fallback and includes self-healing features that pull failing models out of rotation. It integrates with Ollama for local models and any OpenAI-compatible remote tier, aiming to provide speed and privacy without wedged models.", "body_md": "**Run LLMs on your own machine, without the crashes.** HybridInfer is a\nreliability-aware router: it sends each request to your **local** model first,\nwatches the local runtime in real time, and the moment local inference stalls,\ncrashes, or is *predicted* to fail, it transparently falls back to a **remote**\nmodel. You keep local-first speed and privacy; you never get left with a wedged\nmodel and no answer.\n\nIt runs as a local **OpenAI-compatible server**, so any tool that can talk to\nthe OpenAI API can point at HybridInfer and get smart routing for free.\n\nThe routing and reliability core is a Python port of the failure-aware runtime-health controller from the HybridInfer research system (\n\n[https://github.com/SimranKoul2026/HybridInfer]).\n\nRunning a model locally is cheap and private, but local runtimes wedge: long prompts stall in prefill, the GPU runs out of memory, a driver hangs. Naive \"local only\" setups then just hang. HybridInfer treats reliability as a first-class routing signal:\n\n**Local-first.** Cheap/short requests stay on your device.**Runtime-health aware.** It learns, per model and per prompt length, how likely local is to fail, and pre-empts the requests that would.**In-request fallback.** If local stalls (no token for N seconds) or errors mid-request, it retries on the remote tier automatically - the caller just gets an answer.**Self-healing.** A model that keeps failing is pulled out of rotation, then probed back in after a cooldown.**Your models, your choice.** Local tier is any model you've pulled in[Ollama](https://ollama.com); remote tier is any OpenAI-compatible endpoint.\n\n```\npip install hybridinfer\n```\n\nYou also need [Ollama](https://ollama.com) for the local tier:\n\n```\nollama pull llama3.2:3b\n# 1. write a starter config to ~/.hybridinfer/config.yaml\nhybridinfer init\n\n# 2. point the remote tier at your provider\nexport OPENAI_API_KEY=sk-...\n\n# 3a. one-shot from the CLI\nhybridinfer run \"Explain the CAP theorem in two sentences.\"\n\n# 3b. or run the server and use it like the OpenAI API\nhybridinfer serve\ncurl http://127.0.0.1:8080/v1/chat/completions \\\n  -H 'Content-Type: application/json' \\\n  -d '{\"messages\":[{\"role\":\"user\",\"content\":\"Write a haiku about GPUs.\"}]}'\n```\n\nPoint any OpenAI client at it - streaming works too:\n\n``` python\nfrom openai import OpenAI\nclient = OpenAI(base_url=\"http://127.0.0.1:8080/v1\", api_key=\"unused\")\n\n# non-streaming\nr = client.chat.completions.create(model=\"auto\", messages=[{\"role\": \"user\", \"content\": \"hi\"}])\nprint(r.choices[0].message.content)\n\n# streaming (Server-Sent Events)\nfor chunk in client.chat.completions.create(\n    model=\"auto\", messages=[{\"role\": \"user\", \"content\": \"hi\"}], stream=True\n):\n    print(chunk.choices[0].delta.content or \"\", end=\"\", flush=True)\n```\n\nOr from the CLI: `hybridinfer run --stream \"...\"`\n\n.\n\nEvery response carries a non-standard `hybridinfer`\n\nblock telling you which tier\nserved it, whether it fell back, and the latency - safe for clients to ignore.\nIn streaming mode this metadata rides on the final chunk (the one with\n`finish_reason: \"stop\"`\n\n), just before `data: [DONE]`\n\n.\n\nStreaming complicates fallback - once a token is on the wire it cannot be\nun-sent. HybridInfer handles this with **first-token-commit** semantics:\n\n- If local fails\n**before** emitting a token (the dominant prefill-wedge case), it falls back to remote cleanly - the client only ever sees the remote stream. - Once local has emitted a token, the request is\n**committed** to local; a mid-stream wedge ends the stream honestly (the final chunk reports the error) rather than garbling output by splicing in a second model.\n\n``` python\nfrom hybridinfer.config import load_settings\nfrom hybridinfer.router import HybridRouter\n\nrouter = HybridRouter(load_settings(\"~/.hybridinfer/config.yaml\"))\nres = router.complete([{\"role\": \"user\", \"content\": \"hello\"}])\nprint(res.text, res.tier, res.fell_back)\n```\n\nFor each request:\n\n**Estimate complexity**(prompt length -> short / medium / long bin).** Predict local failure risk**from a self-calibrating profile keyed by`(backend, model, length-bin)`\n\n. If it is above`risk_prefer_remote`\n\n, skip local and go straight to remote.**Try local** under a hard timeout**and** a stall watchdog (no new token for`local_stall_timeout_s`\n\n=> treated as a wedge).**On any local failure, fall back to remote** in the same request.**Record the outcome** to update the risk profile and a safety state machine (`LOCAL_ELIGIBLE -> CAUTION -> UNSAFE -> RECOVERING -> RESTORED`\n\n) that pulls a failing local tier out and probes it back after a cooldown.\n\n`hybridinfer init`\n\nwrites an annotated `config.yaml`\n\n. Key knobs:\n\n| Key | Meaning |\n|---|---|\n`local` / `remote` |\nbackend (`ollama` / `openai` ), model, base_url, api_key_env |\n`routing.local_stall_timeout_s` |\nno-token gap that counts as a wedge |\n`routing.risk_prefer_remote` |\npredicted-failure prob at/above which local is skipped |\n`routing.enable_in_request_fallback` |\nauto-retry on remote when local fails |\n`routing.enable_recovery` |\nhold out a failing local tier, then probe it back |\n`risk_profile_path` |\nwhere the learned risk profile is persisted |\n\nThe `force_local`\n\n/ `force_remote`\n\nflags and the `enable_*`\n\ngates also let you\nreproduce the research A0-A3 reliability ablation arms.\n\n- This is a\n**router**, not an inference engine - it orchestrates Ollama and a remote API, it does not run model weights itself. - The desktop build uses\n**runtime-health** signals (latency, stalls, errors, learned risk). The Android research additionally used on-device**thermal** headroom, which has no portable desktop equivalent. - Streaming (SSE) is supported with first-token-commit fallback (above). A mid-stream local wedge after the first token cannot be recovered by fallback.\n\nApache-2.0. See [LICENSE](/SimranKoul2026/HybridInfer-Python-tool/blob/main/LICENSE).", "url": "https://wpnews.pro/news/hybridinfer-local-first-llm-router-that-falls-back-to-cloud-on-failure", "canonical_source": "https://github.com/SimranKoul2026/HybridInfer-Python-tool", "published_at": "2026-09-02 01:36:28+00:00", "updated_at": "2026-09-02 01:51:59.629780+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-tools", "ai-infrastructure"], "entities": ["HybridInfer", "Ollama", "OpenAI"], "alternates": {"html": "https://wpnews.pro/news/hybridinfer-local-first-llm-router-that-falls-back-to-cloud-on-failure", "markdown": "https://wpnews.pro/news/hybridinfer-local-first-llm-router-that-falls-back-to-cloud-on-failure.md", "text": "https://wpnews.pro/news/hybridinfer-local-first-llm-router-that-falls-back-to-cloud-on-failure.txt", "jsonld": "https://wpnews.pro/news/hybridinfer-local-first-llm-router-that-falls-back-to-cloud-on-failure.jsonld"}}