HybridInfer: Local-first LLM router that falls back to cloud on failure

HybridInfer, a local-first LLM router that falls back to a remote OpenAI-compatible endpoint when local inference stalls or fails, has been released as a Python package (pip install hybridinfer) with an OpenAI-compatible server, CLI, and streaming support. The router, a port of the failure-aware runtime-health controller from the HybridInfer research system, uses first-token-commit semantics to handle streaming fallback and includes self-healing features that pull failing models out of rotation. It integrates with Ollama for local models and any OpenAI-compatible remote tier, aiming to provide speed and privacy without wedged models.

Run LLMs on your own machine, without the crashes. HybridInfer is a reliability-aware router: it sends each request to your local model first, watches the local runtime in real time, and the moment local inference stalls, crashes, or is predicted to fail, it transparently falls back to a remote model. You keep local-first speed and privacy; you never get left with a wedged model and no answer. It runs as a local OpenAI-compatible server , so any tool that can talk to the OpenAI API can point at HybridInfer and get smart routing for free. The routing and reliability core is a Python port of the failure-aware runtime-health controller from the HybridInfer research system https://github.com/SimranKoul2026/HybridInfer . Running a model locally is cheap and private, but local runtimes wedge: long prompts stall in prefill, the GPU runs out of memory, a driver hangs. Naive "local only" setups then just hang. HybridInfer treats reliability as a first-class routing signal: Local-first. Cheap/short requests stay on your device. Runtime-health aware. It learns, per model and per prompt length, how likely local is to fail, and pre-empts the requests that would. In-request fallback. If local stalls no token for N seconds or errors mid-request, it retries on the remote tier automatically - the caller just gets an answer. Self-healing. A model that keeps failing is pulled out of rotation, then probed back in after a cooldown. Your models, your choice. Local tier is any model you've pulled in Ollama https://ollama.com ; remote tier is any OpenAI-compatible endpoint. pip install hybridinfer You also need Ollama https://ollama.com for the local tier: ollama pull llama3.2:3b 1. write a starter config to ~/.hybridinfer/config.yaml hybridinfer init 2. point the remote tier at your provider export OPENAI API KEY=sk-... 3a. one-shot from the CLI hybridinfer run "Explain the CAP theorem in two sentences." 3b. or run the server and use it like the OpenAI API hybridinfer serve curl http://127.0.0.1:8080/v1/chat/completions \ -H 'Content-Type: application/json' \ -d '{"messages": {"role":"user","content":"Write a haiku about GPUs."} }' Point any OpenAI client at it - streaming works too: python from openai import OpenAI client = OpenAI base url="http://127.0.0.1:8080/v1", api key="unused" non-streaming r = client.chat.completions.create model="auto", messages= {"role": "user", "content": "hi"} print r.choices 0 .message.content streaming Server-Sent Events for chunk in client.chat.completions.create model="auto", messages= {"role": "user", "content": "hi"} , stream=True : print chunk.choices 0 .delta.content or "", end="", flush=True Or from the CLI: hybridinfer run --stream "..." . Every response carries a non-standard hybridinfer block telling you which tier served it, whether it fell back, and the latency - safe for clients to ignore. In streaming mode this metadata rides on the final chunk the one with finish reason: "stop" , just before data: DONE . Streaming complicates fallback - once a token is on the wire it cannot be un-sent. HybridInfer handles this with first-token-commit semantics: - If local fails before emitting a token the dominant prefill-wedge case , it falls back to remote cleanly - the client only ever sees the remote stream. - Once local has emitted a token, the request is committed to local; a mid-stream wedge ends the stream honestly the final chunk reports the error rather than garbling output by splicing in a second model. python from hybridinfer.config import load settings from hybridinfer.router import HybridRouter router = HybridRouter load settings "~/.hybridinfer/config.yaml" res = router.complete {"role": "user", "content": "hello"} print res.text, res.tier, res.fell back For each request: Estimate complexity prompt length - short / medium / long bin . Predict local failure risk from a self-calibrating profile keyed by backend, model, length-bin . If it is above risk prefer remote , skip local and go straight to remote. Try local under a hard timeout and a stall watchdog no new token for local stall timeout s = treated as a wedge . On any local failure, fall back to remote in the same request. Record the outcome to update the risk profile and a safety state machine LOCAL ELIGIBLE - CAUTION - UNSAFE - RECOVERING - RESTORED that pulls a failing local tier out and probes it back after a cooldown. hybridinfer init writes an annotated config.yaml . Key knobs: | Key | Meaning | |---|---| local / remote | backend ollama / openai , model, base url, api key env | routing.local stall timeout s | no-token gap that counts as a wedge | routing.risk prefer remote | predicted-failure prob at/above which local is skipped | routing.enable in request fallback | auto-retry on remote when local fails | routing.enable recovery | hold out a failing local tier, then probe it back | risk profile path | where the learned risk profile is persisted | The force local / force remote flags and the enable gates also let you reproduce the research A0-A3 reliability ablation arms. - This is a router , not an inference engine - it orchestrates Ollama and a remote API, it does not run model weights itself. - The desktop build uses runtime-health signals latency, stalls, errors, learned risk . The Android research additionally used on-device thermal headroom, which has no portable desktop equivalent. - Streaming SSE is supported with first-token-commit fallback above . A mid-stream local wedge after the first token cannot be recovered by fallback. Apache-2.0. See LICENSE /SimranKoul2026/HybridInfer-Python-tool/blob/main/LICENSE .