HybridInfer: Local-first LLM router that falls back to cloud on failure

Run LLMs on your own machine, without the crashes. HybridInfer is a reliability-aware router: it sends each request to your local model first, watches the local runtime in real time, and the moment local inference stalls, crashes, or is predicted to fail, it transparently falls back to a remote model. You keep local-first speed and privacy; you never get left with a wedged model and no answer.

It runs as a local OpenAI-compatible server, so any tool that can talk to the OpenAI API can point at HybridInfer and get smart routing for free.

The routing and reliability core is a Python port of the failure-aware runtime-health controller from the HybridInfer research system (

[https://github.com/SimranKoul2026/HybridInfer]).

Running a model locally is cheap and private, but local runtimes wedge: long prompts stall in prefill, the GPU runs out of memory, a driver hangs. Naive "local only" setups then just hang. HybridInfer treats reliability as a first-class routing signal:

Local-first. Cheap/short requests stay on your device.Runtime-health aware. It learns, per model and per prompt length, how likely local is to fail, and pre-empts the requests that would.In-request fallback. If local stalls (no token for N seconds) or errors mid-request, it retries on the remote tier automatically - the caller just gets an answer.Self-healing. A model that keeps failing is pulled out of rotation, then probed back in after a cooldown.Your models, your choice. Local tier is any model you've pulled inOllama; remote tier is any OpenAI-compatible endpoint.

pip install hybridinfer

You also need Ollama for the local tier:

ollama pull llama3.2:3b
hybridinfer init

export OPENAI_API_KEY=sk-...

hybridinfer run "Explain the CAP theorem in two sentences."

hybridinfer serve
curl http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"messages":[{"role":"user","content":"Write a haiku about GPUs."}]}'

Point any OpenAI client at it - streaming works too:

from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:8080/v1", api_key="unused")

r = client.chat.completions.create(model="auto", messages=[{"role": "user", "content": "hi"}])
print(r.choices[0].message.content)

for chunk in client.chat.completions.create(
    model="auto", messages=[{"role": "user", "content": "hi"}], stream=True
):
    print(chunk.choices[0].delta.content or "", end="", flush=True)

Or from the CLI: hybridinfer run --stream "..."

.

Every response carries a non-standard hybridinfer

block telling you which tier served it, whether it fell back, and the latency - safe for clients to ignore. In streaming mode this metadata rides on the final chunk (the one with finish_reason: "stop"

), just before data: [DONE]

.

Streaming complicates fallback - once a token is on the wire it cannot be un-sent. HybridInfer handles this with first-token-commit semantics:

• If local fails before emitting a token (the dominant prefill-wedge case), it falls back to remote cleanly - the client only ever sees the remote stream. - Once local has emitted a token, the request is committed to local; a mid-stream wedge ends the stream honestly (the final chunk reports the error) rather than garbling output by splicing in a second model.

from hybridinfer.config import load_settings
from hybridinfer.router import HybridRouter

router = HybridRouter(load_settings("~/.hybridinfer/config.yaml"))
res = router.complete([{"role": "user", "content": "hello"}])
print(res.text, res.tier, res.fell_back)

For each request:

Estimate complexity(prompt length -> short / medium / long bin).** Predict local failure risk**from a self-calibrating profile keyed by(backend, model, length-bin)

. If it is aboverisk_prefer_remote

, skip local and go straight to remote.Try local under a hard timeoutand a stall watchdog (no new token forlocal_stall_timeout_s

=> treated as a wedge).On any local failure, fall back to remote in the same request.Record the outcome to update the risk profile and a safety state machine (LOCAL_ELIGIBLE -> CAUTION -> UNSAFE -> RECOVERING -> RESTORED

) that pulls a failing local tier out and probes it back after a cooldown.

hybridinfer init

writes an annotated config.yaml

. Key knobs:

The force_local

/ force_remote

flags and the enable_*

gates also let you reproduce the research A0-A3 reliability ablation arms.

• This is a router, not an inference engine - it orchestrates Ollama and a remote API, it does not run model weights itself. - The desktop build uses runtime-health signals (latency, stalls, errors, learned risk). The Android research additionally used on-devicethermal headroom, which has no portable desktop equivalent. - Streaming (SSE) is supported with first-token-commit fallback (above). A mid-stream local wedge after the first token cannot be recovered by fallback.

Apache-2.0. See LICENSE.