Local-First LLM Routing: A Decision Table for Latency, Secrets, and Offline Mode

A field-service team built a local-first LLM routing system that decides whether each request runs on a local model or a cloud endpoint based on latency, payload sensitivity, and network availability. The router, implemented as a pure Python function, prioritizes privacy and offline capability over raw model strength, falling back to the cloud only when the payload is safe and the network is healthy.

A field-service team built a support chatbot that sent every message to a cloud LLM endpoint. The design held until a technician drove through a tunnel, and the request queue grew into an eleven-minute backlog. The same week, a support ticket containing a customer's account number appeared in a third-party log because the payload was never classified. The fix was not a bigger cloud budget but a local-first router that decides where each request runs. Latency is the first failure mode, because a round trip to a hosted endpoint adds network time on top of model time. Autocomplete-style features feel broken when every keystroke waits for a distant server instead of a local process. Secrets are the second failure, because any payload sent to a third party can leak into logs or vendor systems. Offline is the third, because a tablet in a tunnel simply has no route to the cloud. Local inference and cloud APIs are two legs of a routing policy, not a binary choice. Each request deserves an evaluation against the same conditions, and the table below captures those conditions. The router implementation in the next section turns that table into executable logic with a small Python module. The recent wave of free and cheap model announcements makes this decision more urgent, because every new endpoint adds another leg to the routing table. | Condition | Local model | Cloud free server | |---|---|---| | Payload contains PII | Always | Never | | Network unreachable | Always | Never | | Latency budget under 300 ms | Prefer | Avoid | | Task requires strong reasoning | Avoid | Prefer | | Local queue deeper than three | Avoid | Prefer | | Token budget nearly exhausted | Prefer | Avoid | The table encodes a simple principle: privacy and availability win over capability. Capability wins only when the network is healthy and the payload is safe. The table also exposes the hidden assumption that a local model is always available, which is why the router treats local inference as the default and the cloud as an exception. The Python module below implements the decision table as a pure function with no side effects. A pure function is trivial to unit test and safe to run in dry-run mode. python local first router.py from dataclasses import dataclass from enum import Enum class Route Enum : LOCAL = "local" CLOUD = "cloud" QUEUE = "queue" @dataclass class RequestContext: payload class: str "public" | "internal" | "pii" network ok: bool latency budget ms: int task complexity: int 1 glue to 5 deep reasoning local queue depth: int token budget remaining: int def decide ctx: RequestContext - Route: if ctx.payload class == "pii" or not ctx.network ok: return Route.LOCAL if ctx.latency budget ms < 300: return Route.LOCAL if ctx.task complexity = 4 and ctx.local queue depth < 3: return Route.CLOUD if ctx.token budget remaining < 1000: return Route.LOCAL return Route.CLOUD The fallback chain below makes the router resilient in both directions without hiding failures. A local failure can retry on the cloud leg only when the payload is safe. A cloud failure always falls back to local instead of dropping the request. python def route ctx: RequestContext, local fn, cloud fn : decision = decide ctx if decision == Route.LOCAL: try: return local fn ctx except Exception: if ctx.network ok and ctx.payload class = "pii": return cloud fn ctx return None try: return cloud fn ctx except Exception: return local fn ctx Step one is payload classification, and every request gets a class before it reaches the router. The classifier itself needs adversarial tests, because a mislabeled payload defeats the whole policy. Step two is a network probe with a hard timeout, since a hanging connection is worse than a fast failure. Step three is a per-feature latency budget, because autocomplete and batch summarization should never share one threshold. Step four is a shadow week in dry-run mode, where the router logs decisions but the old path still serves traffic. Step five is the cutover, which happens only after the logs confirm the expected split. The dry-run phase is where teams discover that their assumptions about traffic and latency are wrong. A typical finding is that eighty percent of requests are safe to run locally, which makes the cloud leg a fallback rather than a primary path. Running the shadow mode is a one-liner that appends every decision to a JSONL file. The sample rate keeps the log small while still covering a full business day. python local first router.py --shadow --log decisions.jsonl --sample-rate 0.1 After a week, a small aggregation command counts the routes and reveals the real split. jq -r .route decisions.jsonl | sort | uniq -c Teams should compare the p95 latency of each leg before the cutover, because the decision table is only as good as its inputs. The JSONL log becomes the source of truth for the cutover decision, and it doubles as a regression fixture for future changes to the policy. The test plan below locks the three rules that matter most, and it runs in seconds with pytest. Each test constructs a context and asserts the expected route. python def test pii never routes to cloud : ctx = RequestContext "pii", True, 500, 5, 0, 50000 assert decide ctx == Route.LOCAL def test offline never routes to cloud : ctx = RequestContext "public", False, 500, 5, 0, 50000 assert decide ctx == Route.LOCAL def test complex task routes to cloud : ctx = RequestContext "public", True, 2000, 5, 1, 50000 assert decide ctx == Route.CLOUD MonkeyCode's free model access and free server option fit the cloud leg of this table. Teams get a hosted fallback without provisioning their own GPU infrastructure. The free server wins when the local machine is asleep or the local model cannot handle a complex refactor. It also wins when the network is healthy but the local queue is saturated. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The free model access includes a 10 million token allowance that suits prototyping and shadow evaluation. Teams should verify the current terms before treating that allowance as a production dependency. The decision table is a heuristic, not a guarantee, and a misclassified payload can still reach the cloud leg through the fallback path. Teams with strict data-residency requirements should avoid third-party free servers entirely, because the classifier itself can fail. Teams running production SLAs need a paid tier or self-hosted infrastructure, since a free server can rate-limit or change terms without notice. The pattern also assumes a local model exists and runs acceptably on the target hardware. That assumption fails for many laptops and many tasks. Run the router in dry-run mode for a week before changing any production path. The logs will answer the only question that matters: whether local-first saves latency and tokens or just adds complexity. AI coding tools promoted every developer to reviewer, yet the routing policies behind those tools rarely receive the same testing discipline. A decision table, a pure function, and a shadow week are enough to turn a gut feeling about local-first into a measured policy.