{"slug": "local-first-llm-routing-a-decision-table-for-latency-secrets-and-offline-mode", "title": "Local-First LLM Routing: A Decision Table for Latency, Secrets, and Offline Mode", "summary": "A field-service team built a local-first LLM routing system that decides whether each request runs on a local model or a cloud endpoint based on latency, payload sensitivity, and network availability. The router, implemented as a pure Python function, prioritizes privacy and offline capability over raw model strength, falling back to the cloud only when the payload is safe and the network is healthy.", "body_md": "A field-service team built a support chatbot that sent every message to a cloud LLM endpoint. The design held until a technician drove through a tunnel, and the request queue grew into an eleven-minute backlog. The same week, a support ticket containing a customer's account number appeared in a third-party log because the payload was never classified. The fix was not a bigger cloud budget but a local-first router that decides where each request runs.\n\nLatency is the first failure mode, because a round trip to a hosted endpoint adds network time on top of model time. Autocomplete-style features feel broken when every keystroke waits for a distant server instead of a local process. Secrets are the second failure, because any payload sent to a third party can leak into logs or vendor systems. Offline is the third, because a tablet in a tunnel simply has no route to the cloud.\n\nLocal inference and cloud APIs are two legs of a routing policy, not a binary choice. Each request deserves an evaluation against the same conditions, and the table below captures those conditions. The router implementation in the next section turns that table into executable logic with a small Python module. The recent wave of free and cheap model announcements makes this decision more urgent, because every new endpoint adds another leg to the routing table.\n\n| Condition | Local model | Cloud free server |\n|---|---|---|\n| Payload contains PII | Always | Never |\n| Network unreachable | Always | Never |\n| Latency budget under 300 ms | Prefer | Avoid |\n| Task requires strong reasoning | Avoid | Prefer |\n| Local queue deeper than three | Avoid | Prefer |\n| Token budget nearly exhausted | Prefer | Avoid |\n\nThe table encodes a simple principle: privacy and availability win over capability. Capability wins only when the network is healthy and the payload is safe. The table also exposes the hidden assumption that a local model is always available, which is why the router treats local inference as the default and the cloud as an exception.\n\nThe Python module below implements the decision table as a pure function with no side effects. A pure function is trivial to unit test and safe to run in dry-run mode.\n\n``` python\n# local_first_router.py\nfrom dataclasses import dataclass\nfrom enum import Enum\n\nclass Route(Enum):\n    LOCAL = \"local\"\n    CLOUD = \"cloud\"\n    QUEUE = \"queue\"\n\n@dataclass\nclass RequestContext:\n    payload_class: str        # \"public\" | \"internal\" | \"pii\"\n    network_ok: bool\n    latency_budget_ms: int\n    task_complexity: int      # 1 (glue) to 5 (deep reasoning)\n    local_queue_depth: int\n    token_budget_remaining: int\n\ndef decide(ctx: RequestContext) -> Route:\n    if ctx.payload_class == \"pii\" or not ctx.network_ok:\n        return Route.LOCAL\n    if ctx.latency_budget_ms < 300:\n        return Route.LOCAL\n    if ctx.task_complexity >= 4 and ctx.local_queue_depth < 3:\n        return Route.CLOUD\n    if ctx.token_budget_remaining < 1000:\n        return Route.LOCAL\n    return Route.CLOUD\n```\n\nThe fallback chain below makes the router resilient in both directions without hiding failures. A local failure can retry on the cloud leg only when the payload is safe. A cloud failure always falls back to local instead of dropping the request.\n\n``` python\ndef route(ctx: RequestContext, local_fn, cloud_fn):\n    decision = decide(ctx)\n    if decision == Route.LOCAL:\n        try:\n            return local_fn(ctx)\n        except Exception:\n            if ctx.network_ok and ctx.payload_class != \"pii\":\n                return cloud_fn(ctx)\n            return None\n    try:\n        return cloud_fn(ctx)\n    except Exception:\n        return local_fn(ctx)\n```\n\nStep one is payload classification, and every request gets a class before it reaches the router. The classifier itself needs adversarial tests, because a mislabeled payload defeats the whole policy. Step two is a network probe with a hard timeout, since a hanging connection is worse than a fast failure. Step three is a per-feature latency budget, because autocomplete and batch summarization should never share one threshold.\n\nStep four is a shadow week in dry-run mode, where the router logs decisions but the old path still serves traffic. Step five is the cutover, which happens only after the logs confirm the expected split. The dry-run phase is where teams discover that their assumptions about traffic and latency are wrong. A typical finding is that eighty percent of requests are safe to run locally, which makes the cloud leg a fallback rather than a primary path.\n\nRunning the shadow mode is a one-liner that appends every decision to a JSONL file. The sample rate keeps the log small while still covering a full business day.\n\n```\npython local_first_router.py --shadow --log decisions.jsonl --sample-rate 0.1\n```\n\nAfter a week, a small aggregation command counts the routes and reveals the real split.\n\n```\njq -r .route decisions.jsonl | sort | uniq -c\n```\n\nTeams should compare the p95 latency of each leg before the cutover, because the decision table is only as good as its inputs. The JSONL log becomes the source of truth for the cutover decision, and it doubles as a regression fixture for future changes to the policy.\n\nThe test plan below locks the three rules that matter most, and it runs in seconds with pytest. Each test constructs a context and asserts the expected route.\n\n``` python\ndef test_pii_never_routes_to_cloud():\n    ctx = RequestContext(\"pii\", True, 500, 5, 0, 50000)\n    assert decide(ctx) == Route.LOCAL\n\ndef test_offline_never_routes_to_cloud():\n    ctx = RequestContext(\"public\", False, 500, 5, 0, 50000)\n    assert decide(ctx) == Route.LOCAL\n\ndef test_complex_task_routes_to_cloud():\n    ctx = RequestContext(\"public\", True, 2000, 5, 1, 50000)\n    assert decide(ctx) == Route.CLOUD\n```\n\nMonkeyCode's free model access and free server option fit the cloud leg of this table. Teams get a hosted fallback without provisioning their own GPU infrastructure. The free server wins when the local machine is asleep or the local model cannot handle a complex refactor. It also wins when the network is healthy but the local queue is saturated. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The free model access includes a 10 million token allowance that suits prototyping and shadow evaluation. Teams should verify the current terms before treating that allowance as a production dependency.\n\nThe decision table is a heuristic, not a guarantee, and a misclassified payload can still reach the cloud leg through the fallback path. Teams with strict data-residency requirements should avoid third-party free servers entirely, because the classifier itself can fail. Teams running production SLAs need a paid tier or self-hosted infrastructure, since a free server can rate-limit or change terms without notice. The pattern also assumes a local model exists and runs acceptably on the target hardware. That assumption fails for many laptops and many tasks.\n\nRun the router in dry-run mode for a week before changing any production path. The logs will answer the only question that matters: whether local-first saves latency and tokens or just adds complexity. AI coding tools promoted every developer to reviewer, yet the routing policies behind those tools rarely receive the same testing discipline. A decision table, a pure function, and a shadow week are enough to turn a gut feeling about local-first into a measured policy.", "url": "https://wpnews.pro/news/local-first-llm-routing-a-decision-table-for-latency-secrets-and-offline-mode", "canonical_source": "https://dev.to/codepro_9661/local-first-llm-routing-a-decision-table-for-latency-secrets-and-offline-mode-2ph9", "published_at": "2026-08-27 03:05:04+00:00", "updated_at": "2026-08-27 03:49:01.261112+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "developer-tools"], "entities": [], "alternates": {"html": "https://wpnews.pro/news/local-first-llm-routing-a-decision-table-for-latency-secrets-and-offline-mode", "markdown": "https://wpnews.pro/news/local-first-llm-routing-a-decision-table-for-latency-secrets-and-offline-mode.md", "text": "https://wpnews.pro/news/local-first-llm-routing-a-decision-table-for-latency-secrets-and-offline-mode.txt", "jsonld": "https://wpnews.pro/news/local-first-llm-routing-a-decision-table-for-latency-secrets-and-offline-mode.jsonld"}}