A field-service team built a support chatbot that sent every message to a cloud LLM endpoint. The design held until a technician drove through a tunnel, and the request queue grew into an eleven-minute backlog. The same week, a support ticket containing a customer's account number appeared in a third-party log because the payload was never classified. The fix was not a bigger cloud budget but a local-first router that decides where each request runs.
Latency is the first failure mode, because a round trip to a hosted endpoint adds network time on top of model time. Autocomplete-style features feel broken when every keystroke waits for a distant server instead of a local process. Secrets are the second failure, because any payload sent to a third party can leak into logs or vendor systems. Offline is the third, because a tablet in a tunnel simply has no route to the cloud.
Local inference and cloud APIs are two legs of a routing policy, not a binary choice. Each request deserves an evaluation against the same conditions, and the table below captures those conditions. The router implementation in the next section turns that table into executable logic with a small Python module. The recent wave of free and cheap model announcements makes this decision more urgent, because every new endpoint adds another leg to the routing table.
| Condition | Local model | Cloud free server |
|---|---|---|
| Payload contains PII | Always | Never |
| Network unreachable | Always | Never |
| Latency budget under 300 ms | Prefer | Avoid |
| Task requires strong reasoning | Avoid | Prefer |
| Local queue deeper than three | Avoid | Prefer |
| Token budget nearly exhausted | Prefer | Avoid |
The table encodes a simple principle: privacy and availability win over capability. Capability wins only when the network is healthy and the payload is safe. The table also exposes the hidden assumption that a local model is always available, which is why the router treats local inference as the default and the cloud as an exception.
The Python module below implements the decision table as a pure function with no side effects. A pure function is trivial to unit test and safe to run in dry-run mode.
from dataclasses import dataclass
from enum import Enum
class Route(Enum):
LOCAL = "local"
CLOUD = "cloud"
QUEUE = "queue"
@dataclass
class RequestContext:
payload_class: str # "public" | "internal" | "pii"
network_ok: bool
latency_budget_ms: int
task_complexity: int # 1 (glue) to 5 (deep reasoning)
local_queue_depth: int
token_budget_remaining: int
def decide(ctx: RequestContext) -> Route:
if ctx.payload_class == "pii" or not ctx.network_ok:
return Route.LOCAL
if ctx.latency_budget_ms < 300:
return Route.LOCAL
if ctx.task_complexity >= 4 and ctx.local_queue_depth < 3:
return Route.CLOUD
if ctx.token_budget_remaining < 1000:
return Route.LOCAL
return Route.CLOUD
The fallback chain below makes the router resilient in both directions without hiding failures. A local failure can retry on the cloud leg only when the payload is safe. A cloud failure always falls back to local instead of dropping the request.
def route(ctx: RequestContext, local_fn, cloud_fn):
decision = decide(ctx)
if decision == Route.LOCAL:
try:
return local_fn(ctx)
except Exception:
if ctx.network_ok and ctx.payload_class != "pii":
return cloud_fn(ctx)
return None
try:
return cloud_fn(ctx)
except Exception:
return local_fn(ctx)
Step one is payload classification, and every request gets a class before it reaches the router. The classifier itself needs adversarial tests, because a mislabeled payload defeats the whole policy. Step two is a network probe with a hard timeout, since a hanging connection is worse than a fast failure. Step three is a per-feature latency budget, because autocomplete and batch summarization should never share one threshold.
Step four is a shadow week in dry-run mode, where the router logs decisions but the old path still serves traffic. Step five is the cutover, which happens only after the logs confirm the expected split. The dry-run phase is where teams discover that their assumptions about traffic and latency are wrong. A typical finding is that eighty percent of requests are safe to run locally, which makes the cloud leg a fallback rather than a primary path.
Running the shadow mode is a one-liner that appends every decision to a JSONL file. The sample rate keeps the log small while still covering a full business day.
python local_first_router.py --shadow --log decisions.jsonl --sample-rate 0.1
After a week, a small aggregation command counts the routes and reveals the real split.
jq -r .route decisions.jsonl | sort | uniq -c
Teams should compare the p95 latency of each leg before the cutover, because the decision table is only as good as its inputs. The JSONL log becomes the source of truth for the cutover decision, and it doubles as a regression fixture for future changes to the policy.
The test plan below locks the three rules that matter most, and it runs in seconds with pytest. Each test constructs a context and asserts the expected route.
def test_pii_never_routes_to_cloud():
ctx = RequestContext("pii", True, 500, 5, 0, 50000)
assert decide(ctx) == Route.LOCAL
def test_offline_never_routes_to_cloud():
ctx = RequestContext("public", False, 500, 5, 0, 50000)
assert decide(ctx) == Route.LOCAL
def test_complex_task_routes_to_cloud():
ctx = RequestContext("public", True, 2000, 5, 1, 50000)
assert decide(ctx) == Route.CLOUD
MonkeyCode's free model access and free server option fit the cloud leg of this table. Teams get a hosted fallback without provisioning their own GPU infrastructure. The free server wins when the local machine is asleep or the local model cannot handle a complex refactor. It also wins when the network is healthy but the local queue is saturated. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The free model access includes a 10 million token allowance that suits prototyping and shadow evaluation. Teams should verify the current terms before treating that allowance as a production dependency.
The decision table is a heuristic, not a guarantee, and a misclassified payload can still reach the cloud leg through the fallback path. Teams with strict data-residency requirements should avoid third-party free servers entirely, because the classifier itself can fail. Teams running production SLAs need a paid tier or self-hosted infrastructure, since a free server can rate-limit or change terms without notice. The pattern also assumes a local model exists and runs acceptably on the target hardware. That assumption fails for many laptops and many tasks.
Run the router in dry-run mode for a week before changing any production path. The logs will answer the only question that matters: whether local-first saves latency and tokens or just adds complexity. AI coding tools promoted every developer to reviewer, yet the routing policies behind those tools rarely receive the same testing discipline. A decision table, a pure function, and a shadow week are enough to turn a gut feeling about local-first into a measured policy.