{"slug": "learn-provider-agnostic-model-routing-by-building-a-tiny-llm-switchboard", "title": "Learn Provider-Agnostic Model Routing by Building a Tiny LLM Switchboard", "summary": "A developer built a ~60-line, standard-library-only Python switchboard that lets users swap LLM providers behind one interface and route tasks to different backends with fallbacks. The project demonstrates separating task definitions from backend implementations, allowing evaluation questions to stay fixed while backends rotate. The switchboard includes a failing fixture that shows routing breaks when a backend accepts empty input, highlighting the need for robust validation.", "body_md": "Every few weeks a new model drops and my study group chat fills up with screenshots: \"this one is cheaper,\" \"this one is better at code,\" \"switch now.\" I can never verify any of it quickly, because my test scripts all hard-code one provider's client. Rewriting the harness is slower than the hype cycle.\n\nSo here is the learning question: **can a ~60-line, standard-library-only Python switchboard let me swap providers behind one interface, route toy tasks to different backends, and prove with a failing fixture where the routing breaks?**\n\nRun the final script and you should see:\n\n```\n[route=summarize] backend=local-echo   cost=0.0000  out='SUMMARY: ...'\n[route=code]       backend=mock-strong cost=0.0020  out='def add(a, b): ...'\n[route=summarize] backend=mock-strong cost=0.0020  out='SUMMARY: ...'  (fallback fired)\n```\n\nThe third line is the interesting one — keep reading.\n\nA wrapper hides one provider's quirks. A **switchboard** does something different: it defines *your* task types (`summarize`\n\n, `code`\n\n, `chat`\n\n) and maps each to a backend plus a fallback. When a new model appears — whatever it is called this month — you add one entry and re-run the same fixtures. Your evaluation questions stay fixed while backends rotate under them.\n\nThat is the actual skill: separating *what you ask* from *who answers*.\n\nSave as `switchboard.py`\n\n:\n\n```\n\"\"\"Tiny provider-agnostic LLM switchboard (stdlib only).\"\"\"\nfrom dataclasses import dataclass, field\nfrom typing import Callable\n\n# --- Backends: same signature (str) -> str, so they are interchangeable ---\n\ndef local_echo(prompt: str) -> str:\n    \"\"\"Free 'backend': deterministic, offline, good enough for smoke tests.\"\"\"\n    return f\"SUMMARY: {prompt[:40]}\"\n\ndef mock_strong(prompt: str) -> str:\n    \"\"\"Pretend paid model. Fails on empty input, like a real API rejects it.\"\"\"\n    if not prompt.strip():\n        raise ValueError(\"backend rejected empty prompt\")\n    if prompt.startswith(\"write code\"):\n        return \"def add(a, b):\\n    return a + b\"\n    return f\"SUMMARY: {prompt[:40]}\"\n\n@dataclass\nclass Backend:\n    name: str\n    fn: Callable[[str], str]\n    cost_per_call: float  # USD, your own pricing notes go here\n\n@dataclass\nclass Switchboard:\n    routes: dict[str, list[Backend]] = field(default_factory=dict)\n    spent: float = 0.0\n\n    def register(self, task: str, backends: list[Backend]) -> None:\n        self.routes[task] = backends\n\n    def run(self, task: str, prompt: str) -> str:\n        if task not in self.routes:\n            raise KeyError(f\"no route registered for task '{task}'\")\n        last_err = None\n        for backend in self.routes[task]:\n            try:\n                out = backend.fn(prompt)\n                self.spent += backend.cost_per_call\n                print(f\"[route={task}] backend={backend.name:<11} \"\n                      f\"cost={backend.cost_per_call:.4f}  out={out.splitlines()[0]!r}\")\n                return out\n            except Exception as e:  # try the fallback backend\n                last_err = e\n        raise RuntimeError(f\"all backends failed for '{task}'\") from last_err\n\nif __name__ == \"__main__\":\n    free = Backend(\"local-echo\", local_echo, 0.0)\n    paid = Backend(\"mock-strong\", mock_strong, 0.002)\n\n    sb = Switchboard()\n    sb.register(\"summarize\", [free, paid])   # cheap first, paid as fallback\n    sb.register(\"code\", [paid])              # only the 'strong' backend\n\n    sb.run(\"summarize\", \"explain gradient descent to a first-year student\")\n    sb.run(\"code\", \"write code to add two numbers\")\n    sb.run(\"summarize\", \"   \")               # free backend accepts junk... or does it?\n    print(f\"total spent: ${sb.spent:.4f}\")\n[route=summarize] backend=local-echo   cost=0.0000  out='SUMMARY: explain gradient descent to a f'\n[route=code]       backend=mock-strong cost=0.0020  out='def add(a, b):'\n[route=summarize] backend=local-echo   cost=0.0000  out='SUMMARY:     '\ntotal spent: $0.0020\n```\n\nWait — the third line did **not** fall back, and it returned a garbage summary of whitespace. My `local_echo`\n\nbackend happily accepts an empty prompt. The failure I promised in the intro only fires if the *primary* backend raises. Before reading on: which fixture input would force the fallback line from the intro to appear? (Answer at the bottom.)\n\nSwap the registration so the strict backend is primary:\n\n```\nsb.register(\"summarize\", [paid, free])  # strict first, lenient as fallback\nsb.run(\"summarize\", \"   \")\n```\n\nNow you get the intro's third line: `mock-strong`\n\nraises on the empty prompt, the switchboard catches it, and `local-echo`\n\nanswers instead. The concept that actually matters: **fallback order is a policy decision, and \"free first\" and \"strict first\" fail in opposite directions.** Free-first silently returns junk; strict-first silently spends money when you expected the cheap path.\n\n`Exception`\n\nis fine for a teaching harness, but in real code you should distinguish \"provider is down\" (retry/fallback) from \"my prompt was rejected\" (fallback just re-fails expensively).I do most of these experiments inside MonkeyCode, since its free model access lets me prototype against a real LLM endpoint instead of only mocks, and the free server option means the harness runs somewhere that is not my laptop between classes. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The switchboard pattern above is deliberately provider-agnostic, though — the same file works against any endpoint, including none at all.\n\nAdd a third route, `\"chat\"`\n\n, and a fixture file of five prompts where you *predict in writing* which backend should handle each one. Then make `mock_strong`\n\nrandomly raise on 30% of calls (seed `random`\n\nfor reproducibility) and check whether your predictions about cost still hold. If your cost estimate assumed zero failures, what does that tell you about launch-week pricing comparisons?\n\nWith `summarize`\n\nregistered as `[free, paid]`\n\n, only an input that makes `local_echo`\n\nitself raise would trigger the fallback — and `local_echo`\n\nnever raises. The intro's third line only appears under the strict-first registration. If you predicted that, you understood the policy-ordering point; if not, run both versions and diff the output.\n\nIf you find a fixture input where the fallback makes things *worse* (e.g., the lenient backend returns something dangerously plausible), post it — minimal counterexamples are the best part of these threads.", "url": "https://wpnews.pro/news/learn-provider-agnostic-model-routing-by-building-a-tiny-llm-switchboard", "canonical_source": "https://dev.to/magickong/learn-provider-agnostic-model-routing-by-building-a-tiny-llm-switchboard-2o0a", "published_at": "2026-08-13 03:04:50+00:00", "updated_at": "2026-08-13 03:15:47.086541+00:00", "lang": "en", "topics": ["developer-tools", "large-language-models", "artificial-intelligence"], "entities": [], "alternates": {"html": "https://wpnews.pro/news/learn-provider-agnostic-model-routing-by-building-a-tiny-llm-switchboard", "markdown": "https://wpnews.pro/news/learn-provider-agnostic-model-routing-by-building-a-tiny-llm-switchboard.md", "text": "https://wpnews.pro/news/learn-provider-agnostic-model-routing-by-building-a-tiny-llm-switchboard.txt", "jsonld": "https://wpnews.pro/news/learn-provider-agnostic-model-routing-by-building-a-tiny-llm-switchboard.jsonld"}}