Learn Provider-Agnostic Model Routing by Building a Tiny LLM Switchboard

A developer built a ~60-line, standard-library-only Python switchboard that lets users swap LLM providers behind one interface and route tasks to different backends with fallbacks. The project demonstrates separating task definitions from backend implementations, allowing evaluation questions to stay fixed while backends rotate. The switchboard includes a failing fixture that shows routing breaks when a backend accepts empty input, highlighting the need for robust validation.

Every few weeks a new model drops and my study group chat fills up with screenshots: "this one is cheaper," "this one is better at code," "switch now." I can never verify any of it quickly, because my test scripts all hard-code one provider's client. Rewriting the harness is slower than the hype cycle. So here is the learning question: can a ~60-line, standard-library-only Python switchboard let me swap providers behind one interface, route toy tasks to different backends, and prove with a failing fixture where the routing breaks? Run the final script and you should see: route=summarize backend=local-echo cost=0.0000 out='SUMMARY: ...' route=code backend=mock-strong cost=0.0020 out='def add a, b : ...' route=summarize backend=mock-strong cost=0.0020 out='SUMMARY: ...' fallback fired The third line is the interesting one — keep reading. A wrapper hides one provider's quirks. A switchboard does something different: it defines your task types summarize , code , chat and maps each to a backend plus a fallback. When a new model appears — whatever it is called this month — you add one entry and re-run the same fixtures. Your evaluation questions stay fixed while backends rotate under them. That is the actual skill: separating what you ask from who answers . Save as switchboard.py : """Tiny provider-agnostic LLM switchboard stdlib only .""" from dataclasses import dataclass, field from typing import Callable --- Backends: same signature str - str, so they are interchangeable --- def local echo prompt: str - str: """Free 'backend': deterministic, offline, good enough for smoke tests.""" return f"SUMMARY: {prompt :40 }" def mock strong prompt: str - str: """Pretend paid model. Fails on empty input, like a real API rejects it.""" if not prompt.strip : raise ValueError "backend rejected empty prompt" if prompt.startswith "write code" : return "def add a, b :\n return a + b" return f"SUMMARY: {prompt :40 }" @dataclass class Backend: name: str fn: Callable str , str cost per call: float USD, your own pricing notes go here @dataclass class Switchboard: routes: dict str, list Backend = field default factory=dict spent: float = 0.0 def register self, task: str, backends: list Backend - None: self.routes task = backends def run self, task: str, prompt: str - str: if task not in self.routes: raise KeyError f"no route registered for task '{task}'" last err = None for backend in self.routes task : try: out = backend.fn prompt self.spent += backend.cost per call print f" route={task} backend={backend.name:<11} " f"cost={backend.cost per call:.4f} out={out.splitlines 0 r}" return out except Exception as e: try the fallback backend last err = e raise RuntimeError f"all backends failed for '{task}'" from last err if name == " main ": free = Backend "local-echo", local echo, 0.0 paid = Backend "mock-strong", mock strong, 0.002 sb = Switchboard sb.register "summarize", free, paid cheap first, paid as fallback sb.register "code", paid only the 'strong' backend sb.run "summarize", "explain gradient descent to a first-year student" sb.run "code", "write code to add two numbers" sb.run "summarize", " " free backend accepts junk... or does it? print f"total spent: ${sb.spent:.4f}" route=summarize backend=local-echo cost=0.0000 out='SUMMARY: explain gradient descent to a f' route=code backend=mock-strong cost=0.0020 out='def add a, b :' route=summarize backend=local-echo cost=0.0000 out='SUMMARY: ' total spent: $0.0020 Wait — the third line did not fall back, and it returned a garbage summary of whitespace. My local echo backend happily accepts an empty prompt. The failure I promised in the intro only fires if the primary backend raises. Before reading on: which fixture input would force the fallback line from the intro to appear? Answer at the bottom. Swap the registration so the strict backend is primary: sb.register "summarize", paid, free strict first, lenient as fallback sb.run "summarize", " " Now you get the intro's third line: mock-strong raises on the empty prompt, the switchboard catches it, and local-echo answers instead. The concept that actually matters: fallback order is a policy decision, and "free first" and "strict first" fail in opposite directions. Free-first silently returns junk; strict-first silently spends money when you expected the cheap path. Exception is fine for a teaching harness, but in real code you should distinguish "provider is down" retry/fallback from "my prompt was rejected" fallback just re-fails expensively .I do most of these experiments inside MonkeyCode, since its free model access lets me prototype against a real LLM endpoint instead of only mocks, and the free server option means the harness runs somewhere that is not my laptop between classes. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The switchboard pattern above is deliberately provider-agnostic, though — the same file works against any endpoint, including none at all. Add a third route, "chat" , and a fixture file of five prompts where you predict in writing which backend should handle each one. Then make mock strong randomly raise on 30% of calls seed random for reproducibility and check whether your predictions about cost still hold. If your cost estimate assumed zero failures, what does that tell you about launch-week pricing comparisons? With summarize registered as free, paid , only an input that makes local echo itself raise would trigger the fallback — and local echo never raises. The intro's third line only appears under the strict-first registration. If you predicted that, you understood the policy-ordering point; if not, run both versions and diff the output. If you find a fixture input where the fallback makes things worse e.g., the lenient backend returns something dangerously plausible , post it — minimal counterexamples are the best part of these threads.