Every few weeks a new model drops and my study group chat fills up with screenshots: "this one is cheaper," "this one is better at code," "switch now." I can never verify any of it quickly, because my test scripts all hard-code one provider's client. Rewriting the harness is slower than the hype cycle.
So here is the learning question: can a ~60-line, standard-library-only Python switchboard let me swap providers behind one interface, route toy tasks to different backends, and prove with a failing fixture where the routing breaks?
Run the final script and you should see:
[route=summarize] backend=local-echo cost=0.0000 out='SUMMARY: ...'
[route=code] backend=mock-strong cost=0.0020 out='def add(a, b): ...'
[route=summarize] backend=mock-strong cost=0.0020 out='SUMMARY: ...' (fallback fired)
The third line is the interesting one — keep reading.
A wrapper hides one provider's quirks. A switchboard does something different: it defines your task types (summarize
, code
, chat
) and maps each to a backend plus a fallback. When a new model appears — whatever it is called this month — you add one entry and re-run the same fixtures. Your evaluation questions stay fixed while backends rotate under them.
That is the actual skill: separating what you ask from who answers.
Save as switchboard.py
:
"""Tiny provider-agnostic LLM switchboard (stdlib only)."""
from dataclasses import dataclass, field
from typing import Callable
def local_echo(prompt: str) -> str:
"""Free 'backend': deterministic, offline, good enough for smoke tests."""
return f"SUMMARY: {prompt[:40]}"
def mock_strong(prompt: str) -> str:
"""Pretend paid model. Fails on empty input, like a real API rejects it."""
if not prompt.strip():
raise ValueError("backend rejected empty prompt")
if prompt.startswith("write code"):
return "def add(a, b):\n return a + b"
return f"SUMMARY: {prompt[:40]}"
@dataclass
class Backend:
name: str
fn: Callable[[str], str]
cost_per_call: float # USD, your own pricing notes go here
@dataclass
class Switchboard:
routes: dict[str, list[Backend]] = field(default_factory=dict)
spent: float = 0.0
def register(self, task: str, backends: list[Backend]) -> None:
self.routes[task] = backends
def run(self, task: str, prompt: str) -> str:
if task not in self.routes:
raise KeyError(f"no route registered for task '{task}'")
last_err = None
for backend in self.routes[task]:
try:
out = backend.fn(prompt)
self.spent += backend.cost_per_call
print(f"[route={task}] backend={backend.name:<11} "
f"cost={backend.cost_per_call:.4f} out={out.splitlines()[0]!r}")
return out
except Exception as e: # try the fallback backend
last_err = e
raise RuntimeError(f"all backends failed for '{task}'") from last_err
if __name__ == "__main__":
free = Backend("local-echo", local_echo, 0.0)
paid = Backend("mock-strong", mock_strong, 0.002)
sb = Switchboard()
sb.register("summarize", [free, paid]) # cheap first, paid as fallback
sb.register("code", [paid]) # only the 'strong' backend
sb.run("summarize", "explain gradient descent to a first-year student")
sb.run("code", "write code to add two numbers")
sb.run("summarize", " ") # free backend accepts junk... or does it?
print(f"total spent: ${sb.spent:.4f}")
[route=summarize] backend=local-echo cost=0.0000 out='SUMMARY: explain gradient descent to a f'
[route=code] backend=mock-strong cost=0.0020 out='def add(a, b):'
[route=summarize] backend=local-echo cost=0.0000 out='SUMMARY: '
total spent: $0.0020
Wait — the third line did not fall back, and it returned a garbage summary of whitespace. My local_echo
backend happily accepts an empty prompt. The failure I promised in the intro only fires if the primary backend raises. Before reading on: which fixture input would force the fallback line from the intro to appear? (Answer at the bottom.)
Swap the registration so the strict backend is primary:
sb.register("summarize", [paid, free]) # strict first, lenient as fallback
sb.run("summarize", " ")
Now you get the intro's third line: mock-strong
raises on the empty prompt, the switchboard catches it, and local-echo
answers instead. The concept that actually matters: fallback order is a policy decision, and "free first" and "strict first" fail in opposite directions. Free-first silently returns junk; strict-first silently spends money when you expected the cheap path.
Exception
is fine for a teaching harness, but in real code you should distinguish "provider is down" (retry/fallback) from "my prompt was rejected" (fallback just re-fails expensively).I do most of these experiments inside MonkeyCode, since its free model access lets me prototype against a real LLM endpoint instead of only mocks, and the free server option means the harness runs somewhere that is not my laptop between classes. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The switchboard pattern above is deliberately provider-agnostic, though — the same file works against any endpoint, including none at all.
Add a third route, "chat"
, and a fixture file of five prompts where you predict in writing which backend should handle each one. Then make mock_strong
randomly raise on 30% of calls (seed random
for reproducibility) and check whether your predictions about cost still hold. If your cost estimate assumed zero failures, what does that tell you about launch-week pricing comparisons?
With summarize
registered as [free, paid]
, only an input that makes local_echo
itself raise would trigger the fallback — and local_echo
never raises. The intro's third line only appears under the strict-first registration. If you predicted that, you understood the policy-ordering point; if not, run both versions and diff the output.
If you find a fixture input where the fallback makes things worse (e.g., the lenient backend returns something dangerously plausible), post it — minimal counterexamples are the best part of these threads.