{"slug": "don-t-fear-the-model-router", "title": "Don't Fear the Model Router", "summary": "Mastra AI argues that model routers require rigorous testing beyond simple model selection, introducing scorers, datasets, and experiments to validate routing decisions across quality, cost, speed, and safety axes. The company provides evaluation infrastructure to make agent behavior visible and testable, turning router hypotheses into measurable system behavior.", "body_md": "# Don't Fear the Model Router\n\nRoute to the best model with confidence.\n\nThe first version of [Don’t Marry Your Model](/llm-routing-mastra-ai) made the easy argument: stop sending every task to the same model just because it won the last bake-off.\n\nUse a cheap model for cheap work. Use a stronger model where the work is actually hard. Keep the routing layer flexible enough that you can swap providers without turning your codebase into a shrine.\n\nThat was right.\n\nIt was also incomplete.\n\nBecause once you add a router, you have a new system behavior to test. The question is no longer “which model is best?” The question is “did the system choose the right route, use the right tools, preserve the right evidence, and stop at the right time?”\n\nIf you do not measure that, your model router is just vibes with a dispatch table.\n\nThe router is not the answer. The router is a hypothesis about how your system should behave.\n\nMastra gives us useful surfaces for turning that hypothesis into something testable: [scorers](https://mastra.ai/docs/evals/overview), [ runEvals](https://mastra.ai/reference/evals/run-evals),\n\n[datasets](https://mastra.ai/docs/evals/datasets/overview), and\n\n[experiments](https://mastra.ai/docs/evals/datasets/running-experiments). The API names sound like evaluation infrastructure, which they are, but the real value is simpler:\n\nThey make agent behavior visible enough to argue with.\n\n## What Are We Testing?\n\nThe model router from the earlier post has three obvious specialist routes:\n\n| Route | What should go there | What would be a bad route |\n|---|---|---|\n`code` | implementation, refactoring, debugging, code review | long-context summarization, simple classification |\n`long-context` | messy documents, transcripts, policy synthesis, many files | short mechanical formatting |\n`general` | classification, formatting, simple Q&A, boring extraction | hard code or evidence-heavy analysis |\n\nThat table is a start, but it is not an eval.\n\nAn eval needs examples and scorers:\n\n| Piece | Job |\n|---|---|\n| Dataset item | ”Here is a representative request.” |\n| Ground truth | ”Here is the route or behavior we expected.” |\n| Scorer | ”Here is how we decide whether the output passed.” |\n| Experiment | ”Here is the run we can compare against future runs.” |\n\nThe important move is to test behavior, not just prose quality.\n\nA model can write a beautiful answer after choosing the wrong specialist. A security agent can produce a plausible report without preserving evidence. A support agent can sound empathetic while skipping the refund policy check. The paragraph is the visible part. The trajectory is where the bugs live.\n\nFor a router, I usually start with four axes:\n\n| Axis | Question | Example scorer |\n|---|---|---|\n| Quality | Did it choose the right route and produce a useful result? | route accuracy, answer completeness, faithfulness |\n| Cost | Did it avoid premium models for boring work? | selected route cost class, token budget |\n| Speed | Did it finish inside the product’s latency budget? | runtime or timeout scorer |\n| Other | Did it obey safety, privacy, and observability constraints? | tool allowlist, evidence-preservation, refusal behavior |\n\nThat last column matters. “Other” is where the production scar tissue lives.\n\n## Make the Router Decision Scorable\n\nIf the router only produces a final answer, it is hard to know why it behaved the way it did. You can still score the output, but you are guessing about the decision.\n\nFor evals, give the routing step a small structured contract:\n\nThe production system does not need to show this JSON to users. It can be an internal step, a workflow handoff, or a trace span. The scorer just needs a surface.\n\nHere is a deliberately small Mastra agent that chooses a route:\n\nYes, this is a little artificial. Good. Evals reward boring seams.\n\nWhen the router decision is explicit, you can test the route before you test the downstream specialist. That is how you find out whether the problem is the router, the selected model, the prompt, the tool surface, or the final answer scorer.\n\n## Write a Scorer That Catches the Boring Failure\n\nMastra’s [ createScorer](https://mastra.ai/reference/evals/create-scorer) can use JavaScript functions, LLM judge prompts, or both. Start with functions whenever the failure is deterministic. They are cheaper, faster, and less mysterious.\n\nFor route accuracy, we do not need a judge model. We need to parse JSON and compare one field.\n\nThat scorer is not glamorous. That is the point.\n\nIf the router cannot consistently produce valid JSON and pick the obvious specialist on a tiny test set, there is no reason to trust it with production traffic. You do not need a philosopher-model grading ontology. You need the equivalent of a smoke alarm with a battery in it.\n\n## Run the Small Eval Loop First\n\nMastra’s [ runEvals](https://mastra.ai/reference/evals/run-evals) is the fast loop. Give it a target, test cases, scorers, and a concurrency limit. It runs the target against the data and returns aggregate scores.\n\nThis is the loop you run while changing the prompt, adding a new route, or trying a cheaper router model.\n\nIt is not enough for a mature system, but it is enough to prevent the most embarrassing regression: “we changed the router prompt and it started sending classification tasks to the premium code model.”\n\nCost, speed, quality, and other all show up here:\n\n- Cost: the router model can stay cheap if accuracy holds.\n- Speed: the eval can enforce timeouts or record latency in the harness.\n- Quality: route accuracy and final answer quality are separate scores.\n- Other: JSON validity, allowed tools, safety, and traceability get their own checks.\n\nDo not roll all of that into one “quality” score. Averages are where useful failures go to retire.\n\n## Add an LLM Judge Only Where It Earns Its Keep\n\nSome router behavior is subjective. A request can be legitimately ambiguous:\n\nIs that `code`\n\nbecause debugging? `long-context`\n\nbecause logs? `general`\n\nbecause summary? The right route depends on the tool surface and your product promise.\n\nThis is where an LLM judge can help, but only with a tight rubric. Mastra scorers can mix function steps and prompt-object steps. Use functions for structure, then use a judge for the part that actually needs judgment.\n\nThis scorer costs money because it calls a judge model. That is fine when the judgment is worth it.\n\nDo not use it to check if JSON parses.\n\n## Promote Good Cases Into a Dataset\n\nHard-coded eval arrays are fine at the beginning. Eventually, your examples become product assets. The failed customer ticket, the weird support conversation, the prompt injection attempt, the request that used to route correctly before last Thursday.\n\nThat belongs in a dataset.\n\nMastra datasets are versioned collections of test cases. Each mutation creates a new version, which means you can rerun an experiment against the exact case set that existed when you made a model decision.\n\nFirst configure storage, because datasets need persistence:\n\nThen create a dataset and add cases:\n\nThe moment you have a dataset, you can stop treating eval cases as throwaway script data. They now have IDs, versions, history, and experiment results.\n\nThat is when evals start feeling less like “test files for prompts” and more like product memory.\n\n## Run Experiments Against the Router\n\nOnce the dataset exists, use [ dataset.startExperiment()](https://mastra.ai/reference/datasets/startExperiment) to run it against a registered agent, workflow, or scorer.\n\nNow the conversation changes.\n\nInstead of “the new router seems better,” you can say:\n\n- The old router scored\n`0.94`\n\non route accuracy. - The new router scored\n`0.98`\n\noverall. - It improved long-context routing.\n- It regressed two code-review cases.\n- It reduced premium-model handoffs by 18%.\n- It added 300ms of router latency.\n\nThat is an engineering conversation. There are tradeoffs. You can decide whether the trade is worth it.\n\n## Score Live Behavior, But Do Not Confuse It With Ground Truth\n\nMastra can also attach scorers directly to agents and workflow steps. Live scorers run asynchronously and store score results in your configured database, with sampling controls so you do not score every production response unless you mean to.\n\nThat is useful, but it is a different job.\n\nLive scoring can tell you the router is still emitting valid decisions. It can catch malformed output, toxic content, forbidden tool calls, missing evidence markers, or suspiciously low confidence.\n\nIt usually cannot tell you route accuracy, because production traffic does not arrive with ground truth stapled to it.\n\nThat distinction matters. Live scoring is monitoring. Dataset experiments are controlled tests. You want both, but they answer different questions.\n\n## What to Measure After Route Accuracy\n\nRoute accuracy is the first rung. It tells you whether the request went to the expected specialist. It does not tell you whether the specialist did good work.\n\nAfter the router passes the basics, score the system in layers:\n\n| Layer | What to score | Why it matters |\n|---|---|---|\n| Router decision | selected route, confidence, reason | Catches misclassification and bad escalation rules |\n| Trajectory | expected tool or agent sequence | Catches “right answer, wrong path” behavior |\n| Specialist output | correctness, faithfulness, usefulness | Catches low-quality work after correct routing |\n| Cost and latency | model choice, tokens, runtime | Catches expensive or slow wins |\n| Safety and scope | allowed tools, refusal boundaries, evidence | Catches product-risk failures |\n\nMastra’s `runEvals`\n\nAPI supports agent-level, workflow-level, step-level, and trajectory scorer configurations. That means you do not have to pretend the final answer is the only artifact.\n\nFor a workflow, the shape can look like this:\n\nThat is the mental model I want for agents in production:\n\nScore the decision. Score the path. Score the answer.\n\nIf you only score the answer, the model can pass by accident.\n\n## The Router Should Get More Boring Over Time\n\nThe first routing prompt is usually a paragraph of judgment calls. That is fine for a prototype.\n\nAs you learn from evals, parts of the router should become less magical:\n\n- Clear lexical cases can become deterministic rules.\n- Risky tasks can require explicit approval or a workflow branch.\n- Ambiguous tasks can ask a clarifying question instead of guessing.\n- Expensive routes can require higher confidence or a second signal.\n- Known failure cases can become dataset items.\n\nThe goal is not to make the router “smarter” forever. The goal is to make the system easier to reason about.\n\nSometimes that means a better model. Sometimes it means a tighter prompt. Sometimes it means a workflow step, a scorer, a hard cap, or a boring `if`\n\nstatement that saves you four figures a month.\n\nThat is the whole point of measuring behavior. You stop arguing from taste and start arguing from evidence.\n\n## A Practical Starting Checklist\n\nIf you are building a Mastra router today, I would start here:\n\n- Make the routing decision structured, even if users never see it.\n- Write deterministic scorers for valid JSON, expected route, and forbidden routes.\n- Use\n`runEvals`\n\nwith 10-20 cases before changing router prompts or models. - Promote real failures into a versioned dataset.\n- Run dataset experiments for meaningful prompt, model, route, or workflow changes.\n- Add live scorers for cheap production invariants.\n- Compare experiments by route, not only by average score.\n\nThe average matters less than the failure cluster.\n\nIf every regression is in long-context policy synthesis, you do not have “a worse router.” You have a route boundary problem. If every failed case uses a specific tool, you have a tool contract problem. If every cheap model fails the same two ambiguous cases, you may need escalation logic instead of a more expensive default.\n\nThis is where evals become useful. Not as a ceremony. Not as a dashboard that makes everyone feel temporarily adult.\n\nAs a way to find the shape of the system.\n\n## Resources\n\n[Mastra scorers overview](https://mastra.ai/docs/evals/overview)[Mastra](https://mastra.ai/reference/evals/create-scorer)`createScorer`\n\nreference[Mastra](https://mastra.ai/reference/evals/run-evals)`runEvals`\n\nreference[Mastra datasets overview](https://mastra.ai/docs/evals/datasets/overview)[Mastra dataset experiments](https://mastra.ai/docs/evals/datasets/running-experiments)[Don’t Marry Your Model](/llm-routing-mastra-ai)[Fight Evils with Evals!](/llm-evals-are-broken)", "url": "https://wpnews.pro/news/don-t-fear-the-model-router", "canonical_source": "https://danlevy.net/dont-fear-the-model-router/", "published_at": "2026-07-02 00:00:00+00:00", "updated_at": "2026-07-07 01:36:15.512415+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "ai-infrastructure", "large-language-models", "mlops"], "entities": ["Mastra", "Mastra AI", "runEvals", "scorers", "datasets", "experiments"], "alternates": {"html": "https://wpnews.pro/news/don-t-fear-the-model-router", "markdown": "https://wpnews.pro/news/don-t-fear-the-model-router.md", "text": "https://wpnews.pro/news/don-t-fear-the-model-router.txt", "jsonld": "https://wpnews.pro/news/don-t-fear-the-model-router.jsonld"}}