# Don't Fear the Model Router

> Source: <https://danlevy.net/dont-fear-the-model-router/>
> Published: 2026-07-02 00:00:00+00:00

# Don't Fear the Model Router

Route to the best model with confidence.

The first version of [Don’t Marry Your Model](/llm-routing-mastra-ai) made the easy argument: stop sending every task to the same model just because it won the last bake-off.

Use a cheap model for cheap work. Use a stronger model where the work is actually hard. Keep the routing layer flexible enough that you can swap providers without turning your codebase into a shrine.

That was right.

It was also incomplete.

Because once you add a router, you have a new system behavior to test. The question is no longer “which model is best?” The question is “did the system choose the right route, use the right tools, preserve the right evidence, and stop at the right time?”

If you do not measure that, your model router is just vibes with a dispatch table.

The router is not the answer. The router is a hypothesis about how your system should behave.

Mastra gives us useful surfaces for turning that hypothesis into something testable: [scorers](https://mastra.ai/docs/evals/overview), [ runEvals](https://mastra.ai/reference/evals/run-evals),

[datasets](https://mastra.ai/docs/evals/datasets/overview), and

[experiments](https://mastra.ai/docs/evals/datasets/running-experiments). The API names sound like evaluation infrastructure, which they are, but the real value is simpler:

They make agent behavior visible enough to argue with.

## What Are We Testing?

The model router from the earlier post has three obvious specialist routes:

| Route | What should go there | What would be a bad route |
|---|---|---|
`code` | implementation, refactoring, debugging, code review | long-context summarization, simple classification |
`long-context` | messy documents, transcripts, policy synthesis, many files | short mechanical formatting |
`general` | classification, formatting, simple Q&A, boring extraction | hard code or evidence-heavy analysis |

That table is a start, but it is not an eval.

An eval needs examples and scorers:

| Piece | Job |
|---|---|
| Dataset item | ”Here is a representative request.” |
| Ground truth | ”Here is the route or behavior we expected.” |
| Scorer | ”Here is how we decide whether the output passed.” |
| Experiment | ”Here is the run we can compare against future runs.” |

The important move is to test behavior, not just prose quality.

A model can write a beautiful answer after choosing the wrong specialist. A security agent can produce a plausible report without preserving evidence. A support agent can sound empathetic while skipping the refund policy check. The paragraph is the visible part. The trajectory is where the bugs live.

For a router, I usually start with four axes:

| Axis | Question | Example scorer |
|---|---|---|
| Quality | Did it choose the right route and produce a useful result? | route accuracy, answer completeness, faithfulness |
| Cost | Did it avoid premium models for boring work? | selected route cost class, token budget |
| Speed | Did it finish inside the product’s latency budget? | runtime or timeout scorer |
| Other | Did it obey safety, privacy, and observability constraints? | tool allowlist, evidence-preservation, refusal behavior |

That last column matters. “Other” is where the production scar tissue lives.

## Make the Router Decision Scorable

If the router only produces a final answer, it is hard to know why it behaved the way it did. You can still score the output, but you are guessing about the decision.

For evals, give the routing step a small structured contract:

The production system does not need to show this JSON to users. It can be an internal step, a workflow handoff, or a trace span. The scorer just needs a surface.

Here is a deliberately small Mastra agent that chooses a route:

Yes, this is a little artificial. Good. Evals reward boring seams.

When the router decision is explicit, you can test the route before you test the downstream specialist. That is how you find out whether the problem is the router, the selected model, the prompt, the tool surface, or the final answer scorer.

## Write a Scorer That Catches the Boring Failure

Mastra’s [ createScorer](https://mastra.ai/reference/evals/create-scorer) can use JavaScript functions, LLM judge prompts, or both. Start with functions whenever the failure is deterministic. They are cheaper, faster, and less mysterious.

For route accuracy, we do not need a judge model. We need to parse JSON and compare one field.

That scorer is not glamorous. That is the point.

If the router cannot consistently produce valid JSON and pick the obvious specialist on a tiny test set, there is no reason to trust it with production traffic. You do not need a philosopher-model grading ontology. You need the equivalent of a smoke alarm with a battery in it.

## Run the Small Eval Loop First

Mastra’s [ runEvals](https://mastra.ai/reference/evals/run-evals) is the fast loop. Give it a target, test cases, scorers, and a concurrency limit. It runs the target against the data and returns aggregate scores.

This is the loop you run while changing the prompt, adding a new route, or trying a cheaper router model.

It is not enough for a mature system, but it is enough to prevent the most embarrassing regression: “we changed the router prompt and it started sending classification tasks to the premium code model.”

Cost, speed, quality, and other all show up here:

- Cost: the router model can stay cheap if accuracy holds.
- Speed: the eval can enforce timeouts or record latency in the harness.
- Quality: route accuracy and final answer quality are separate scores.
- Other: JSON validity, allowed tools, safety, and traceability get their own checks.

Do not roll all of that into one “quality” score. Averages are where useful failures go to retire.

## Add an LLM Judge Only Where It Earns Its Keep

Some router behavior is subjective. A request can be legitimately ambiguous:

Is that `code`

because debugging? `long-context`

because logs? `general`

because summary? The right route depends on the tool surface and your product promise.

This is where an LLM judge can help, but only with a tight rubric. Mastra scorers can mix function steps and prompt-object steps. Use functions for structure, then use a judge for the part that actually needs judgment.

This scorer costs money because it calls a judge model. That is fine when the judgment is worth it.

Do not use it to check if JSON parses.

## Promote Good Cases Into a Dataset

Hard-coded eval arrays are fine at the beginning. Eventually, your examples become product assets. The failed customer ticket, the weird support conversation, the prompt injection attempt, the request that used to route correctly before last Thursday.

That belongs in a dataset.

Mastra datasets are versioned collections of test cases. Each mutation creates a new version, which means you can rerun an experiment against the exact case set that existed when you made a model decision.

First configure storage, because datasets need persistence:

Then create a dataset and add cases:

The moment you have a dataset, you can stop treating eval cases as throwaway script data. They now have IDs, versions, history, and experiment results.

That is when evals start feeling less like “test files for prompts” and more like product memory.

## Run Experiments Against the Router

Once the dataset exists, use [ dataset.startExperiment()](https://mastra.ai/reference/datasets/startExperiment) to run it against a registered agent, workflow, or scorer.

Now the conversation changes.

Instead of “the new router seems better,” you can say:

- The old router scored
`0.94`

on route accuracy. - The new router scored
`0.98`

overall. - It improved long-context routing.
- It regressed two code-review cases.
- It reduced premium-model handoffs by 18%.
- It added 300ms of router latency.

That is an engineering conversation. There are tradeoffs. You can decide whether the trade is worth it.

## Score Live Behavior, But Do Not Confuse It With Ground Truth

Mastra can also attach scorers directly to agents and workflow steps. Live scorers run asynchronously and store score results in your configured database, with sampling controls so you do not score every production response unless you mean to.

That is useful, but it is a different job.

Live scoring can tell you the router is still emitting valid decisions. It can catch malformed output, toxic content, forbidden tool calls, missing evidence markers, or suspiciously low confidence.

It usually cannot tell you route accuracy, because production traffic does not arrive with ground truth stapled to it.

That distinction matters. Live scoring is monitoring. Dataset experiments are controlled tests. You want both, but they answer different questions.

## What to Measure After Route Accuracy

Route accuracy is the first rung. It tells you whether the request went to the expected specialist. It does not tell you whether the specialist did good work.

After the router passes the basics, score the system in layers:

| Layer | What to score | Why it matters |
|---|---|---|
| Router decision | selected route, confidence, reason | Catches misclassification and bad escalation rules |
| Trajectory | expected tool or agent sequence | Catches “right answer, wrong path” behavior |
| Specialist output | correctness, faithfulness, usefulness | Catches low-quality work after correct routing |
| Cost and latency | model choice, tokens, runtime | Catches expensive or slow wins |
| Safety and scope | allowed tools, refusal boundaries, evidence | Catches product-risk failures |

Mastra’s `runEvals`

API supports agent-level, workflow-level, step-level, and trajectory scorer configurations. That means you do not have to pretend the final answer is the only artifact.

For a workflow, the shape can look like this:

That is the mental model I want for agents in production:

Score the decision. Score the path. Score the answer.

If you only score the answer, the model can pass by accident.

## The Router Should Get More Boring Over Time

The first routing prompt is usually a paragraph of judgment calls. That is fine for a prototype.

As you learn from evals, parts of the router should become less magical:

- Clear lexical cases can become deterministic rules.
- Risky tasks can require explicit approval or a workflow branch.
- Ambiguous tasks can ask a clarifying question instead of guessing.
- Expensive routes can require higher confidence or a second signal.
- Known failure cases can become dataset items.

The goal is not to make the router “smarter” forever. The goal is to make the system easier to reason about.

Sometimes that means a better model. Sometimes it means a tighter prompt. Sometimes it means a workflow step, a scorer, a hard cap, or a boring `if`

statement that saves you four figures a month.

That is the whole point of measuring behavior. You stop arguing from taste and start arguing from evidence.

## A Practical Starting Checklist

If you are building a Mastra router today, I would start here:

- Make the routing decision structured, even if users never see it.
- Write deterministic scorers for valid JSON, expected route, and forbidden routes.
- Use
`runEvals`

with 10-20 cases before changing router prompts or models. - Promote real failures into a versioned dataset.
- Run dataset experiments for meaningful prompt, model, route, or workflow changes.
- Add live scorers for cheap production invariants.
- Compare experiments by route, not only by average score.

The average matters less than the failure cluster.

If every regression is in long-context policy synthesis, you do not have “a worse router.” You have a route boundary problem. If every failed case uses a specific tool, you have a tool contract problem. If every cheap model fails the same two ambiguous cases, you may need escalation logic instead of a more expensive default.

This is where evals become useful. Not as a ceremony. Not as a dashboard that makes everyone feel temporarily adult.

As a way to find the shape of the system.

## Resources

[Mastra scorers overview](https://mastra.ai/docs/evals/overview)[Mastra](https://mastra.ai/reference/evals/create-scorer)`createScorer`

reference[Mastra](https://mastra.ai/reference/evals/run-evals)`runEvals`

reference[Mastra datasets overview](https://mastra.ai/docs/evals/datasets/overview)[Mastra dataset experiments](https://mastra.ai/docs/evals/datasets/running-experiments)[Don’t Marry Your Model](/llm-routing-mastra-ai)[Fight Evils with Evals!](/llm-evals-are-broken)