Building Maestro AI: Routing LLM Calls So Your Agent Doesn't Burn Sonnet on Summaries

A developer built Maestro AI, a harness-agnostic model router that routes LLM calls to appropriate models based on task complexity, preventing expensive models like Claude Sonnet from being used for simple tasks. The system uses deterministic classification rules, a decision tree with fallback tiers, and MCP tools for integration with coding agents like Cursor and Claude Code. Key challenges included false positives from ambiguous keywords, meta-prompt contamination, and zombie LiteLLM processes, which were addressed with bounded patterns, explicit task instructions, and diagnostic tools.

How I built a harness-agnostic model router for Cursor and Claude Code and what broke along the way. When you use Claude Sonnet for everything in Cursor, you pay premium prices for work a local Llama could handle in two seconds. When you use only Ollama, you get worse results on architecture reviews and multi-step refactors. I wanted a middle layer: cheap by default, escalate when the task actually needs it. Not a new chatbot, a dispatcher the existing agent could call for subtasks. That layer became Maestro AI . Think of your coding agent as a conductor. It should keep architecture, multi-file edits, and complex reasoning. Maestro is the section leader who picks which musician plays each part: The conductor doesn't stop conducting. It delegates the small solos. Cursor picks one model per chat. Agent sessions spawn dozens of implicit subtasks; summarize this file, rewrite this message, extract these fields, format this JSON. The harness doesn't automatically route those to cheaper models. Maestro adds per-call routing with: We classify prompts with deterministic rules: keywords, patterns, tool presence, context size. Examples: "summarize this README" → summarization , easy, low risk → local strong "Design system architecture for event sourcing" → architecture , hard → premium "make me an HTML demo page" → code edit , easy → local fast Early versions had painful false positives. The word "design" in "design a simple HTML landing page" triggered architecture routing and sent demo pages to Sonnet. We fixed that with bounded patterns: isSimpleUiTask , isDemoShowcaseTask , and stricter SYSTEM ARCHITECTURE SIGNALS vs casual "design" usage. Another gotcha: agents sent meta-prompts to the router, "Determine routing for building a demonstration of model-router capabilities" , instead of the literal task. That hit architecture keywords again. MCP tool descriptions now explicitly say: pass the literal task, not a routing meta-description. Routing is a decision tree: hard / high-risk / tool+code / long context / architecture → premium medium coding / debugging / refactoring → hosted oss summarization / rewriting / extraction → local strong easy / formatting / simple UI → local fast Then overlays: max tier budget usd always prefer local Our stack: :11434 - llama3.2, qwen3:8b :4000 - proxies to Featherless GLM, Qwen Coder and Bedrock Sonnet Config shape: "local strong": { "primary": { "provider": "litellm", "model": "glm", "baseUrl": "http://localhost:4000/v1" }, "fallback": { "provider": "ollama", "model": "qwen3:8b", "baseUrl": "http://localhost:11434/v1" } } When LiteLLM dies, local strong stays on-tier via Ollama; it doesn't jump to premium or drop to tiny Llama for a summarization task. We learned this the hard way: a zombie LiteLLM process existed but wasn't listening on :4000 . Routing failed mysteriously until we built maestro doctor ; process check, port check, /v1/models , FEATHERLESS API KEY , per-tier endpoint probes. routedLLMCall doesn't fire-and-forget. After each call: completion tokens 0 but no visible text? Fail and escalate.If checks fail → retry same tier once → escalate to next tier → log telemetry. We hit a real bug here: GLM returned an empty string after 130 seconds , but non empty passed. Root cause cluster: trim .length 0 let through \u0000 and zero-width characters reasoning content or array parts we didn't parse ProviderError "empty" didn't trigger escalationFix: shared content-extract.ts , hasMeaningfulContent , explicit empty escalation, content integrity check for token-without-text. Agents don't shell out reliably. Maestro exposes MCP tools: | Tool | Purpose | |---|---| maestro route | Dry-run routing; no LLM call | maestro ask | Route + execute + auto-escalate | maestro doctor | Infrastructure diagnostics | maestro stats | Telemetry dashboard | maestro feedback | Thumbs up/down for tuning | Every route/ask response includes a full routing report : analysis, debug trace, probe status, fallback reason. We learned that hiding debug behind debug: true cost two debugging rounds; visibility is now always on. Pass once per session: { "session id": "cursor-main", "budget usd": 0.50, "max tier": "hosted oss", "always prefer local": true } Budget is enforced - telemetry sums spend per session id , caps tier selection, blocks escalation when exhausted. Maestro is published on npm as maestro-ai https://www.npmjs.com/package/maestro-ai . You don't need to clone the repo — the package ships the compiled CLI, MCP server, and bundled config profiles. npm install -g maestro-ai maestro init --profile ollama-only or default / cloud-only ollama pull llama3.2:latest ollama pull qwen3:8b maestro doctor maestro route "summarize this paragraph" --debug Two binaries are exposed: | Command | What it runs | |---|---| maestro | CLI — init , route , ask , doctor , stats , … | maestro-mcp | MCP server for Cursor / Claude Code | npm install runs prepare , which builds dist/ from TypeScript — consumers get a ready-to-run package, not raw source. npx maestro-ai init --profile ollama-only npx maestro route "rewrite this commit message" --debug npx maestro-mcp run MCP server once stdio npx downloads the package on first use and caches it. Good for trying Maestro or CI scripts that need a single routed call. After maestro init , check ~/.maestro-ai/mcp-config.json . For npm installs it typically looks like: { "mcpServers": { "maestro-ai": { "command": "npx", "args": "-y", "maestro-mcp" , "env": { "MAESTRO CONFIG": "/Users/you/.maestro-ai/config.json", "LITELLM MASTER KEY": "sk-litellm-local" } } } } No hardcoded clone paths — npx maestro-mcp resolves the binary from the published package. Merge that block into Cursor → Settings → MCP and reload. npm install maestro-ai js import { routedLLMCall, dryRunRoute } from "maestro-ai"; const preview = await dryRunRoute { messages: { role: "user", content: "Extract dates from this email." } , } ; console.log preview.routing.tier, preview.routing.model ; const result = await routedLLMCall { messages: { role: "user", content: "Extract dates from this email." } , overrides: { session: { sessionId: "my-app", maxTier: "hosted oss", budgetUsd: 0.25 }, }, } ; console.log result.response.content ; console.log result.telemetryId ; Config and telemetry still live under ~/.maestro-ai/ created by maestro init . Override with MAESTRO CONFIG if you want a project-local config file. npm | git clone | | |---|---|---| | Best for | Using Maestro, MCP, CLI | Contributing, patching routing rules | | Install | npm install -g maestro-ai | git clone + npm install | | MCP | npx maestro-mcp | node dist/mcp-server.js in clone | | Updates | npm update -g maestro-ai | git pull + npm run build | We dogfood the npm package locally too: npm pack produces a tarball you can install with npm install -g ./maestro-ai-0.4.1.tgz to verify the published artifact before release. Personal hardcoded paths don't scale. maestro init : ~/.maestro-ai/ default , ollama-only , cloud-only npx maestro-mcp when npm-installed ollama pull models .env.example Colleagues can run maestro init --profile ollama-only with no LiteLLM, no cloud keys ; just Ollama. Rules stay the floor; ML can sit on top later. | Choice | Why | |---|---| | TypeScript standalone package | Fits Cursor MCP, CLI, npm, any Node harness | Raw fetch to /v1/chat/completions | Ollama and LiteLLM are OpenAI-compatible | | JSONL telemetry | Simple, grep-friendly, no DB | | Vitest | Fast, 77 tests covering routing edge cases | | MCP over custom HTTP | Agents already speak MCP natively | No LangChain. The router is ~27 source files; you can read the whole thing in an afternoon. What works well: maestro doctor catching zombie processes before routing failsWhat's still manual: git clone https://github.com/David-J-Shibley/maestro-ai.git cd maestro-ai && npm install maestro init --profile ollama-only ollama pull llama3.2:latest && ollama pull qwen3:8b maestro doctor maestro route "summarize this paragraph" --debug Merge ~/.maestro-ai/mcp-config.json into Cursor MCP settings. In chat, the agent can call maestro ask for cheap subtasks while keeping hard work in its own session. maestro init mattered more than another routing featureHarness wiring Benchy, Claude Code hooks , feedback-driven stats, premium pool rotation, and eventually learned routing once telemetry has enough rows. Maestro AI is open source MIT : github.com/David-J-Shibley/maestro-ai