{"slug": "building-maestro-ai-routing-llm-calls-so-your-agent-doesn-t-burn-sonnet-on", "title": "Building Maestro AI: Routing LLM Calls So Your Agent Doesn't Burn Sonnet on Summaries", "summary": "A developer built Maestro AI, a harness-agnostic model router that routes LLM calls to appropriate models based on task complexity, preventing expensive models like Claude Sonnet from being used for simple tasks. The system uses deterministic classification rules, a decision tree with fallback tiers, and MCP tools for integration with coding agents like Cursor and Claude Code. Key challenges included false positives from ambiguous keywords, meta-prompt contamination, and zombie LiteLLM processes, which were addressed with bounded patterns, explicit task instructions, and diagnostic tools.", "body_md": "*How I built a harness-agnostic model router for Cursor and Claude Code and what broke along the way.*\n\nWhen you use Claude Sonnet for everything in Cursor, you pay premium prices for work a local Llama could handle in two seconds. When you use only Ollama, you get worse results on architecture reviews and multi-step refactors.\n\nI wanted a middle layer: **cheap by default, escalate when the task actually needs it.** Not a new chatbot, a dispatcher the existing agent could call for subtasks.\n\nThat layer became ** Maestro AI**.\n\nThink of your coding agent as a conductor. It should keep architecture, multi-file edits, and complex reasoning. Maestro is the section leader who picks which musician plays each part:\n\nThe conductor doesn't stop conducting. It delegates the small solos.\n\nCursor picks one model per chat. Agent sessions spawn dozens of implicit subtasks; summarize this file, rewrite this message, extract these fields, format this JSON. The harness doesn't automatically route those to cheaper models.\n\nMaestro adds **per-call routing** with:\n\nWe classify prompts with deterministic rules: keywords, patterns, tool presence, context size. Examples:\n\n`\"summarize this README\"`\n\n→ `summarization`\n\n, easy, low risk → `local_strong`\n\n`\"Design system architecture for event sourcing\"`\n\n→ `architecture`\n\n, hard → `premium`\n\n`\"make me an HTML demo page\"`\n\n→ `code_edit`\n\n, easy → `local_fast`\n\nEarly versions had painful false positives. The word **\"design\"** in \"design a simple HTML landing page\" triggered architecture routing and sent demo pages to Sonnet. We fixed that with bounded patterns: `isSimpleUiTask()`\n\n, `isDemoShowcaseTask()`\n\n, and stricter `SYSTEM_ARCHITECTURE_SIGNALS`\n\nvs casual \"design\" usage.\n\nAnother gotcha: agents sent **meta-prompts** to the router, *\"Determine routing for building a demonstration of model-router capabilities\"*, instead of the literal task. That hit architecture keywords again. MCP tool descriptions now explicitly say: pass the literal task, not a routing meta-description.\n\nRouting is a decision tree:\n\n```\nhard / high-risk / tool+code / long context / architecture → premium\nmedium coding / debugging / refactoring → hosted_oss\nsummarization / rewriting / extraction → local_strong\neasy / formatting / simple UI → local_fast\n```\n\nThen overlays:\n\n`max_tier`\n\n`budget_usd`\n\n`always_prefer_local`\n\nOur stack:\n\n`:11434`\n\n- llama3.2, qwen3:8b`:4000`\n\n- proxies to Featherless (GLM, Qwen Coder) and Bedrock (Sonnet)Config shape:\n\n```\n\"local_strong\": {\n  \"primary\": { \"provider\": \"litellm\", \"model\": \"glm\", \"baseUrl\": \"http://localhost:4000/v1\" },\n  \"fallback\": { \"provider\": \"ollama\", \"model\": \"qwen3:8b\", \"baseUrl\": \"http://localhost:11434/v1\" }\n}\n```\n\nWhen LiteLLM dies, `local_strong`\n\nstays on-tier via Ollama; it doesn't jump to premium or drop to tiny Llama for a summarization task.\n\nWe learned this the hard way: a **zombie LiteLLM process** existed but wasn't listening on `:4000`\n\n. Routing failed mysteriously until we built `maestro doctor`\n\n; process check, port check, `/v1/models`\n\n, `FEATHERLESS_API_KEY`\n\n, per-tier endpoint probes.\n\n`routedLLMCall()`\n\ndoesn't fire-and-forget. After each call:\n\n`completion_tokens > 0`\n\nbut no visible text? Fail and escalate.If checks fail → retry same tier (once) → escalate to next tier → log telemetry.\n\nWe hit a real bug here: GLM returned an **empty string after 130 seconds**, but `non_empty`\n\npassed. Root cause cluster:\n\n`trim().length > 0`\n\nlet through `\\u0000`\n\nand zero-width characters`reasoning_content`\n\nor array parts we didn't parse`ProviderError(\"empty\")`\n\ndidn't trigger escalationFix: shared `content-extract.ts`\n\n, `hasMeaningfulContent()`\n\n, explicit empty escalation, `content_integrity`\n\ncheck for token-without-text.\n\nAgents don't shell out reliably. Maestro exposes MCP tools:\n\n| Tool | Purpose |\n|---|---|\n`maestro_route` |\nDry-run routing; no LLM call |\n`maestro_ask` |\nRoute + execute + auto-escalate |\n`maestro_doctor` |\nInfrastructure diagnostics |\n`maestro_stats` |\nTelemetry dashboard |\n`maestro_feedback` |\nThumbs up/down for tuning |\n\nEvery route/ask response includes a **full routing report**: analysis, debug trace, probe status, fallback reason. We learned that hiding debug behind `debug: true`\n\ncost two debugging rounds; visibility is now always on.\n\nPass once per session:\n\n```\n{\n  \"session_id\": \"cursor-main\",\n  \"budget_usd\": 0.50,\n  \"max_tier\": \"hosted_oss\",\n  \"always_prefer_local\": true\n}\n```\n\nBudget is **enforced** - telemetry sums spend per `session_id`\n\n, caps tier selection, blocks escalation when exhausted.\n\nMaestro is published on npm as [ maestro-ai](https://www.npmjs.com/package/maestro-ai). You don't need to clone the repo — the package ships the compiled CLI, MCP server, and bundled config profiles.\n\n```\nnpm install -g maestro-ai\n\nmaestro init --profile ollama-only   # or default / cloud-only\nollama pull llama3.2:latest\nollama pull qwen3:8b\nmaestro doctor\nmaestro route \"summarize this paragraph\" --debug\n```\n\nTwo binaries are exposed:\n\n| Command | What it runs |\n|---|---|\n`maestro` |\nCLI — `init` , `route` , `ask` , `doctor` , `stats` , … |\n`maestro-mcp` |\nMCP server for Cursor / Claude Code |\n\n`npm install`\n\nruns `prepare`\n\n, which builds `dist/`\n\nfrom TypeScript — consumers get a ready-to-run package, not raw source.\n\n```\nnpx maestro-ai init --profile ollama-only\nnpx maestro route \"rewrite this commit message\" --debug\nnpx maestro-mcp   # run MCP server once (stdio)\n```\n\n`npx`\n\ndownloads the package on first use and caches it. Good for trying Maestro or CI scripts that need a single routed call.\n\nAfter `maestro init`\n\n, check `~/.maestro-ai/mcp-config.json`\n\n. For npm installs it typically looks like:\n\n```\n{\n  \"mcpServers\": {\n    \"maestro-ai\": {\n      \"command\": \"npx\",\n      \"args\": [\"-y\", \"maestro-mcp\"],\n      \"env\": {\n        \"MAESTRO_CONFIG\": \"/Users/you/.maestro-ai/config.json\",\n        \"LITELLM_MASTER_KEY\": \"sk-litellm-local\"\n      }\n    }\n  }\n}\n```\n\nNo hardcoded clone paths — `npx maestro-mcp`\n\nresolves the binary from the published package. Merge that block into Cursor → Settings → MCP and reload.\n\n```\nnpm install maestro-ai\njs\nimport { routedLLMCall, dryRunRoute } from \"maestro-ai\";\n\nconst preview = await dryRunRoute({\n  messages: [{ role: \"user\", content: \"Extract dates from this email.\" }],\n});\n\nconsole.log(preview.routing.tier, preview.routing.model);\n\nconst result = await routedLLMCall({\n  messages: [{ role: \"user\", content: \"Extract dates from this email.\" }],\n  overrides: {\n    session: { sessionId: \"my-app\", maxTier: \"hosted_oss\", budgetUsd: 0.25 },\n  },\n});\n\nconsole.log(result.response.content);\nconsole.log(result.telemetryId);\n```\n\nConfig and telemetry still live under `~/.maestro-ai/`\n\n(created by `maestro init`\n\n). Override with `MAESTRO_CONFIG`\n\nif you want a project-local config file.\n\nnpm |\ngit clone |\n|\n|---|---|---|\n| Best for | Using Maestro, MCP, CLI | Contributing, patching routing rules |\n| Install | `npm install -g maestro-ai` |\n`git clone` + `npm install`\n|\n| MCP | `npx maestro-mcp` |\n`node dist/mcp-server.js` in clone |\n| Updates | `npm update -g maestro-ai` |\n`git pull` + `npm run build`\n|\n\nWe dogfood the npm package locally too: `npm pack`\n\nproduces a tarball you can install with `npm install -g ./maestro-ai-0.4.1.tgz`\n\nto verify the published artifact before release.\n\nPersonal hardcoded paths don't scale. `maestro init`\n\n:\n\n`~/.maestro-ai/`\n\n`default`\n\n, `ollama-only`\n\n, `cloud-only`\n\n)`npx maestro-mcp`\n\nwhen npm-installed)`ollama pull`\n\nmodels`.env.example`\n\nColleagues can run `maestro init --profile ollama-only`\n\nwith **no LiteLLM, no cloud keys**; just Ollama.\n\nRules stay the floor; ML can sit on top later.\n\n| Choice | Why |\n|---|---|\n| TypeScript standalone package | Fits Cursor MCP, CLI, npm, any Node harness |\nRaw `fetch` to `/v1/chat/completions`\n|\nOllama and LiteLLM are OpenAI-compatible |\n| JSONL telemetry | Simple, grep-friendly, no DB |\n| Vitest | Fast, 77 tests covering routing edge cases |\n| MCP over custom HTTP | Agents already speak MCP natively |\n\nNo LangChain. The router is ~27 source files; you can read the whole thing in an afternoon.\n\nWhat works well:\n\n`maestro doctor`\n\ncatching zombie processes before routing failsWhat's still manual:\n\n```\ngit clone https://github.com/David-J-Shibley/maestro-ai.git\ncd maestro-ai && npm install\nmaestro init --profile ollama-only\nollama pull llama3.2:latest && ollama pull qwen3:8b\nmaestro doctor\nmaestro route \"summarize this paragraph\" --debug\n```\n\nMerge `~/.maestro-ai/mcp-config.json`\n\ninto Cursor MCP settings. In chat, the agent can call `maestro_ask`\n\nfor cheap subtasks while keeping hard work in its own session.\n\n`maestro init`\n\nmattered more than another routing featureHarness wiring (Benchy, Claude Code hooks), feedback-driven stats, premium pool rotation, and eventually learned routing once telemetry has enough rows.\n\n*Maestro AI is open source (MIT): github.com/David-J-Shibley/maestro-ai*", "url": "https://wpnews.pro/news/building-maestro-ai-routing-llm-calls-so-your-agent-doesn-t-burn-sonnet-on", "canonical_source": "https://dev.to/david_shibley/building-maestro-ai-routing-llm-calls-so-your-agent-doesnt-burn-sonnet-on-summaries-1m36", "published_at": "2026-07-12 16:19:08+00:00", "updated_at": "2026-07-12 16:45:01.924362+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-agents", "developer-tools", "ai-infrastructure"], "entities": ["Maestro AI", "Cursor", "Claude Code", "Claude Sonnet", "Ollama", "LiteLLM", "Featherless", "Bedrock"], "alternates": {"html": "https://wpnews.pro/news/building-maestro-ai-routing-llm-calls-so-your-agent-doesn-t-burn-sonnet-on", "markdown": "https://wpnews.pro/news/building-maestro-ai-routing-llm-calls-so-your-agent-doesn-t-burn-sonnet-on.md", "text": "https://wpnews.pro/news/building-maestro-ai-routing-llm-calls-so-your-agent-doesn-t-burn-sonnet-on.txt", "jsonld": "https://wpnews.pro/news/building-maestro-ai-routing-llm-calls-so-your-agent-doesn-t-burn-sonnet-on.jsonld"}}