{"slug": "most-ai-work-can-wait", "title": "Most AI Work Can Wait", "summary": "Most AI agent teams prioritize model selection over system architecture, but the key to cost efficiency is a router that directs 70-80% of traffic to local or async models, cutting AI spend by 90%. Coinbase reduced AI costs by half through better routing, caching, and defaults, while token usage grew exponentially.", "body_md": "Most teams building agents pick the model first & the architecture second. That is backwards. The model choice is the last decision, not the first.\n\nWhat matters is the router, a small piece of code that decides which tier of model handles each request. Get the router right & 70-80% of traffic runs on local models that cost nothing per call, or on async models 1 that reduce AI spend by 90%+.\n\nBrian Armstrong made the same point last week about how Coinbase cut AI spend in half while token usage grew 2, paraphrasing :\n\nHow to keep AI spend flat while token usage grows exponentially : not with friction & spend alerts. With better defaults, routing, & caching. Engineers can choose any model they want, but defaults matter.\n\nThe routing problem has three layers, and each does a distinct job :\n\n**Skill classifier** turns a raw user request into a concrete operation. It answers what the task is. Draft-a-reply, summarize-a-repo, run-a-migration. The classifier is intent recognition.**Router** decides which tier executes the classified operation. It answers which model runs it. The router does not read the prompt. It reads the classifier’s label plus a few features : complexity, context size, historical success rate.**Model selector** picks the cheapest model within a tier that meets a confidence threshold.\n\nClassifier & router are not the same. The classifier is a language problem ; the router is a scheduling problem. Conflating them buries the model choice inside the prompt & kills the ability to A/B different models against the same operation.\n\nLocal compute is close to free. Async batch reasoning runs two orders of magnitude cheaper than real-time inference 1. So the real question is narrower : what fraction of work needs real-time answers?\n\nSurprisingly little, once the system can queue work.\n\nQueueing is why this works. A draft reply, a repo summary, a diligence memo, a nightly evaluator run : none of these need to return in a second.\n\nWe built the first version of this into our agent runtime. The router already scored tasks on complexity, context size, & local memory retrieval. Two feedback mechanisms now sit on top of the router, & they operate on different time scales :\n\n**Synchronous failure-mode signals.** A predictor annotates each incoming route with five features : missing repo context, long dependency chains, risky migrations, security-sensitive prompts, & high-consequence writes.**Nightly closed-loop feedback.** A batch evaluator scores yesterday’s traces overnight & updates the router’s weights, running on[async inference on Sail](/sail-inference-queue/)to keep the evaluation cost near zero.\n\nThe synchronous predictor catches known-hard tasks before they fail. The nightly loop discovers new failure modes the predictor missed.\n\nOnce skill distillation flattens the operation set, 70-80% of agent traffic can run on local models 3 for most non-coding work.\n\nThe implication : design your system around routing, not around models. Pick your models last.\n\n-\n[Full Sail on Asynchronous Inference](/sail-inference-queue/)— the cost delta between real-time & async batch inference.[↩︎](#fnref:1)[↩︎](#fnref1:1) -\n[Brian Armstrong on X](https://x.com/brian_armstrong/status/2070670644577280109)— Coinbase cut AI spend nearly in half while token usage grew, via better defaults, routing, & caching.[↩︎](#fnref:2) -\n[Skill Distillation](/the-pi-agent-skill-distillation/),[Teaching Local Models to Call Tools Like Claude](/distilling-claude-into-local-models/), &[The Minimill of AI](/using-local-ai-to-work-faster/).[↩︎](#fnref:3)", "url": "https://wpnews.pro/news/most-ai-work-can-wait", "canonical_source": "https://www.tomtunguz.com/ai-execution-routing/", "published_at": "2026-07-01 00:00:00+00:00", "updated_at": "2026-07-07 01:53:35.162222+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-agents", "ai-infrastructure", "ai-products"], "entities": ["Brian Armstrong", "Coinbase", "Sail"], "alternates": {"html": "https://wpnews.pro/news/most-ai-work-can-wait", "markdown": "https://wpnews.pro/news/most-ai-work-can-wait.md", "text": "https://wpnews.pro/news/most-ai-work-can-wait.txt", "jsonld": "https://wpnews.pro/news/most-ai-work-can-wait.jsonld"}}