{"slug": "stop-streaming-tools-through-your-llm", "title": "Stop Streaming Tools Through Your LLM", "summary": "A new agent pattern called Code Mode is replacing traditional tool-calling in LLM agents, cutting context window usage by up to 99.4% in a demo. Instead of streaming every tool definition and intermediate result through the model, the agent writes a single script that calls tools directly inside a sandbox, with only the final answer returned. The approach, adopted by Anthropic and Cloudflare, reduces token consumption from tens of thousands to a few hundred for the same task.", "body_md": "*The 2026 shift from tool-calling to Code Mode: let the agent write one script instead of narrating fifty tool calls — and watch context tokens drop ~99%.*\n\n**TL;DR:** The classic agent loop loads *every* tool definition into the context window and pipes *every* intermediate result back through the model. Connect a few dozen tools and the context is full before the user even speaks. The 2026 move — **Code Mode** — is to let the agent write one short script that calls tools directly; bulk data stays in a sandbox and only the final answer comes back. In a tiny runnable demo it cut context from **36,781 tokens to 222** — a **99.4%** reduction. Same answer, no API key.\n\n**Mental model:** instead of reading a worker every page of 50 manuals and every row of a spreadsheet out loud, you hand them the 3 manuals the job needs and let them do the math at their own desk. You only get back the final answer — not the raw data.\n\nFunction/tool-calling is how most agents act today. It works beautifully with five tools. But the whole tool surface gets serialized into the context window *on every request*, and every intermediate result the model asks for is streamed back through the context too.\n\nSo a task like *\"count open support tickets per plan tier and save the report\"* looks like:\n\n`list_tickets`\n\n→ 2,000 rows come back `get_customers`\n\n→ 400 rows come back `save_report`\n\n.The model paid for 50 schemas it mostly didn't need, and for 2,400 rows of raw data it only needed to *aggregate*, not *read*. Anthropic measured a real Google-Drive-to-Salesforce task at **150,000 tokens**; Cloudflare hit **~1.17M tokens** of tool definitions on a 2,500-endpoint API.\n\nCode Mode (Anthropic's \"code execution with MCP\", Cloudflare's \"Code Mode\") flips the loop:\n\nHere's the entire script the \"model\" writes in the demo — the 2,000 tickets and 400 customers are joined *inside the sandbox* and never touch the context:\n\n```\ntickets = list_tickets(\"open\")            # 2000 rows: fetched and joined entirely in the sandbox\nplan = {c[\"id\"]: c[\"plan\"] for c in get_customers()}\ncounts = {t: 0 for t in (\"free\", \"pro\", \"enterprise\")}\nfor tk in tickets:\n    counts[plan[tk[\"customer_id\"]]] += 1\nrows = [{\"tier\": t, \"open_tickets\": counts[t]} for t in counts]\nresult = save_report(\"open_by_tier\", rows)\n```\n\nThe sandbox exposes only the tool API — no builtins, no imports — so the script can compose tools but can't reach the rest of the process:\n\n```\napi = {\"list_tickets\": list_tickets, \"get_customers\": get_customers, \"save_report\": save_report}\nsandbox = {\"__builtins__\": {}, **api}\nexec(script, sandbox)      # only the script text + final answer ever cross the context window\nCode Mode — write code that calls tools, don't stream tools through the model\n\n  Task: count open tickets per plan tier over 2000 tickets / 400 customers.\n  Tools connected to the agent: 50 (this task needs 3).\n\n   classic tool-calling    36,781 context tokens   (all schemas + raw data pass through)\n   code mode                  222 context tokens   (3 signatures + one script + answer)\n   ------------------------------------------------\n   context reduction        99.4%\n\n  Same result either way: [free: 720, pro: 596, enterprise: 684]\n```\n\nThe gap isn't a constant — it **compounds** with tool count and data size.\n\n**Reality check:** the exact figure is from the toy model above — treat it as directional, not a benchmark. The *shape* is real and measured in production: Anthropic's Drive→Salesforce task dropped from ~150k to ~2k tokens, and independent reproductions land anywhere from 78% to 99.9% depending on tool count and data size.\n\nThe proven part: LLMs are extremely good at writing code, and a code API is a far denser way to express \"do these five things and combine the results\" than five separate tool-call round trips. Anthropic, Cloudflare, and an [independent study of MCP design choices](https://arxiv.org/abs/2602.15945) all converge on the same finding — token usage becomes roughly **constant in tool count** because the model only reads what it opens.\n\nWhere it's heading: as agents connect to hundreds of MCP servers, \"load everything up front\" simply stops being viable, and the tool-calling layer moves *onto the computer*. Expect the sandbox — not the tool-call — to become the default action primitive, with tool schemas synced to a filesystem and disclosed on demand.\n\nIt's a minimal model, not a benchmark: \"tokens\" are a `chars/4`\n\nproxy, the \"model\" writes a fixed script, and the sandbox is a restricted `exec`\n\n. Real systems generate the script with an LLM and isolate it far more aggressively (gVisor, containers, V8 isolates) — which is the pattern's main cost: **you now run untrusted, model-written code**, so sandboxing and limits are mandatory. The token economics, though, are exactly what production reports show.\n\n```\npython3 demo.py   # standard library only\n```\n\n**Papers**\n\n**Engineering blogs**", "url": "https://wpnews.pro/news/stop-streaming-tools-through-your-llm", "canonical_source": "https://dev.to/shridhar_shah2297/stop-streaming-tools-through-your-llm-2f1i", "published_at": "2026-08-05 16:19:51+00:00", "updated_at": "2026-08-05 17:02:59.208830+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-agents", "ai-infrastructure", "developer-tools"], "entities": ["Anthropic", "Cloudflare", "MCP"], "alternates": {"html": "https://wpnews.pro/news/stop-streaming-tools-through-your-llm", "markdown": "https://wpnews.pro/news/stop-streaming-tools-through-your-llm.md", "text": "https://wpnews.pro/news/stop-streaming-tools-through-your-llm.txt", "jsonld": "https://wpnews.pro/news/stop-streaming-tools-through-your-llm.jsonld"}}