{"slug": "how-i-cut-agent-token-usage-by-89-without-touching-the-agent", "title": "How I Cut Agent Token Usage by 89% Without Touching the Agent", "summary": "A developer built a Go proxy called Trooper that sits between AI agents and large language models, reducing token usage by 89% without modifying the agent itself. The proxy replaces full conversation history with a structured \"SITREP\" — a situation report capturing intent, decisions, constraints, and open questions — after the first few turns of a session. In a real 15-turn debugging session, Trooper cut token consumption from 10,820 tokens per request to just 1,157 tokens while maintaining the model's ability to answer questions about earlier decisions.", "body_md": "Every time your agent calls an LLM, it sends the full conversation history.\n\nTurn 20 includes turns 1–19. Turn 50 includes turns 1–49. Nobody notices because it's happening inside the agent, silently, on every single request.\n\nI noticed it while building [Trooper](https://github.com/shouvik12/trooper) - a Go proxy that sits between agents and LLMs. I was watching token counts climb across a long debugging session and realised the agent was replaying the same context over and over. Most of it was noise.\n\nThe model didn't need a transcript. It needed state.\n\nAfter a few turns, most of what matters in a session falls into four categories:\n\nThat's it. Everything else — the back and forth, the verbose LLM responses explaining things, the repeated context — is replay. The model doesn't need it again.\n\nI added structured session memory to Trooper. After enough turns, Trooper's local Llama model generates a SITREP — a situation report — from the user messages in the session.\n\nIt looks like this:\n\n```\nINTENT: Build a RAG pipeline with ChromaDB and nomic-embed-text\n\nDECISIONS: Use cosine similarity over MMR — focused queries not broad;\n           Chunk size 256, overlap 30 — locked;\n           Pure vector search — ChromaDB no hybrid support;\n           Top k set to 5\n\nCONSTRAINTS: Node 18 locked — platform team constraint, no exceptions;\n             Re-ranking ruled out — latency jumped 200ms to 800ms\n\nOPEN: Poor recall on technical queries — nomic-embed-text struggles with domain jargon;\n      Evaluating bge-small as alternative\n```\n\nFrom that point forward, every request to the LLM sends:\n\n```\nAnchor (first 2 turns verbatim)\n+ SITREP (structured state)\n+ Tail (last N turns verbatim)\n```\n\nInstead of the full history.\n\nFrom a real 15-turn session:\n\n```\nFull history:    10,820 tokens per request\nWith Trooper:     1,157 tokens per request\nReduction:             89%\n```\n\nVisible live on the dashboard.\n\nThis was the question that mattered. Token savings are worthless if the model loses coherence.\n\nTo test it: I took the auto-generated SITREP, opened a completely fresh chat with no history, and asked questions about decisions made in the original session.\n\n**Questions:**\n\n**Result:** All four answered correctly. The model worked entirely from the SITREP. No history. No context bleed.\n\nThat's the claim: structured state is sufficient for the model to continue reasoning correctly — and it costs 89% less to send.\n\nTrooper is a Go proxy — one binary, no SDK, no instrumentation. You point your existing agent at it by changing one URL.\n\n```\n# Before\nexport ANTHROPIC_BASE_URL=https://api.anthropic.com\n\n# After\nexport ANTHROPIC_BASE_URL=http://localhost:3000\n```\n\nNothing else changes. Trooper intercepts every request, maintains session state, and when the SITREP is ready, rewrites the messages array before forwarding to the LLM.\n\nThe SITREP is built by a local Llama 3.1 8b model running via Ollama — fast, private, no cloud cost. The extraction happens asynchronously in the background. The main request path is not blocked.\n\n```\n// GetTripleAnchor assembles what gets sent to the LLM\nfunc (s *SessionStore) GetTripleAnchor(sessionID string) []map[string]string {\n    payload := append([]map[string]string{}, state.Anchor...)\n    if state.SITREP != \"\" {\n        payload = append(payload, map[string]string{\n            \"role\":    \"system\",\n            \"content\": fmt.Sprintf(\"[STATE_SITREP: %s]\", state.SITREP),\n        })\n    }\n    return append(payload, state.Tail...)\n}\n```\n\nThe dashboard shows the compression ratio live:\n\n```\nHISTORY COMPRESSED    89%\nTOKENS SAVED          459\nCONFIDENCE            100%\n```\n\nMost summarisation tools compress what was said. The SITREP extracts what matters for the next action.\n\nCopilot's context compaction summarises the full conversation — useful for humans in long chats. The SITREP is structured specifically for agents: decisions, constraints, open loops, ruled-out paths. Not a narrative summary. A state snapshot.\n\nThe result is that subsequent turns stay coherent on intent without replaying noise. More relevant for agents running repeated structured workflows than for general chat.\n\nThe SITREP works best for structured agentic workflows — debugging sessions, research pipelines, multi-step build tasks. For open-ended creative work where tangential context might become important later, you'd want a larger tail window or higher fidelity compression.\n\nThe tail window is configurable. You can keep more raw context for less structured sessions.\n\nThe compression is the latest addition. Trooper also:\n\n`x_force_local`\n\n`/recovery/{session_id}`\n\ntells you exactly where to resumeAll from one URL change.\n\nWe tend to treat conversation history as memory. But a transcript is a log. Memory is state.\n\nHumans don't replay every prior conversation before making a decision. They carry forward conclusions, constraints, unresolved questions, and relevant context — a structured snapshot, not a full transcript.\n\nLong-running agents may need to do the same. Not because of token costs — though that helps — but because state is a better abstraction for agent memory than history.\n\nThe SITREP is an experiment in that direction.\n\n[github.com/shouvik12/trooper](https://github.com/shouvik12/trooper) — Go, MIT, zero dependencies beyond Ollama.", "url": "https://wpnews.pro/news/how-i-cut-agent-token-usage-by-89-without-touching-the-agent", "canonical_source": "https://dev.to/shouvik12/how-i-cut-agent-token-usage-by-89-without-touching-the-agent-3g7o", "published_at": "2026-06-05 13:10:22+00:00", "updated_at": "2026-06-05 13:42:53.189449+00:00", "lang": "en", "topics": ["ai-agents", "large-language-models", "ai-tools", "ai-infrastructure", "mlops"], "entities": ["Trooper", "ChromaDB", "nomic-embed-text", "Llama", "Node 18"], "alternates": {"html": "https://wpnews.pro/news/how-i-cut-agent-token-usage-by-89-without-touching-the-agent", "markdown": "https://wpnews.pro/news/how-i-cut-agent-token-usage-by-89-without-touching-the-agent.md", "text": "https://wpnews.pro/news/how-i-cut-agent-token-usage-by-89-without-touching-the-agent.txt", "jsonld": "https://wpnews.pro/news/how-i-cut-agent-token-usage-by-89-without-touching-the-agent.jsonld"}}