{"slug": "llm-agent-token-cost-why-a-kept-chat-grows-with-the-square-of-the-steps-and-the", "title": "LLM agent token cost: why a kept chat grows with the square of the steps, and the measured fix (prompt caching counted)", "summary": "Shalendra Gaur and Rajnish Choubey built AXI, a host-side runtime for LLM agents that holds conversation state on the host and shows the model only the exact state, instead of resending the full chat each step. Measured on Gemini 3.8 Flash, the approach cut prompt tokens per step by 85 times and total tokens over 400 turns by 76 times, with 4.5 times lower cost per step that ends in the correct state, even though prompt caching served 97 percent of the kept-chat tokens. On a 14B model, an exact-state step cost 21.8 joules versus 123.1 joules with the whole chat kept.", "body_md": "**An agent that keeps its whole chat resends everything on every step, so the tokens it sends grow with the square of the number of steps. Holding the state on the host and showing the model the exact state keeps the prompt the size of the state. Measured on Gemini 3.8 Flash: 85 times fewer prompt tokens a step and 76 times fewer in all over 400 turns, and 4.5 times lower cost per step that ends with the right state.**\n\n*Naming note: \"AXI\" here is the AXI host-side runtime for LLM agents, an architecture by Rajnish Choubey. It is not the ARM AMBA AXI chip-bus protocol.*\n\nIf step t sends a prompt of a + b·t tokens (the fixed part plus everything said so far), n steps send n·a + b·n(n+1)/2 tokens in all. The second term grows with n squared: twice the turns, about four times the tokens, at the growth rate measured in Test 23. A host that holds the state sends a prompt the size of the state, so n steps send about n times that size: a straight line.\n\nAll runs: Gemini 3.8 Flash, each job run once with the chat kept and once with the host holding the state.\n\n| Job | Prompt tokens a step, chat kept | With a host-held state | Tokens sent in all | \n|---|---|---|---|\n| 400 turns of plain, ambiguous sentences (Test 23) | 99,809 on average | 1,177 (85 times fewer) | 39.9 million against 0.53 million (76 times fewer) | \n| Long job with heavy turns (Test 22) | 476,835 on average, up to 949,591 | 2,824 (169 times fewer) | 217.5 million (457 turns) against 4.6 million (1,500 steps) | \n| Four ARC-AGI-3 games (Test 24) | n/a | n/a | 25.7 million against 1.95 million (13.2 times fewer) | \n\nCaching is the first thing to switch on, and it helps: the service served 97 percent of the kept chat's prompt tokens from its cache. But sent tokens and computed tokens are different bills, so here are both.\n\nCost per token hides the real unit. Over the first 400 requests of a 4,000-request Gemini run, at $0.75 in and $3.75 out per million tokens with thinking counted as output:\n\n| Way of using the model's replies | Steps with the right state (of 400) | Spent | Cost of one step with the right state | \n|---|---|---|---|\n| Host layer: check, complete, commit | 400 | $0.47 | $0.0012 | \n| Replies applied as written | 85 | $0.45 | $0.0052 (4.5 times as much) | \n| A gate that only refuses bad replies | 84 | $0.48 | $0.0057 (4.9 times as much) | \n\nThe spend was about the same. What differed was how many steps ended with the right state.\n\nOn a 14B model (Test 44), a step cost 21.8 joules with the exact state shown, 39.4 to 48.0 joules with summaries, a window or retrieval, and 123.1 joules with the whole chat kept. On one NVIDIA L4 with a 7B model (Test 38), 256 sessions ran with every step exact: 105,230 exact steps per GPU hour at 2.458 joules per exact step.\n\nNo key, no GPU. This replays the saved replies of the 4,000-request run with the repository's own code and prints the 4,000, 3,954, 85 and 4.5 times figures:\n\n```\ngit clone https://github.com/gaurshalendra2412/axi-runtime\ncd axi-runtime/python\npython3 quickstart_replay.py\n```\n\nGemini 3.8 Flash for the sent-token figures; open models on vLLM 0.31.0 and one NVIDIA L4, greedy decoding, no retry; synthetic service graphs; the cost rows are arithmetic added after the results, from the saved runs, not a pre-registered line. Whether the bill follows sent or computed tokens depends on how a company prices cached tokens; the repository does not price them. Joules are the card's board power read during the run. Test pages: [github.com/gaurshalendra2412/axi-runtime](https://github.com/gaurshalendra2412/axi-runtime). White paper: [doi.org/10.5281/zenodo.23958737](https://doi.org/10.5281/zenodo.23958737).\n\n*Architecture by Rajnish Choubey. Tests built and run by Shalendra Gaur.*", "url": "https://wpnews.pro/news/llm-agent-token-cost-why-a-kept-chat-grows-with-the-square-of-the-steps-and-the", "canonical_source": "https://dev.to/shalendra_gaur_ecd0567a49/llm-agent-token-cost-why-a-kept-chat-grows-with-the-square-of-the-steps-and-the-measured-fix-1021", "published_at": "2026-11-18 05:08:38+00:00", "updated_at": "2026-11-18 05:14:32.295569+00:00", "lang": "en", "topics": ["ai-agents", "large-language-models", "ai-infrastructure", "mlops"], "entities": ["Rajnish Choubey", "Shalendra Gaur", "AXI", "Gemini 3.8 Flash", "NVIDIA L4", "vLLM", "ARC-AGI-3"], "also_reported_by": [], "takeaway": "Holding agent state on the host and sending only the exact state cut Gemini 3.8 Flash prompt tokens per step from 99,809 to 1,177 (85x) and total tokens over 400 turns by 76x, with 4.5x lower cost per correct-state step.", "recommendation": {"advice": "Keep agent state on the host and send the model only the exact current state rather than the full chat history; the source measured 85x fewer prompt tokens per step and 4.5x lower cost per correct-state step.", "playbook": "llm-cost", "playbook_url": "https://wpnews.pro/playbooks/llm-cost"}, "alternates": {"html": "https://wpnews.pro/news/llm-agent-token-cost-why-a-kept-chat-grows-with-the-square-of-the-steps-and-the", "markdown": "https://wpnews.pro/news/llm-agent-token-cost-why-a-kept-chat-grows-with-the-square-of-the-steps-and-the.md", "text": "https://wpnews.pro/news/llm-agent-token-cost-why-a-kept-chat-grows-with-the-square-of-the-steps-and-the.txt", "jsonld": "https://wpnews.pro/news/llm-agent-token-cost-why-a-kept-chat-grows-with-the-square-of-the-steps-and-the.jsonld"}}