cd /news/ai-agents/llm-agent-token-cost-why-a-kept-chat… · home › topics › ai-agents › article
[ARTICLE · art-171539] src=dev.to ↗ pub= topic=ai-agents verified=true sentiment=↑ positive

LLM agent token cost: why a kept chat grows with the square of the steps, and the measured fix (prompt caching counted)

Shalendra Gaur and Rajnish Choubey built AXI, a host-side runtime for LLM agents that holds conversation state on the host and shows the model only the exact state, instead of resending the full chat each step. Measured on Gemini 3.8 Flash, the approach cut prompt tokens per step by 85 times and total tokens over 400 turns by 76 times, with 4.5 times lower cost per step that ends in the correct state, even though prompt caching served 97 percent of the kept-chat tokens. On a 14B model, an exact-state step cost 21.8 joules versus 123.1 joules with the whole chat kept.

recommendation

Keep agent state on the host and send the model only the exact current state rather than the full chat history; the source measured 85x fewer prompt tokens per step and 4.5x lower cost per correct-state step.

more in playbook: Controlling LLM cost
by read3 min views3 publishedNov 18, 2026

An agent that keeps its whole chat resends everything on every step, so the tokens it sends grow with the square of the number of steps. Holding the state on the host and showing the model the exact state keeps the prompt the size of the state. Measured on Gemini 3.8 Flash: 85 times fewer prompt tokens a step and 76 times fewer in all over 400 turns, and 4.5 times lower cost per step that ends with the right state.

Naming note: "AXI" here is the AXI host-side runtime for LLM agents, an architecture by Rajnish Choubey. It is not the ARM AMBA AXI chip-bus protocol.

If step t sends a prompt of a + b·t tokens (the fixed part plus everything said so far), n steps send n·a + b·n(n+1)/2 tokens in all. The second term grows with n squared: twice the turns, about four times the tokens, at the growth rate measured in Test 23. A host that holds the state sends a prompt the size of the state, so n steps send about n times that size: a straight line.

All runs: Gemini 3.8 Flash, each job run once with the chat kept and once with the host holding the state.

Job Prompt tokens a step, chat kept With a host-held state Tokens sent in all
400 turns of plain, ambiguous sentences (Test 23) 99,809 on average 1,177 (85 times fewer) 39.9 million against 0.53 million (76 times fewer)
Long job with heavy turns (Test 22) 476,835 on average, up to 949,591 2,824 (169 times fewer) 217.5 million (457 turns) against 4.6 million (1,500 steps)
Four ARC-AGI-3 games (Test 24) n/a n/a 25.7 million against 1.95 million (13.2 times fewer)

Caching is the first thing to switch on, and it helps: the service served 97 percent of the kept chat's prompt tokens from its cache. But sent tokens and computed tokens are different bills, so here are both.

Cost per token hides the real unit. Over the first 400 requests of a 4,000-request Gemini run, at $0.75 in and $3.75 out per million tokens with thinking counted as output:

Way of using the model's replies Steps with the right state (of 400) Spent Cost of one step with the right state
Host layer: check, complete, commit 400 $0.47 $0.0012
Replies applied as written 85 $0.45 $0.0052 (4.5 times as much)
A gate that only refuses bad replies 84 $0.48 $0.0057 (4.9 times as much)

The spend was about the same. What differed was how many steps ended with the right state.

On a 14B model (Test 44), a step cost 21.8 joules with the exact state shown, 39.4 to 48.0 joules with summaries, a window or retrieval, and 123.1 joules with the whole chat kept. On one NVIDIA L4 with a 7B model (Test 38), 256 sessions ran with every step exact: 105,230 exact steps per GPU hour at 2.458 joules per exact step.

No key, no GPU. This replays the saved replies of the 4,000-request run with the repository's own code and prints the 4,000, 3,954, 85 and 4.5 times figures:

git clone https://github.com/gaurshalendra2412/axi-runtime
cd axi-runtime/python
python3 quickstart_replay.py

Gemini 3.8 Flash for the sent-token figures; open models on vLLM 0.31.0 and one NVIDIA L4, greedy decoding, no retry; synthetic service graphs; the cost rows are arithmetic added after the results, from the saved runs, not a pre-registered line. Whether the bill follows sent or computed tokens depends on how a company prices cached tokens; the repository does not price them. Joules are the card's board power read during the run. Test pages: github.com/gaurshalendra2412/axi-runtime. White paper: doi.org/10.5281/zenodo.23958737.

Architecture by Rajnish Choubey. Tests built and run by Shalendra Gaur.

── more in #ai-agents 4 stories · sorted by recency
── more on @rajnish choubey 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/llm-agent-token-cost…] indexed:0 read:3min 2026-11-18 · —