{"slug": "why-llm-latency-grows-with-conversation-length-and-the-measured-fix-a-flat-0-s-0", "title": "Why LLM latency grows with conversation length, and the measured fix: a flat 0.1605 s to 0.1651 s step over 20,000 steps", "summary": "Shalendra Gaur built and tested AXI, a host-side runtime for LLM agents that holds conversation state on the host and shows the model only the exact current state rather than the full history, keeping prompt size roughly constant. In a 20,000-step run on a 7B model, step time stayed nearly flat at 0.1605 s early versus 0.1651 s late (+2.8 percent) with prompts near 620 tokens, and on Gemini 3.8 Flash across 400 turns the host-held state cut median first-token latency to 2.41 s versus 5.57 s for a kept chat once prompts passed 150,000 tokens. The architecture is by Rajnish Choubey, with code and test pages published on GitHub.", "body_md": "**LLM latency grows with conversation length because the prompt the model must read grows with it. If the host holds the state and shows the model the exact state instead of the history, the prompt stays the size of the state and the step time stays flat. Measured: a step took 0.1605 s early and 0.1651 s late (+2.8 percent) over 20,000 steps on a 7B model, with the prompt near 620 tokens throughout.**\n\n*Naming note: \"AXI\" here is the AXI host-side runtime for LLM agents, an architecture by Rajnish Choubey. It is not the ARM AMBA AXI chip-bus protocol.*\n\nBefore the first output token the model reads the prompt. A kept chat makes the prompt longer every turn, so the time to the first token rises with it, and every later token is generated against a longer context. A cache removes part of the reading cost, not all of it, and a chat that outgrows the context window simply stops.\n\nGemini 3.8 Flash, 400 turns of ambiguous text (Test 23), chat kept against a host-held state:\n\n|  | Chat kept | Host-held state | \n|---|---|---|\n| Median first token, first 50 turns | 1.86 s | about 2.4 s, flat (2.3 to 2.8 s in every tenth of the job) | \n| Median first token, once the prompt passes 150,000 tokens | 5.57 s | 2.41 s (2.31 times lower) | \n| Seconds a step at that prompt size | 5.70 | 2.76 (2.07 times faster) | \n| 95th percentile first token | 7.4 s | 3.8 s | \n\nOn a long job with heavy turns (Test 22), the kept chat's first token went from 2.2 s in the first band to 9.1 s over all turns past 500,000 tokens; with the host holding the state it stayed at 2.3 s.\n\nOpen models on vLLM 0.31.0, one NVIDIA L4, greedy decoding:\n\nBoth are standard answers, and both are worth switching on. They are measured here too, so you can see what each one leaves.\n\nCaching and compaction make the history cheaper to carry. A host-held state does not carry the history at all, so the two can be used together, and neither changes whether the state is exactly right.\n\nAt a short prompt. In Test 23 the kept chat was the faster arm in the first 50 turns (1,609 exact steps an hour against 1,129). The crossover came past turn 120, at about 70,000 prompt tokens. Over all 400 turns the host-held state made 1.33 times the exact steps an hour. If your jobs never get past a few thousand tokens, latency is not the reason to move state to the host; exactness and cost per correct step are.\n\nDo not let the prompt grow with the job: hold the state on the host, check every change the model proposes, commit it all or not at all, and show the model the exact state again each step.\n\n```\ngit clone https://github.com/gaurshalendra2412/axi-runtime\ncd axi-runtime/python\npython3 quickstart_replay.py\n```\n\nThat replays the saved replies of the 4,000-request run (no key, no GPU). The test pages for Tests 22, 23, 42 and 43 hold the rules written before each run and the saved results.\n\nLatency on Gemini is as the API served it, including network time. Open-model times are one NVIDIA L4 on Colab, vLLM 0.31.0, greedy decoding, no retry, synthetic service graphs. Claims are about the state the host holds. Test pages: [github.com/gaurshalendra2412/axi-runtime](https://github.com/gaurshalendra2412/axi-runtime). White paper: [doi.org/10.5281/zenodo.23958737](https://doi.org/10.5281/zenodo.23958737).\n\n*Architecture by Rajnish Choubey. Tests built and run by Shalendra Gaur.*", "url": "https://wpnews.pro/news/why-llm-latency-grows-with-conversation-length-and-the-measured-fix-a-flat-0-s-0", "canonical_source": "https://dev.to/shalendra_gaur_ecd0567a49/why-llm-latency-grows-with-conversation-length-and-the-measured-fix-a-flat-01605-s-to-01651-s-2oi6", "published_at": "2026-11-18 05:09:35+00:00", "updated_at": "2026-11-18 05:14:19.071759+00:00", "lang": "en", "topics": ["large-language-models", "ai-agents", "ai-infrastructure", "mlops"], "entities": ["Shalendra Gaur", "Rajnish Choubey", "AXI", "Gemini 3.8 Flash", "vLLM", "NVIDIA L4", "GitHub", "Zenodo"], "also_reported_by": [], "takeaway": "Holding agent state on the host and re-showing the model only the exact state kept step time flat (0.1605 s to 0.1651 s over 20,000 steps) and cut median first-token latency from 5.57 s to 2.41 s once prompts passed 150,000 tokens.", "recommendation": {"advice": "For long-running agents, hold state on the host and show the model the exact state each step instead of the growing history; the source measured flat step time and 2.3x lower first-token latency past 150,000 tokens.", "playbook": "agent-architecture", "playbook_url": "https://wpnews.pro/playbooks/agent-architecture"}, "alternates": {"html": "https://wpnews.pro/news/why-llm-latency-grows-with-conversation-length-and-the-measured-fix-a-flat-0-s-0", "markdown": "https://wpnews.pro/news/why-llm-latency-grows-with-conversation-length-and-the-measured-fix-a-flat-0-s-0.md", "text": "https://wpnews.pro/news/why-llm-latency-grows-with-conversation-length-and-the-measured-fix-a-flat-0-s-0.txt", "jsonld": "https://wpnews.pro/news/why-llm-latency-grows-with-conversation-length-and-the-measured-fix-a-flat-0-s-0.jsonld"}}