Why LLM latency grows with conversation length, and the measured fix: a flat 0.1605 s to 0.1651 s step over 20,000 steps Shalendra Gaur built and tested AXI, a host-side runtime for LLM agents that holds conversation state on the host and shows the model only the exact current state rather than the full history, keeping prompt size roughly constant. In a 20,000-step run on a 7B model, step time stayed nearly flat at 0.1605 s early versus 0.1651 s late (+2.8 percent) with prompts near 620 tokens, and on Gemini 3.8 Flash across 400 turns the host-held state cut median first-token latency to 2.41 s versus 5.57 s for a kept chat once prompts passed 150,000 tokens. The architecture is by Rajnish Choubey, with code and test pages published on GitHub. LLM latency grows with conversation length because the prompt the model must read grows with it. If the host holds the state and shows the model the exact state instead of the history, the prompt stays the size of the state and the step time stays flat. Measured: a step took 0.1605 s early and 0.1651 s late +2.8 percent over 20,000 steps on a 7B model, with the prompt near 620 tokens throughout. Naming note: "AXI" here is the AXI host-side runtime for LLM agents, an architecture by Rajnish Choubey. It is not the ARM AMBA AXI chip-bus protocol. Before the first output token the model reads the prompt. A kept chat makes the prompt longer every turn, so the time to the first token rises with it, and every later token is generated against a longer context. A cache removes part of the reading cost, not all of it, and a chat that outgrows the context window simply stops. Gemini 3.8 Flash, 400 turns of ambiguous text Test 23 , chat kept against a host-held state: | | Chat kept | Host-held state | |---|---|---| | Median first token, first 50 turns | 1.86 s | about 2.4 s, flat 2.3 to 2.8 s in every tenth of the job | | Median first token, once the prompt passes 150,000 tokens | 5.57 s | 2.41 s 2.31 times lower | | Seconds a step at that prompt size | 5.70 | 2.76 2.07 times faster | | 95th percentile first token | 7.4 s | 3.8 s | On a long job with heavy turns Test 22 , the kept chat's first token went from 2.2 s in the first band to 9.1 s over all turns past 500,000 tokens; with the host holding the state it stayed at 2.3 s. Open models on vLLM 0.31.0, one NVIDIA L4, greedy decoding: Both are standard answers, and both are worth switching on. They are measured here too, so you can see what each one leaves. Caching and compaction make the history cheaper to carry. A host-held state does not carry the history at all, so the two can be used together, and neither changes whether the state is exactly right. At a short prompt. In Test 23 the kept chat was the faster arm in the first 50 turns 1,609 exact steps an hour against 1,129 . The crossover came past turn 120, at about 70,000 prompt tokens. Over all 400 turns the host-held state made 1.33 times the exact steps an hour. If your jobs never get past a few thousand tokens, latency is not the reason to move state to the host; exactness and cost per correct step are. Do not let the prompt grow with the job: hold the state on the host, check every change the model proposes, commit it all or not at all, and show the model the exact state again each step. git clone https://github.com/gaurshalendra2412/axi-runtime cd axi-runtime/python python3 quickstart replay.py That replays the saved replies of the 4,000-request run no key, no GPU . The test pages for Tests 22, 23, 42 and 43 hold the rules written before each run and the saved results. Latency on Gemini is as the API served it, including network time. Open-model times are one NVIDIA L4 on Colab, vLLM 0.31.0, greedy decoding, no retry, synthetic service graphs. Claims are about the state the host holds. Test pages: github.com/gaurshalendra2412/axi-runtime https://github.com/gaurshalendra2412/axi-runtime . White paper: doi.org/10.5281/zenodo.23958737 https://doi.org/10.5281/zenodo.23958737 . Architecture by Rajnish Choubey. Tests built and run by Shalendra Gaur.