{"slug": "parallel-agent-requests-can-slow-down-an-ollama-session", "title": "Parallel agent requests can slow down an Ollama session", "summary": "Two unmodified Qwen Code coding-agent sessions sharing a single Ollama slot waited a median of about 46 seconds for a response to start, versus about one second when each session had its own slot, Tensward found in tests against Ollama running Qwen3.8-27B. With a shared slot, 54% of the prompt tokens the server processed were repeats, falling to 0.4% when each session had a slot, because Qwen3.8-27B's recurrent layers overwrite their running state in place and cannot reuse the attention KV cache alone; the server instead saves checkpoints of roughly 150 MB each at selected token positions.", "body_md": "[← Field notes](https://tensward.com/blog)\n\n# How parallel agent requests can slow down an entire Ollama session\n\nTwo coding-agent sessions sharing one Ollama slot waited a median 46 seconds for a response to start, against one second with a slot each. Over half the prompt tokens the server processed were repeats. Why hybrid models lose their place, and which settings change it.\n\n- L4 · Ollama and llama.cpp · Qwen3.8-27B\n- Ollama\n- llama.cpp\n- prompt cache\n- coding agents\n\nAsk a coding agent to fix a failing test. It might read a file, run the test, edit a function, then run the test again. Each time a tool returns a result, the agent sends another request to the model. From your terminal it looks like one conversation. From the server’s side it’s a series of increasingly long prompts, most of which it has already seen.\n\nNormally, that repetition is cheap. The server keeps the work it did on the previous turn and picks up where the new text begins. Now open a second terminal and give the same agent a task in another project. Both sessions send the same tool descriptions and much of the same setup text, but each has its own instructions and history. What happens to the first session’s work while the server answers the second?\n\nWe tried this at Tensward with two unmodified Qwen Code sessions running against Ollama. When both sessions took turns on one *slot* (the place where the server holds an active request’s model state), the median request waited **about 46 seconds** for the response to start. When we gave each session a slot of its own, it waited **about one second**.\n\nTwo sessions sharing a slot have to wait for each other. We expected some queueing. What surprised us in the logs was how much work the server did while they waited. With a shared slot, **54% of the prompt tokens it processed were repeats**. With a slot each, that fell to 0.4%.\n\n## Why couldn’t Ollama reuse the first 8,921 tokens?\n\nImagine pausing the server just after it reads token 8,921 of session A. Up to this point, sessions A and B are identical. The model is in exactly the state B needs. But the server keeps reading A, and B arrives only after A’s prompt is finished. Can the server get back to that moment?\n\nIn a model that uses full causal attention throughout, the server keeps a pair of vectors for each token in each attention layer: a *key* and a *value*. Together these entries form the **KV cache**. Reading token 8,922 adds another entry. The entries for tokens 1 through 8,921 stay as they were, because earlier tokens cannot attend to later ones. As long as those entries are still in memory, the server can keep that prefix, discard A’s suffix, and start processing B where the prompts diverge.\n\nQwen3.8-27B is a hybrid. Its full-attention layers have that same KV cache. Its recurrent layers also carry a running state that each token updates in place. When a layer processes token 8,922, its new state replaces the one it had at 8,921. Trimming the attention layers’ KV cache gets *those layers* back to the right point. The recurrent layers still hold the result of reading the rest of A.\n\nThat’s why the server takes a **checkpoint**, also called a **snapshot** here. Both words mean a saved copy of the state needed to resume at a particular token position. If the server had saved one at 8,921, B could start with its first new token. A checkpoint at 8,000 would also help: the server could restore it and process the remaining 921 shared tokens to rebuild the state it needed. The server saves checkpoints at selected positions; each one took about 150 MB in this test.\n\nNow follow that one slot through three requests. **A goes first.** When A finishes, the slot holds A’s state. **B goes next.** A’s earliest checkpoint, at token 10,151, already includes 1,230 tokens specific to A. B has to start from token one. As the server processes B, it replaces A’s state in the slot with B’s. **Then A comes back with a tool result.** A needs exactly the state it left behind one request ago. The slot now holds B’s state, and the server never saved a copy of A’s.\n\n## Why the slowdown lasted across turns\n\nllama.cpp, the engine inside the Ollama version we tested, can save the outgoing session’s state in host RAM before handing its slot to another session. That would let A pick up where it left off when its tool result arrives. A whole conversation state was about 1 GB in our test. The engine checks how much of the old prompt the incoming request shares before deciding whether to make that copy.\n\nOur requests shared 60–66% of their tokens. That cleared the engine’s 50% threshold for reusing the slot directly, even though the matching text contained no usable checkpoint. On the switch from A to B, that decision cost A its saved place. On the switch back, it cost B its place. **As long as the sessions kept taking turns this way, each one displaced the state the other would need next.** Both agents kept their conversation histories and could finish their work. The server kept paying to rebuild those histories’ model state before it could answer the next request.\n\n## Measuring the cost with the queue still there\n\nGiving each agent a slot changes both cache reuse and queueing. To see what the cache decision alone could change, we kept both agents on **one slot** in standalone llama.cpp. That build exposes a flag, `-sps 0`, that sends each request through the RAM cache. We ran the same coding task twice per setting.\n\n| Two sessions sharing one slot | Default | With `-sps 0` | \n|---|---|---|\n| Total prompt tokens processed | about 128,800 (128,776 and 128,865) | 43,300 and 43,700 | \n| Median wait for the first byte | 48.5 s and 47.1 s | 17.4 s and 16.8 s | \n| Peak host memory | 2.4–2.9 GiB | 7.2 GiB | \n\nThe second agent still needed its first prompt processed in full. After that, each agent could resume from its own saved conversation. Saving and loading state cost about 0.8 seconds per request. From the timestamps, we estimate that roughly 14 of the remaining 17 seconds were spent waiting for the other session to finish its response.\n\nOn **Ollama 0.40.16**, we could give each session a slot, but the runner didn’t expose the similarity flag. We used `OLLAMA_NUM_PARALLEL=2` and `OLLAMA_CONTEXT_LENGTH=16384` on our 24 GB L4. That split the one-slot run’s 32,768-token budget between the sessions. The largest prompt was 14,956 tokens, so it fit. The 46-to-1-second result includes both the change in cache behavior and the ability to serve requests concurrently.\n\n## What happens when eight requests start together?\n\nGiving the two coding agents separate slots solved their switching problem. A different experiment showed how slots can also miss a useful checkpoint while processing their *own* prompts. This time we started eight short conversations together. They all used the same system prompt, so the end of that prompt was a natural place for each slot to save state for future requests.\n\nRead across a row. On the left is where that slot saved state during its first request. On the right is how many tokens it had to process again on later requests. The three slots that saved state at the end of the shared prompt averaged 125 tokens. The other five averaged 370.\n\nIn this setup, the server processed up to 512 prompt tokens per batch, split among active slots. Some slots ran out of room in the batch just before the end of the shared prompt and saved an earlier checkpoint. When the next batch crossed the desired boundary, the default spacing rule skipped the new checkpoint because the previous one was fewer than 8,192 tokens back.\n\nAllowing closer checkpoints with `--checkpoint-min-step 0` let all eight slots save at the boundary. Across four paired Ollama runs, throughput rose 10.9% and median time to first token fell 17.5%. These were short tool-calling conversations. We did not measure this setting’s cost over a long agent session.\n\n## Which setting changes which behavior?\n\nWe could see both decisions in a smaller test with two assistants that shared tool definitions but had different instructions. After a switch, the server needed to recover the returning assistant’s conversation. On subsequent requests, it also needed a checkpoint at the end of that assistant’s system prompt. On llama.cpp, changing the spacing alone reduced median prompt processing on follow-ups from 506 tokens to 126.5, while switches still took 426.5 tokens. Changing both settings brought switches down to 124.5 tokens, roughly the amount of new input.\n\nFor the coding-agent case, the first things we’d check are how many sessions share each slot and how many prompt tokens the server actually processes after a switch. Those numbers let you tell whether a user waiting on the next tool call is also paying to process the same instructions again.\n\n## Test conditions and prior work\n\nWe used Qwen Code 0.27.4, Qwen3.8-27B Q4_K_M on one NVIDIA L4, Ollama 0.40.16 and standalone llama.cpp b12441. The coding-agent comparison covered one small fix-the-tests task, with one run per Ollama configuration and two per llama.cpp configuration. First byte measures when an individual response started, including time in the queue. One coding session alone, including our test with two sub-agents, showed little repeated processing. In one llama.cpp run with the shortcut disabled, a background helper reached our eight-minute limit after the coding work and tests were done.\n\nA [llama.cpp fork](https://github.com/Piggidragon/llama.cpp/pull/34) had already changed the similarity decision to consider the newest usable checkpoint. An [Ollama discussion](https://github.com/ollama/ollama/issues/18938) describes hybrid-model clients displacing each other’s cached state. We encountered this behavior while investigating the [cost of saving model state](https://tensward.com/blog/huge-pages-llama-cpp) and measured what it meant for two real coding sessions.", "url": "https://wpnews.pro/news/parallel-agent-requests-can-slow-down-an-ollama-session", "canonical_source": "https://tensward.com/blog/ollama-agent-cache", "published_at": "2026-11-18 17:55:18+00:00", "updated_at": "2026-11-18 18:17:41.802074+00:00", "lang": "en", "topics": ["ai-agents", "ai-infrastructure", "large-language-models", "ai-tools"], "entities": ["Ollama", "llama.cpp", "Qwen3.8-27B", "Qwen Code", "Tensward"], "also_reported_by": [], "takeaway": "Giving each concurrent coding-agent session its own Ollama slot cut median time-to-first-response from about 46 seconds to about one second and repeat prompt tokens from 54% to 0.4%, because Qwen3.8-27B's recurrent layers overwrite state in place and force the server to fall back on ~150 MB checkpoi", "recommendation": {"advice": "Give each concurrent coding-agent session its own Ollama slot when running hybrid models like Qwen3.8-27B; sharing one slot raised median wait to ~46 seconds and repeat prompt tokens to 54%.", "playbook": "inference-serving", "playbook_url": "https://wpnews.pro/playbooks/inference-serving"}, "alternates": {"html": "https://wpnews.pro/news/parallel-agent-requests-can-slow-down-an-ollama-session", "markdown": "https://wpnews.pro/news/parallel-agent-requests-can-slow-down-an-ollama-session.md", "text": "https://wpnews.pro/news/parallel-agent-requests-can-slow-down-an-ollama-session.txt", "jsonld": "https://wpnews.pro/news/parallel-agent-requests-can-slow-down-an-ollama-session.jsonld"}}