Voice agent latency degrades after turn 7-8 despite fixed system prompt + limited history — looking for mitigation ideas beyond what we've already tried

Voice agent latency degrades after turn 7-8 in production, occasionally crossing Retell's ~3s reconnect threshold and dropping calls, despite using keep_alive: -1, KV cache reuse, and two-WebSocket race condition fixes. The developer reports that Ollama re-processes the full KV cache every call, causing prefill time to climb with conversation length, and is considering aggressive summarization or splitting inbound/outbound models by turn depth as mitigations.

Following up on an earlier thread KV cache reuse, two-WebSocket race condition, RAG embedding overhead . Six weeks later, in production, the core problem hasn’t moved: first-token latency still climbs as a call gets longer, and past turn 7-8 it occasionally crosses Retell’s ~3s reconnect threshold and drops the call. What’s already in place: keep alive: -1 None of that fixes the underlying issue: Ollama re-processes the full KV cache every call, so total input tokens system + RAG + memory + history climbing with conversation length means prefill time climbing with it. No stateful session between turns as far as we can tell. Two things we haven’t tried yet, wanted to sanity-check before sinking time into either: 1. Aggressive summarization instead of raw history truncation. Right now hitting 6 turns just drops the oldest ones. Alternative: once past turn 4-5, replace older turns with a single running summary generated once, cheaply, off the hot path instead of dropping them outright — keeps the model aware of earlier context without paying full prefill cost for it. Anyone done this in a live voice pipeline and hit issues with the summary going stale or losing something the model needed a few turns later? 2. Splitting inbound fast, simple vs outbound complex, slower onto genuinely different model sizes per turn-depth , not just per-agent like we do now. E.g., use the 8B model for turns 1-6, and if a call goes long, hand off remaining turns to something smaller/faster even at some quality cost — the tradeoff being a slightly worse late-call response beats a dropped call. Feels hacky. Is this a known pattern, or does the model-swap overhead mid-call eat the savings? Also still curious, from the original thread, whether anyone has gotten real mileage out of speculative decoding or KV cache persistence with Ollama specifically — still doesn’t look natively supported, but wondering if anyone’s forced it via a custom build. Stack: Retell AI · Ollama · llama3.1:8b-instruct-q4 K M inbound · qwen2.5:14b outbound · Qdrant · FastAPI WebSockets · Hetzner GEX44 · RTX 4000 Ada.