{"slug": "voice-agent-latency-degrades-after-turn-7-8-despite-fixed-system-prompt-limited", "title": "Voice agent latency degrades after turn 7-8 despite fixed system prompt + limited history — looking for mitigation ideas beyond what we've already tried", "summary": "Voice agent latency degrades after turn 7-8 in production, occasionally crossing Retell's ~3s reconnect threshold and dropping calls, despite using keep_alive: -1, KV cache reuse, and two-WebSocket race condition fixes. The developer reports that Ollama re-processes the full KV cache every call, causing prefill time to climb with conversation length, and is considering aggressive summarization or splitting inbound/outbound models by turn depth as mitigations.", "body_md": "Following up on an earlier thread (KV cache reuse, two-WebSocket race condition, RAG embedding overhead). Six weeks later, in production, the core problem hasn’t moved: first-token latency still climbs as a call gets longer, and past turn 7-8 it occasionally crosses Retell’s ~3s reconnect threshold and drops the call.\n\nWhat’s already in place:\n\n`keep_alive: -1`\n\n)None of that fixes the underlying issue: Ollama re-processes the full KV cache every call, so total input tokens (system + RAG + memory + history) climbing with conversation length means prefill time climbing with it. No stateful session between turns as far as we can tell.\n\nTwo things we haven’t tried yet, wanted to sanity-check before sinking time into either:\n\n**1. Aggressive summarization instead of raw history truncation.** Right now hitting 6 turns just drops the oldest ones. Alternative: once past turn 4-5, replace older turns with a single running summary (generated once, cheaply, off the hot path) instead of dropping them outright — keeps the model aware of earlier context without paying full prefill cost for it. Anyone done this in a live voice pipeline and hit issues with the summary going stale or losing something the model needed a few turns later?\n\n**2. Splitting inbound (fast, simple) vs outbound (complex, slower) onto genuinely different model sizes per turn-depth**, not just per-agent like we do now. E.g., use the 8B model for turns 1-6, and if a call goes long, hand off remaining turns to something smaller/faster even at some quality cost — the tradeoff being a slightly worse late-call response beats a dropped call. Feels hacky. Is this a known pattern, or does the model-swap overhead mid-call eat the savings?\n\nAlso still curious, from the original thread, whether anyone has gotten real mileage out of speculative decoding or KV cache persistence with Ollama specifically — still doesn’t look natively supported, but wondering if anyone’s forced it via a custom build.\n\nStack: Retell AI · Ollama · llama3.1:8b-instruct-q4_K_M (inbound) · qwen2.5:14b (outbound) · Qdrant · FastAPI WebSockets · Hetzner GEX44 · RTX 4000 Ada.", "url": "https://wpnews.pro/news/voice-agent-latency-degrades-after-turn-7-8-despite-fixed-system-prompt-limited", "canonical_source": "https://discuss.huggingface.co/t/voice-agent-latency-degrades-after-turn-7-8-despite-fixed-system-prompt-limited-history-looking-for-mitigation-ideas-beyond-what-weve-already-tried/178232#post_1", "published_at": "2026-07-27 14:55:23+00:00", "updated_at": "2026-07-27 15:04:54.085501+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-infrastructure", "ai-products"], "entities": ["Retell AI", "Ollama", "llama3.1:8b-instruct-q4_K_M", "qwen2.5:14b", "Qdrant", "FastAPI", "Hetzner GEX44", "RTX 4000 Ada"], "alternates": {"html": "https://wpnews.pro/news/voice-agent-latency-degrades-after-turn-7-8-despite-fixed-system-prompt-limited", "markdown": "https://wpnews.pro/news/voice-agent-latency-degrades-after-turn-7-8-despite-fixed-system-prompt-limited.md", "text": "https://wpnews.pro/news/voice-agent-latency-degrades-after-turn-7-8-despite-fixed-system-prompt-limited.txt", "jsonld": "https://wpnews.pro/news/voice-agent-latency-degrades-after-turn-7-8-despite-fixed-system-prompt-limited.jsonld"}}