Following up on an earlier thread (KV cache reuse, two-WebSocket race condition, RAG embedding overhead). Six weeks later, in production, the core problem hasn’t moved: first-token latency still climbs as a call gets longer, and past turn 7-8 it occasionally crosses Retell’s ~3s reconnect threshold and drops the call.
What’s already in place:
keep_alive: -1
)None of that fixes the underlying issue: Ollama re-processes the full KV cache every call, so total input tokens (system + RAG + memory + history) climbing with conversation length means prefill time climbing with it. No stateful session between turns as far as we can tell.
Two things we haven’t tried yet, wanted to sanity-check before sinking time into either:
1. Aggressive summarization instead of raw history truncation. Right now hitting 6 turns just drops the oldest ones. Alternative: once past turn 4-5, replace older turns with a single running summary (generated once, cheaply, off the hot path) instead of dropping them outright — keeps the model aware of earlier context without paying full prefill cost for it. Anyone done this in a live voice pipeline and hit issues with the summary going stale or losing something the model needed a few turns later?
2. Splitting inbound (fast, simple) vs outbound (complex, slower) onto genuinely different model sizes per turn-depth, not just per-agent like we do now. E.g., use the 8B model for turns 1-6, and if a call goes long, hand off remaining turns to something smaller/faster even at some quality cost — the tradeoff being a slightly worse late-call response beats a dropped call. Feels hacky. Is this a known pattern, or does the model-swap overhead mid-call eat the savings?
Also still curious, from the original thread, whether anyone has gotten real mileage out of speculative decoding or KV cache persistence with Ollama specifically — still doesn’t look natively supported, but wondering if anyone’s forced it via a custom build.
Stack: Retell AI · Ollama · llama3.1:8b-instruct-q4_K_M (inbound) · qwen2.5:14b (outbound) · Qdrant · FastAPI WebSockets · Hetzner GEX44 · RTX 4000 Ada.