{"slug": "real-time-voice-agents-with-local-llms-the-latency-problem-nobody-fully-solves", "title": "Real-time voice agents with local LLMs: the latency problem nobody fully solves", "summary": "Real-time voice agents using local LLMs face a hard latency ceiling of ~3 seconds imposed by Retell AI's WebSocket timeout, which forces reconnections and call termination if the first token is not produced in time. Running llama3.1:8b and qwen2.5:14b on a Hetzner RTX 4000 Ada, the team at a Spanish vocational training school finds that longer context from conversation history pushes first-token latency over the limit, and even with optimizations like reduced context size and preload race-condition fixes, the problem remains unsolved.", "body_md": "We’re running two voice agents in production for a vocational training school in Spain. Both use Retell AI for the telephony layer, Ollama for local inference on a Hetzner box with an RTX 4000 Ada (20GB VRAM), and Qdrant for RAG. The inbound agent (Aitana) runs `llama3.1:8b-instruct-q4_K_M`\n\n. The outbound sales agent (Gabriel) runs `qwen2.5:14b-instruct-q4_K_M`\n\n.\n\nWe implemented most of what the community suggested last time: fixed system prompt (~444 tokens), RAG injected into the user message instead of system, history limited to last 6 turns, both models pinned to VRAM with `keep_alive: -1`\n\n.\n\nThe conversations are better. The latency problem is not solved.\n\nRetell AI has an implicit ~3 second timeout before it reconnects the WebSocket. If the LLM doesn’t produce a first token within that window, Retell closes the connection and opens a new one. After 2 reconnections, it terminates the call entirely with `Max ping pong reconnection count reached`\n\n.\n\nThis is the constraint everything else flows from.\n\nWith `llama3.1:8b`\n\nfully loaded in VRAM, first token times from our logs:\n\nThe model is deterministic in terms of VRAM usage, but as the conversation grows the prompt gets longer. Even with history limited to 6 turns, by turn 7 the effective context (system prompt + 6 turns + RAG + memory) is approaching `num_ctx=2048`\n\n. Ollama processes the full KV cache on every call — there’s no stateful session between turns.\n\nThe math is simple and brutal: longer context = more tokens to prefill = higher first token latency. With a hard 3s ceiling, you have roughly 1500 tokens of budget before you start hitting reconnections consistently.\n\n**What we tried:** Reducing `num_ctx`\n\nto 1024 and `num_predict`\n\nto 60. It helped at the cost of truncating context and producing shorter, lower-quality responses. Not a real fix.\n\nEvery call opens two WebSocket connections to our server. Retell documents this as a “backup connection” behavior. The second connection receives a `call_details`\n\nevent, which triggers our preload logic (memory lookup, case continuity from PostgreSQL).\n\nIf that preload runs concurrently with an early `response_required`\n\non the primary connection, Ollama is handling two requests simultaneously. First token on the primary goes up by 0.5–1.5s, which is enough to cross the timeout threshold on the first user utterance.\n\nWe solved this by checking `if \"_pre_contexto\" in state`\n\nbefore running preload on reconnections. But the race condition window at call start is still real.\n\n`nomic-embed-text`\n\nis pinned in VRAM (~570MB). When a `response_required`\n\narrives, we embed the user’s question and query Qdrant before calling the LLM. The embedding call itself is fast (~20ms) but it briefly occupies the GPU compute pipeline.\n\nIn practice we see ~40ms overhead per turn from the embedding. Not catastrophic, but it’s 40ms we don’t have to spare when the budget is 3000ms total.\n\nWhat’s worse: if the model was slightly swapped or the GPU was mid-operation on something else (Redis, another process), the embedding can spike to 300–700ms. We’ve seen this exactly once in logs but it’s enough to cause a failure.\n\nWe use inline markers in the LLM output to trigger actions during the call. For ticket creation, the format is:\n\n```\n[CREAR_TICKET:tipo|resumen|prioridad]\n```\n\nThe model emits this correctly about 60% of the time. The other 40%:\n\n`[CREAR_TICKET:`\n\nand then stops before the closing bracket (the stream ends before the marker is complete)We handle the incomplete case by waiting for `]`\n\nbefore parsing. We handle duplicates with a `_ticket_ya_creado`\n\nflag. The wrong-content case we can’t reliably detect — the ticket gets created with a hallucinated summary.\n\nThe underlying issue is that `llama3.1:8b`\n\nis a small model and structured output inside a conversational streaming response is genuinely hard for it. It’s trying to produce natural speech AND a machine-readable structured payload in the same generation pass.\n\nOur current budget breakdown per turn:\n\n| Component | Tokens |\n|---|---|\n| System prompt (fixed) | ~444 |\n| RAG (3 chunks avg) | ~250 |\n| Episodic memory | ~120 |\n| 6 turns of history (avg) | ~600 |\n| User question | ~30 |\nTotal input |\n~1444 |\n| Available for response | ~604 |\n`num_predict` limit |\n80 |\n\nSo we’re already at 70% of `num_ctx=2048`\n\nbefore the model generates a single output token. By turn 8–10 the history grows, RAG chunks are longer, and we’re truncating.\n\nOllama handles overflow by silently truncating the oldest tokens in the context. For a voice agent this means the model can lose the beginning of the conversation — including the part where the user explained their actual problem.\n\nWe don’t have a clean solution. These are the options we’re considering:\n\n**Option A: Speculative decoding or a draft model** Use a smaller model (1B–3B) to draft tokens, verify with 8B. Faster time-to-first-token at the cost of implementation complexity. Ollama doesn’t support this natively.\n\n**Option B: KV cache reuse between turns** If we could persist the KV cache for the system prompt across turns (it never changes), we’d save ~444 tokens of prefill on every call. Ollama doesn’t expose this either.\n\n**Option C: Separate the action layer from the generation layer** Generate the conversational response with the LLM (fast, short), then in a second non-streaming pass extract structured data (ticket type, summary, priority). The second pass doesn’t block Retell — it runs after the response is sent. This is architecturally cleaner but adds latency to ticket creation.\n\n**Option D: Accept the 8B model can’t do this reliably and move to a hosted model** `gpt-4o-mini`\n\nwith streaming has a first token around 300ms. The latency problem goes away. The cost goes up, the privacy argument goes away, and we lose the local inference advantage.\n\nAcross 47 calls logged this week:\n\nThe 6 terminated calls all happened at turn 7 or later. No call terminated at turn 1–3. This confirms the latency issue is context-length-dependent, not a cold start problem.\n\nHappy to share more logs or code if useful.\n\n*Stack: Retell AI · Ollama · llama3.1:8b-instruct-q4_K_M · Qdrant · FastAPI WebSockets · Hetzner GEX44 · RTX 4000 Ada*", "url": "https://wpnews.pro/news/real-time-voice-agents-with-local-llms-the-latency-problem-nobody-fully-solves", "canonical_source": "https://discuss.huggingface.co/t/real-time-voice-agents-with-local-llms-the-latency-problem-nobody-fully-solves/178025#post_1", "published_at": "2026-07-20 08:44:59+00:00", "updated_at": "2026-07-20 13:40:12.425302+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-infrastructure", "ai-products", "ai-tools"], "entities": ["Retell AI", "Ollama", "Hetzner", "RTX 4000 Ada", "Qdrant", "llama3.1:8b", "qwen2.5:14b", "nomic-embed-text"], "alternates": {"html": "https://wpnews.pro/news/real-time-voice-agents-with-local-llms-the-latency-problem-nobody-fully-solves", "markdown": "https://wpnews.pro/news/real-time-voice-agents-with-local-llms-the-latency-problem-nobody-fully-solves.md", "text": "https://wpnews.pro/news/real-time-voice-agents-with-local-llms-the-latency-problem-nobody-fully-solves.txt", "jsonld": "https://wpnews.pro/news/real-time-voice-agents-with-local-llms-the-latency-problem-nobody-fully-solves.jsonld"}}