We’re running two voice agents in production for a vocational training school in Spain. Both use Retell AI for the telephony layer, Ollama for local inference on a Hetzner box with an RTX 4000 Ada (20GB VRAM), and Qdrant for RAG. The inbound agent (Aitana) runs llama3.1:8b-instruct-q4_K_M
. The outbound sales agent (Gabriel) runs qwen2.5:14b-instruct-q4_K_M
.
We implemented most of what the community suggested last time: fixed system prompt (~444 tokens), RAG injected into the user message instead of system, history limited to last 6 turns, both models pinned to VRAM with keep_alive: -1
.
The conversations are better. The latency problem is not solved.
Retell AI has an implicit ~3 second timeout before it reconnects the WebSocket. If the LLM doesn’t produce a first token within that window, Retell closes the connection and opens a new one. After 2 reconnections, it terminates the call entirely with Max ping pong reconnection count reached
.
This is the constraint everything else flows from.
With llama3.1:8b
fully loaded in VRAM, first token times from our logs:
The model is deterministic in terms of VRAM usage, but as the conversation grows the prompt gets longer. Even with history limited to 6 turns, by turn 7 the effective context (system prompt + 6 turns + RAG + memory) is approaching num_ctx=2048
. Ollama processes the full KV cache on every call — there’s no stateful session between turns.
The math is simple and brutal: longer context = more tokens to prefill = higher first token latency. With a hard 3s ceiling, you have roughly 1500 tokens of budget before you start hitting reconnections consistently.
What we tried: Reducing num_ctx
to 1024 and num_predict
to 60. It helped at the cost of truncating context and producing shorter, lower-quality responses. Not a real fix.
Every call opens two WebSocket connections to our server. Retell documents this as a “backup connection” behavior. The second connection receives a call_details
event, which triggers our preload logic (memory lookup, case continuity from PostgreSQL).
If that preload runs concurrently with an early response_required
on the primary connection, Ollama is handling two requests simultaneously. First token on the primary goes up by 0.5–1.5s, which is enough to cross the timeout threshold on the first user utterance.
We solved this by checking if "_pre_contexto" in state
before running preload on reconnections. But the race condition window at call start is still real.
nomic-embed-text
is pinned in VRAM (~570MB). When a response_required
arrives, we embed the user’s question and query Qdrant before calling the LLM. The embedding call itself is fast (~20ms) but it briefly occupies the GPU compute pipeline.
In practice we see ~40ms overhead per turn from the embedding. Not catastrophic, but it’s 40ms we don’t have to spare when the budget is 3000ms total.
What’s worse: if the model was slightly swapped or the GPU was mid-operation on something else (Redis, another process), the embedding can spike to 300–700ms. We’ve seen this exactly once in logs but it’s enough to cause a failure.
We use inline markers in the LLM output to trigger actions during the call. For ticket creation, the format is:
[CREAR_TICKET:tipo|resumen|prioridad]
The model emits this correctly about 60% of the time. The other 40%:
[CREAR_TICKET:
and then stops before the closing bracket (the stream ends before the marker is complete)We handle the incomplete case by waiting for ]
before parsing. We handle duplicates with a _ticket_ya_creado
flag. The wrong-content case we can’t reliably detect — the ticket gets created with a hallucinated summary.
The underlying issue is that llama3.1:8b
is a small model and structured output inside a conversational streaming response is genuinely hard for it. It’s trying to produce natural speech AND a machine-readable structured payload in the same generation pass.
Our current budget breakdown per turn:
| Component | Tokens |
|---|---|
| System prompt (fixed) | ~444 |
| RAG (3 chunks avg) | ~250 |
| Episodic memory | ~120 |
| 6 turns of history (avg) | ~600 |
| User question | ~30 |
| Total input | |
| ~1444 | |
| Available for response | ~604 |
num_predict limit |
|
| 80 |
So we’re already at 70% of num_ctx=2048
before the model generates a single output token. By turn 8–10 the history grows, RAG chunks are longer, and we’re truncating.
Ollama handles overflow by silently truncating the oldest tokens in the context. For a voice agent this means the model can lose the beginning of the conversation — including the part where the user explained their actual problem.
We don’t have a clean solution. These are the options we’re considering:
Option A: Speculative decoding or a draft model Use a smaller model (1B–3B) to draft tokens, verify with 8B. Faster time-to-first-token at the cost of implementation complexity. Ollama doesn’t support this natively.
Option B: KV cache reuse between turns If we could persist the KV cache for the system prompt across turns (it never changes), we’d save ~444 tokens of prefill on every call. Ollama doesn’t expose this either.
Option C: Separate the action layer from the generation layer Generate the conversational response with the LLM (fast, short), then in a second non-streaming pass extract structured data (ticket type, summary, priority). The second pass doesn’t block Retell — it runs after the response is sent. This is architecturally cleaner but adds latency to ticket creation.
Option D: Accept the 8B model can’t do this reliably and move to a hosted model gpt-4o-mini
with streaming has a first token around 300ms. The latency problem goes away. The cost goes up, the privacy argument goes away, and we lose the local inference advantage.
Across 47 calls logged this week:
The 6 terminated calls all happened at turn 7 or later. No call terminated at turn 1–3. This confirms the latency issue is context-length-dependent, not a cold start problem.
Happy to share more logs or code if useful.
Stack: Retell AI · Ollama · llama3.1:8b-instruct-q4_K_M · Qdrant · FastAPI WebSockets · Hetzner GEX44 · RTX 4000 Ada