{"slug": "from-9-seconds-of-voice-ai-latency-to-1-5-seconds-building-an-in-house-voice-ai", "title": "From 9 Seconds of Voice AI Latency to 1.5 Seconds: Building an In-House Voice AI System", "summary": "A developer reduced voice AI response latency from roughly nine seconds to about 1.5 seconds by replacing a VAPI-based integration with an in-house architecture that orchestrates Twilio for telephony, Deepgram for speech-to-text, and Cartesia for expressive text-to-speech. The project began as a straightforward platform integration for automated user onboarding, but growing customization demands, particularly a more natural and emotional voice, pushed the team to build and control its own orchestration layer.", "body_md": "Nine seconds of silence. That's how long a caller waited after asking our AI assistant a simple question like \"What's the TB test process?\" long enough that most people would hang up, assuming the call had dropped.\n\nWe got that down to about 1.5 seconds. This is the story of how a straightforward platform-integration task turned into designing an in-house Voice AI architecture from scratch and the latency problem that became the real engineering challenge.\n\n**Quick summary:**\n\nThe project began as a task to integrate VAPI, a Voice AI platform for building conversational phone agents, into our existing system. The goal: automate user onboarding, so callers could get answers and guidance from an AI assistant instead of waiting for a human agent.\n\nThe integration went well. Then the client asked for something VAPI couldn't easily give us — a more natural, expressive voice — and one workaround led to another, until it became clear we'd eventually need to build the platform ourselves.\n\nThe project took an interesting turn when the client requested additional customizations.\n\nOne of the major requirements was a more natural and expressive voice capable of conveying emotions during conversations. At that time, fulfilling these requirements directly through VAPI was not as straightforward as we needed.\n\nTo address this, I implemented a temporary solution. I retrieved available voices from Cartesia, stored their voice IDs in a JSON configuration, and integrated Cartesia directly during the voice selection process.\n\nThis was only one example of the customizations the client wanted.\n\nAs the number of customization requests increased, it became clear that relying entirely on a third-party platform would eventually limit our flexibility. Like many developers experience when working with managed platforms or pre-built solutions, customization becomes increasingly difficult as requirements become more specific.\n\nThat led to a simple question:\n\n**Why don't we build our own Voice AI platform?**\n\nOnce the idea was proposed, the first step was estimating the cost and evaluating the technologies required.\n\nThe first platform that came to mind was Twilio. Whenever telephony and phone systems are discussed, Twilio is usually one of the first names that appears in the conversation.\n\nHowever, telephony was only one piece of the puzzle.\n\nVoice AI systems must handle challenges such as:\n\nWhile researching these areas, I explored Deepgram, which is widely used for Speech-to-Text (STT) processing and converts raw audio into text using deep learning models.\n\nAs I continued investigating, I realized that platforms such as Twilio, Deepgram, and Cartesia collectively provided many of the capabilities that VAPI offered.\n\nThis led to another important decision.\n\nShould we simply use one platform? Not really. If we relied entirely on a single platform, we would be back to the same integration approach we had already implemented with VAPI.\n\nShould we combine multiple platforms? That brought us back to the original question: cost.\n\nI created a cost estimation document comparing different combinations and presented the options to the client. Interestingly, the client chose the combination that could be described as the \"best of the best\":\n\nThe goal was to leverage each platform for what it does best while maintaining full control over the orchestration layer ourselves.\n\nInitially, I assumed this would be another integration project.\n\nThat assumption changed quickly.\n\nMy lead pointed out that integrating three major platforms into a unified Voice AI system was not a simple task. The project would require designing an architecture capable of coordinating multiple real-time services while maintaining a natural conversational experience.\n\nAs I explored the requirements further, one challenge stood out immediately: **latency**.\n\nVoice conversations are highly sensitive to delays. Even a few seconds of waiting can make an AI assistant feel slow, unnatural, or broken.\n\nThe second major challenge was retrieving company-specific information efficiently through a RAG (Retrieval-Augmented Generation) pipeline.\n\nI started by designing the overall architecture.\n\nThe first decision was that Twilio would never communicate directly with Deepgram or Cartesia. Instead, our in-house Voice AI platform would act as the central orchestrator for the entire conversation.\n\nFor every phone call, Twilio establishes a bidirectional media stream with our Voice AI server using WebSockets. Audio flows in both directions in real time.\n\nA simple way to think about it: Twilio acts as the phone line, while our Voice AI server becomes the intelligence layer that coordinates the entire pipeline.\n\nThe pipeline looks like this:\n\n```\nDeepgram → LLM → Cartesia\n```\n\nThe system also needed the ability to:\n\nThe call flow works roughly as follows:\n\nOnce the transcript arrives, the system determines which path the conversation should follow — casual conversation, company-related questions, or appointment-related requests.\n\nFor general conversation, the transcript and conversation history are sent directly to the LLM. For company-related queries, relevant information is first retrieved through our RAG service before being sent to the LLM. For appointment-related interactions, appointment-specific context is gathered and included before generating a response.\n\nSince the system needed access to company knowledge, implementing a RAG pipeline became necessary.\n\nThe overall process was straightforward:\n\nThis allows the AI assistant to answer questions using company-specific information instead of relying solely on its general training data.\n\nWhile the implementation concept sounds simple, making retrieval fast enough for real-time voice conversations introduced its own set of challenges. And that is where the real latency optimization journey began.\n\nVoice conversations are different from normal applications.\n\nA web application can take a few seconds to load something, and the user might tolerate it. A phone conversation is different. When a caller stops speaking, they expect the AI to respond almost immediately.\n\nThe number that matters most is **time-to-first-audio**: how long the caller waits before hearing the first part of the AI's response.\n\nOur initial pipeline was essentially sequential:\n\n```\nCaller stops talking\n      ↓\nWait for final transcript\n      ↓\nGenerate query embedding\n      ↓\nSearch vector database\n      ↓\nWait for the complete LLM response\n      ↓\nSend response to TTS\n      ↓\nStart playback\n```\n\nEach individual step seemed reasonable. Together, they made the assistant feel slow.\n\nA typical knowledge-based request looked roughly like this:\n\nIndividually, none of these numbers looked terrible. Together, they could result in roughly nine seconds of silence after a question such as \"What's the TB test process?\"\n\nAnd nine seconds of silence on a phone call feels much longer than nine seconds on a webpage. That became the real challenge.\n\nThe design decision that changed the system was simple: **we stopped optimizing for the total time required to generate the response and started optimizing for time-to-first-audio.**\n\nConsider two scenarios. In the first, the AI generates a four-second response but starts speaking after 1.5 seconds. In the second, it generates a two-second response but doesn't start speaking until four seconds later.\n\nThe first one feels significantly faster. The caller doesn't care that the AI is still generating the rest of its answer while it is already speaking. What matters is that the conversation continues naturally.\n\nThis changed the way I thought about the entire pipeline.\n\nRAG wants more context. Voice wants less waiting.\n\nIf we retrieve information, wait for the complete LLM response, and only then generate the speech, the knowledge retrieval path becomes one of the slowest parts of the conversation.\n\nAnd not every conversation needs RAG. A caller saying \"Hi, how are you?\" shouldn't trigger a vector search. Neither should \"Okay.\" or \"Thanks, that's all.\"\n\nSo the live pipeline became more intentional — the routing decision from earlier now determines whether retrieval even happens at all.\n\nThe important idea here: **RAG is a branch of the conversation, not the default path.** That was both a latency optimization and a quality improvement.\n\nThe first implementation treated every conversation turn almost like a batch process:\n\n```\nRetrieve → Generate → Speak\n```\n\nThe caller wouldn't hear anything until the final step. That doesn't work well for voice.\n\nFor knowledge-based questions, retrieval still needs to happen before the grounded answer can be generated. But once the relevant context is available, the LLM can start streaming its response.\n\nInstead of waiting for the entire answer, we buffer the generated text until we have a complete sentence. As soon as that sentence is available, it is sent to Cartesia through the live connection.\n\nSo instead of waiting for the full answer — \"The TB test is required before your onboarding process can be completed. You can complete it at…\" — the system can begin speaking \"The TB test is required before…\" while the rest of the answer is still being generated.\n\nThe pipeline therefore became something closer to:\n\n```\nTwilio audio\n      ↓\nDeepgram\n      ↓\nPartial transcripts + end-of-turn\n      ↓\nCheap router\n      ↓\n ┌───────────────┐\n │ Small talk    │\n │ Knowledge     │\n │ Tools         │\n └───────────────┘\n      ↓\nRetrieve context when required\n      ↓\nLLM streaming\n      ↓\nSentence buffer\n      ↓\nCartesia\n      ↓\nTwilio\n```\n\nThe \"cheap router\" is a fast, lightweight classifier that looks at the transcript and decides which path the conversation should take — small talk, knowledge question, or tool call — before anything expensive like retrieval or the LLM gets involved. It has to be fast and low-cost since it sits in the critical path before every single response.\n\nThe three major components — Deepgram, the LLM, and Cartesia — were no longer simply running one after another. They started overlapping. That distinction made a huge difference.\n\nThis might sound like a small implementation detail. It wasn't.\n\nThe first version was paying connection setup costs repeatedly. The LLM connection could become idle and require another connection setup. Text-to-speech was also opening a new request for individual responses.\n\nThose delays might not be obvious in a local development environment. They become much more obvious when you're having an actual phone conversation.\n\nWe changed the connection strategy. Connections to the LLM stay warm across turns, allowing subsequent requests to reuse an existing session rather than repeatedly paying connection setup costs. For Cartesia, we use a persistent WebSocket for the lifetime of the call rather than creating a new HTTPS request for every sentence. The server also maintains a small pool of warm sockets so that the initial greeting doesn't have to pay the full connection setup cost while the caller is already waiting.\n\nThis is not the most exciting part of building a Voice AI system. But these small delays add up. Sometimes the difference between a system that feels instant and one that feels slightly sluggish is hidden in hundreds of milliseconds that nobody notices individually.\n\nA live onboarding call contains much more than knowledge questions. A caller might say \"Hi, how are you?\" or \"Okay.\" or \"Can I book Thursday at 2?\" None of these should automatically trigger a vector search.\n\nOnly factual, company-specific questions should enter the retrieval path. This improves latency, but it also improves response quality. Running RAG for \"Thanks.\" could retrieve unrelated company information and provide unnecessary context to the LLM. Likewise, an appointment request shouldn't make the model search through company documents when what it actually needs is live appointment information.\n\nAgain, the important architectural principle is: **retrieval is conditional.**\n\nAnother interesting optimization came from looking at when we generated query embeddings.\n\nInitially, embedding started only after speech-to-text confirmed that the caller had finished speaking. That meant we were already paying the end-of-turn latency before even beginning the embedding step.\n\nBut during a real conversation, we receive partial transcripts while the caller is still speaking. Those partial transcripts can be useful. Once enough meaningful words are available, we can speculatively generate the embedding from the current hypothesis. By the time the end-of-turn signal arrives, the embedding may already be available.\n\nWe also introduced caching for embeddings. Filler words such as \"um\" and \"uh\" don't meaningfully change the query, so they can be removed before generating the cache key. That means questions such as \"um what is onboarding\" and \"what is onboarding\" can share the same cached embedding.\n\nThe vector search itself was never the biggest bottleneck. The embedding step was. For live retrieval, we eventually moved query embedding on-box using a small local model, bringing that part of the process down to roughly 10–30 milliseconds, instead of introducing another cloud round trip into the critical path.\n\nVoice answers should generally be concise.\n\nSending huge document chunks into the LLM doesn't just increase the amount of information the model has to process — it can also make the assistant sound unnatural. Imagine calling a company and asking a simple question, only for the AI to start reading an entire policy document back to you. That's not a good voice experience.\n\nSo we keep the retrieved context intentionally small. We used a small top-k, limit the size of retrieved chunks, and use a faster model for spoken knowledge responses where appropriate. We also used a similarity threshold — if the retrieved information isn't sufficiently relevant, we don't force it into the prompt. Sometimes \"I don't have that detail. I can connect you with someone on the team.\" is much better than a slow or potentially incorrect answer.\n\nThis is one of the biggest differences between RAG in a chatbot and RAG in a voice system. In a text interface, additional context is relatively cheap. In a phone conversation, additional context can directly affect how long the caller waits before hearing the first word.\n\nThere is another problem with RAG in Voice AI that isn't immediately obvious: the query isn't typed by a user, it's produced from speech. That means the retrieval system has to deal with transcription errors and conversational context.\n\nImagine the caller says \"How long does that take?\" That sentence by itself isn't very useful as a search query. But perhaps the previous conversation was about a TB test — the real question is \"How long does the TB test take?\"\n\nSpeech recognition can introduce another problem too. A caller might say \"TB test\" and the transcription might produce \"TV test.\" Sending that raw transcript directly into vector search can reduce retrieval quality.\n\nInstead of adding another expensive LLM call just to rewrite the query, we introduced a lightweight rewrite layer. It can:\n\nFor example, \"How long does it take?\" can become something closer to \"How long does the TB test take?\" The important part is that this happens without adding another expensive model call to the critical path.\n\nAnother architectural decision was separating knowledge ingestion from live retrieval. These are two completely different workloads. Document ingestion can take time. A phone call cannot.\n\nWhen a document is uploaded, we can:\n\nThat work happens during ingestion rather than while someone is waiting on a phone call. The live retrieval path should be much simpler:\n\n```\nUser query\n   ↓\nQuery embedding\n   ↓\nVector search\n   ↓\nSmall context block\n   ↓\nLLM\n   ↓\nStreaming response\n```\n\nThe phone call should never have to wait for PDF parsing, document chunking, or other ingestion work. This separation is what makes RAG practical for a real-time voice system.\n\nAssistant scoping is important here as well. We don't search the entire company's knowledge base for every question — we search the documents associated with the specific assistant handling the call. Apart from being a correctness issue, searching unnecessary data also means doing unnecessary work.\n\nOnce the basic pipeline was working, several problems became apparent.\n\n**Turn detection.** If the AI waits too long after a caller pauses, the conversation feels slow. If it responds too quickly, it can interrupt the caller. We use conversational speech-to-text with an end-of-turn signal, along with an eager end-of-turn mechanism that allows the system to begin responding slightly earlier — saving hundreds of milliseconds. But it also introduces another problem: the same utterance can sometimes arrive more than once, with the second transcript containing additional words. The orchestrator therefore needs to understand whether the new event is the same utterance, a continuation, or a completely new question. Otherwise, the assistant can end up answering the same question twice.\n\n**Barge-in.** Real people interrupt. When the caller starts speaking while the AI is talking, the system needs to stop the current response. We abort the current LLM turn, cancel in-flight speech, and instruct Twilio to discard leftover audio. But we don't tear down the Cartesia connection — reconnecting the TTS connection during a call would introduce another latency problem. After an interruption, a short backoff also prevents the pipeline from constantly starting and stopping if the caller and assistant talk over each other.\n\n**Backchannels.** Then there are the small things people naturally say while listening — \"Okay.\", \"Mm-hmm.\", \"Got it.\" These aren't necessarily interruptions. But \"No.\" or \"Stop.\" might be. The system therefore needs to distinguish between conversational backchannels and actual interruptions.\n\nThese details aren't particularly impressive in an architecture diagram. But they're the difference between a Voice AI demo and something that people can actually use on a phone call.\n\nNot every interaction is a knowledge question. Appointment booking, lookups, rescheduling, cancellations, and similar actions are handled through tools connected to our existing backend. Transfers also follow routing rules — an emergency can require a human agent, while another conversation may need to be transferred to a different AI assistant.\n\nBut tool calls can take time. If the caller hears complete silence while the backend checks appointment availability, the system feels like it has stopped working. So the assistant can provide a short conversational filler while the tool is executing and then continue with the actual result. The filler isn't replacing the result — it's simply covering the time required for the backend operation.\n\nTransfer intent is also handled with latency in mind. Simple phrase matching can be checked first because it is inexpensive. Semantic similarity can then be used when the wording is less obvious. Some of the data required for these decisions can also be prepared earlier in the call so that the first meaningful turn doesn't have to pay the entire cost.\n\nVoice AI isn't only about getting an answer from an LLM. The system also needs guardrails. Some instructions are handled through the system prompt. Other patterns can be detected locally to block or modify unsafe inputs and outputs.\n\nThe important part is that these checks need to fit into the same real-time pipeline. A guardrail that takes long enough to create an obvious pause becomes another latency problem.\n\nThis is an important distinction. When I say we reduced latency from roughly 9 seconds to around 1.5 seconds, I'm referring to time-to-first-audio on a typical knowledge turn. It assumes a warm path where:\n\nIt does not mean that every possible Voice AI interaction completes in 1.5 seconds. For example, a cold appointment-booking request may require checking availability, confirming information, calling backend services, and generating multiple time slots — that's a different latency measurement.\n\nIt also isn't the total duration required to speak the entire answer. Those are separate metrics.\n\nThe important change was the experience the caller actually perceived. Instead of waiting in silence after asking a question, the assistant could begin responding while the remaining work continued in the background. The conversation started feeling like a conversation again.\n\nOf course, none of these optimizations came for free.\n\nStreaming the first sentence means that the system can sometimes begin speaking before the complete answer is known. We mitigate this with shorter responses and retrieval confidence thresholds rather than waiting for the entire response.\n\nLocal embeddings are significantly faster, but they need to remain aligned with the embeddings generated during document ingestion.\n\nHeuristic query rewriting works extremely well for the specific domain we are dealing with, but it requires maintenance and won't necessarily generalize perfectly to every industry.\n\nAnd perhaps the biggest tradeoff is the one we started this entire journey with: once you stop relying entirely on a managed Voice AI platform, you own the entire orchestration layer. That means dealing with:\n\nThat's the real cost of building your own Voice AI system.\n\nThe biggest lesson from this project wasn't really about Twilio, Deepgram, Cartesia, RAG, or even the LLM. Those components are replaceable.\n\nThe real product is the orchestration loop that decides:\n\nAnd most importantly: **how quickly can I make the caller hear something useful?**\n\nThat is what ultimately determines whether a Voice AI system feels like a real conversation — or just an API pipeline talking over a phone call.\n\nIf you're building or evaluating Voice AI — whether that's sticking with a managed platform or owning the orchestration layer yourself — I'd genuinely like to compare notes. Feel free to reach out or drop a comment with what you're running into.\n\n*Mehar Aziz is a Software Engineer working on full-stack development including AI/ML. Find me on [\\[LinkedIn\\]](https://www.linkedin.com/in/mehar-aziz).*", "url": "https://wpnews.pro/news/from-9-seconds-of-voice-ai-latency-to-1-5-seconds-building-an-in-house-voice-ai", "canonical_source": "https://dev.to/mehar_aziz/from-9-seconds-of-voice-ai-latency-to-15-seconds-building-an-in-house-voice-ai-system-bal", "published_at": "2026-09-11 22:22:26+00:00", "updated_at": "2026-09-11 22:50:55.621514+00:00", "lang": "en", "topics": ["ai-agents", "natural-language-processing", "ai-infrastructure", "ai-tools", "developer-tools"], "entities": ["VAPI", "Twilio", "Deepgram", "Cartesia"], "alternates": {"html": "https://wpnews.pro/news/from-9-seconds-of-voice-ai-latency-to-1-5-seconds-building-an-in-house-voice-ai", "markdown": "https://wpnews.pro/news/from-9-seconds-of-voice-ai-latency-to-1-5-seconds-building-an-in-house-voice-ai.md", "text": "https://wpnews.pro/news/from-9-seconds-of-voice-ai-latency-to-1-5-seconds-building-an-in-house-voice-ai.txt", "jsonld": "https://wpnews.pro/news/from-9-seconds-of-voice-ai-latency-to-1-5-seconds-building-an-in-house-voice-ai.jsonld"}}