RAG Is Not One Problem: Enterprise Search vs. a Voice Agent Mid-Call

A developer building RAG systems for both enterprise search and voice agents found that the two environments impose nearly opposite constraints, making 'adding RAG' a misleadingly simple phrase. While enterprise search tolerates seconds of latency and imperfect ranking because humans can iterate, a voice agent mid-call requires retrieval to fit within a conversational rhythm where even 500ms can feel like a pause. The developer argues that the same core components—embeddings, vector search, chunking, and LLMs—must be engineered differently depending on whether a human is in the loop.

Part 1 of a 3-part series on building retrieval-augmented generation for two very different environments. Two seconds is nothing. On a search page, you can spend two seconds looking at a spinner and barely notice. On a call, two seconds of silence is enough to make you wonder if the other person is still there. And that sounds like a small product difference. It isn't. Once we started building RAG for voice agents, we realized that almost every assumption we had made around enterprise search stopped making sense. Same RAG → completely different problem. This is Part 1 of a 3-part series on what we learned building retrieval-augmented generation for two very different environments. "We're adding RAG" sounds like a single engineering task. At a high level, the idea is simple: take some knowledge → retrieve the relevant parts → give them to the model → let it answer. The interesting part is that "relevant" and "fast enough" mean very different things depending on where the model is being used. A support agent searching company documentation and a voice agent answering a customer on the phone may both use embeddings, vector search, chunking and an LLM. But the constraints around them are almost opposites. We built both, and that difference became impossible to ignore. Enterprise search RAG lives in a pretty forgiving environment. A user types a question into a search bar or chatbot → the system retrieves some documents → the results appear → the human decides what matters. There is usually enough time for the system to do a little work. If your search takes one or two seconds, that is usually fine. There is already a natural pause between asking a question and reading the result, and a spinner can make the delay feel perfectly normal. Nobody has ever rage-quit because their internal company search took 1.7 seconds. The human is already expecting some amount of waiting. Suppose the first result isn't perfect. The user can look at the second result, then the third. They can change the query, scan the surrounding text, or figure out what is actually useful themselves. The human is still doing part of the retrieval job. That makes enterprise search surprisingly tolerant of imperfect ranking. Enterprise knowledge is rarely a beautiful collection of perfectly written documents. It is usually some combination of wikis, tickets, contracts, PDFs, slide decks, old documentation and spreadsheets, all written by different people with different vocabularies over a period of years. So retrieval has to deal with a large, heterogeneous and constantly changing corpus. You may need metadata filters → access control → query rewriting → hybrid search → reranking → vector similarity. Vector search is just one part of the problem. A new document doesn't necessarily need to be searchable milliseconds after it gets uploaded. You can process it asynchronously, embed it in a batch, retry it later, or rebuild an index overnight. The user usually doesn't care what happens behind the scenes. There is no hard requirement that every change becomes searchable immediately. The system gives you the wrong result. You search again. That's basically it. There is a human sitting between the retrieval system and the final action. And that changes everything. Take the same retrieval problem and put it inside a conversation. Now the AI has to listen to someone → understand what they said → figure out what information it needs → retrieve that information → generate a response → speak it back. And it has to do all of that while the caller is sitting there waiting. Suddenly, things get much more interesting. A phone conversation has a rhythm. A short pause is normal. A long pause feels weird, and a really long pause feels like someone has put you on hold. Retrieval isn't the only thing consuming that time either. You have speech-to-text → retrieval → LLM → text-to-speech → audio delivery. So if retrieval takes 500ms, those aren't just "500ms of retrieval latency." That's 500ms taken from the total time the caller waits before hearing the next response. This is why voice systems force you to think about latency differently. A search page can afford to wait. A conversation can't. In a UI, a spinner is just a UI state. On a phone call, silence is a social interaction. Nobody sees your loading indicator or a progress bar saying "Retrieving relevant context..." The caller just hears nothing. And after a couple of seconds, the human brain starts filling in the blanks: "Did it hear me?", "Did it crash?", "Should I say something?", "Is this thing alive?" We used to think about latency as a performance problem. In voice, it becomes a UX problem. This is where the architecture starts changing. In enterprise search, it is perfectly reasonable to do work when the request arrives: embed the query → rewrite it → run retrieval → rerank → fetch metadata → process the result. In a voice agent, every extra operation in the live path costs you time, and time is exactly what you don't have. So the obvious question becomes: What can we compute before the call starts? The answer is: almost everything. Document parsing → chunking → verification → embeddings → index construction → metadata preparation. Anything expensive should ideally happen before a real human is waiting on the other end of the phone. The live path should be boring. Really boring. This sounds strange if you're coming from enterprise search. Why not give the voice agent the entire company knowledge base? Because you usually don't need to. A voice agent handling billing questions for a particular product does not need every document the company has ever created. It needs the billing policies, relevant FAQs, product information and the things it is actually allowed to talk about. That smaller corpus is a feature, not a limitation. You're trading breadth for speed and reliability. And when your latency budget is tiny, that trade is often worth a lot. This one sounds obvious, but it changes the stakes. Imagine a search system retrieves a slightly outdated policy. You see the result, decide it looks wrong, and click something else. Now imagine a voice agent retrieving that same policy and confidently saying it to a customer. The customer doesn't see the source document or a confidence score. They just heard the company's AI tell them something that may not be true. In a search interface, humans naturally filter results. In a voice call, the AI is often the interface. That means the tolerance for bad retrieval gets much lower. You can make all of these arguments on a whiteboard. We did. But two incidents made them much more real. One of our early retrieval implementations had a simple approach to extracting text from a document: take the relevant document, grab the first couple thousand characters, and send that into the retrieval pipeline. It worked surprisingly well. Until it didn't. A caller asked about something described much further down in a long document. The information existed, the search system found the document, but the relevant text never made it into the context window. We had effectively told the agent: this document exists, but only the first part of it is real. The agent obviously had no idea the answer was sitting thousands of characters away. In a search UI, this would probably be a minor annoyance. The user tries a different query and keeps going. On a call, it is much worse. The agent confidently starts answering around something it doesn't know. Not because the model hallucinated. Not because the vector database failed. Because the correct information was never a retrieval candidate in the first place. The fix was fairly boring: instead of always starting extraction at character zero, we centered the window around the actual keyword match and widened it. The implementation was simple. The lesson wasn't. Retrieval correctness isn't just about finding the right document. It's about making sure the right piece of that document actually reaches the model. A search technically running does not mean retrieval succeeded. The other incident was much less exciting. Which is exactly why it matters. We had an offline job responsible for periodically regenerating embeddings across knowledge bases. It ran for a while, then hit its execution timeout halfway through the batch. No big deal. We retry. Except the retry then ran into a second problem: a long-idle database connection had already been dropped by the server underneath it. Nothing exotic. No distributed systems paper required. Just a long-running background job doing what long-running background jobs eventually do. And honestly, this is where a lot of the real engineering in voice RAG lives. Not in the glamorous part. Not in the live request. The live request should barely have any work left to do. The hard part is building an offline pipeline that can reliably prepare everything the live system will need. It needs to survive large batches → retries → timeouts → database failures → partial progress → model changes → reprocessing. And it needs to do all of that without requiring the voice agent to suddenly become clever when a caller asks a question. There is another interesting side effect of this architecture. Because the expensive work is pushed offline, you can change that pipeline without touching the live call path. Later, we switched the embedding model to one with better multilingual coverage. That meant re-embedding the existing knowledge bases, while also dealing with a tighter input-length limit than the previous model supported. So we had a migration problem. But we didn't have a live-call problem. The offline pipeline did its job → rebuilt the data → made the new vectors available. The live system kept doing what it was supposed to do. That separation turned out to be incredibly valuable. | Dimension | Enterprise Search RAG | Voice Agent on a Live Call | |---|---|---| | Acceptable retrieval latency | Seconds | Single-digit milliseconds preferred | | Embeddings | On demand or batch | Precomputed before the call | | Corpus | Large, heterogeneous, evolving | Smaller, curated, purpose-built | | Retrieval strategy | Broad recall is useful | Precision and predictability matter more | | Slow retrieval | Slightly slower UI | Audible dead air | | Wrong retrieval | User searches again | Wrong answer spoken to a customer | | Verification | Often human-in-the-loop | Needs to happen upstream | | Indexing | Scheduled or on upload | Must be ready before the agent needs it | | Live computation | Often acceptable | Minimize as much as possible | It is tempting to take an enterprise-search RAG system and say: "Okay, now let's just make it faster." But that's not really the problem. The two systems are operating under fundamentally different constraints. Enterprise search can trade time for accuracy. Voice cannot. A search system can afford to retrieve a bunch of things and let the user figure out what matters. A voice agent needs to decide much faster what it should know and what it should say. A search system can perform work when the question arrives. A voice system needs as much work as possible to happen before the question even exists. That changes where the engineering effort goes. You spend more time thinking about: Documents → preparation → chunking → relevance → verification → embeddings → indexing → retries → freshness. The live path then becomes almost boring. And that's exactly what you want. When someone is on a phone call, you don't want your infrastructure trying to discover a new solution to vector search. You want it to already know the answer. RAG is often described as one pipeline. In practice, the environment around that pipeline determines almost everything. Enterprise search can be patient. Enterprise search can put the human back in the loop. Voice often can't. Enterprise search can perform some work when the question arrives. Voice has to do much of that work beforehand. Once you take that seriously, the architecture starts looking very different. The goal isn't to make retrieval fast enough for a voice call. The goal is to make sure retrieval has almost nothing left to do when the call starts.