{"slug": "rag-is-not-one-problem-enterprise-search-vs-a-voice-agent-mid-call", "title": "RAG Is Not One Problem: Enterprise Search vs. a Voice Agent Mid-Call", "summary": "A developer building RAG systems for both enterprise search and voice agents found that the two environments impose nearly opposite constraints, making 'adding RAG' a misleadingly simple phrase. While enterprise search tolerates seconds of latency and imperfect ranking because humans can iterate, a voice agent mid-call requires retrieval to fit within a conversational rhythm where even 500ms can feel like a pause. The developer argues that the same core components—embeddings, vector search, chunking, and LLMs—must be engineered differently depending on whether a human is in the loop.", "body_md": "*Part 1 of a 3-part series on building retrieval-augmented generation for two very different environments.*\n\nTwo seconds is nothing.\n\nOn a search page, you can spend two seconds looking at a spinner and barely notice. On a call, two seconds of silence is enough to make you wonder if the other person is still there.\n\nAnd that sounds like a small product difference.\n\nIt isn't.\n\nOnce we started building RAG for voice agents, we realized that almost every assumption we had made around enterprise search stopped making sense.\n\nSame RAG → completely different problem.\n\nThis is Part 1 of a 3-part series on what we learned building retrieval-augmented generation for two very different environments.\n\n\"We're adding RAG\" sounds like a single engineering task.\n\nAt a high level, the idea is simple: take some knowledge → retrieve the relevant parts → give them to the model → let it answer.\n\nThe interesting part is that \"relevant\" and \"fast enough\" mean very different things depending on where the model is being used.\n\nA support agent searching company documentation and a voice agent answering a customer on the phone may both use embeddings, vector search, chunking and an LLM.\n\nBut the constraints around them are almost opposites.\n\nWe built both, and that difference became impossible to ignore.\n\nEnterprise search RAG lives in a pretty forgiving environment.\n\nA user types a question into a search bar or chatbot → the system retrieves some documents → the results appear → the human decides what matters.\n\nThere is usually enough time for the system to do a little work.\n\nIf your search takes one or two seconds, that is usually fine. There is already a natural pause between asking a question and reading the result, and a spinner can make the delay feel perfectly normal.\n\nNobody has ever rage-quit because their internal company search took 1.7 seconds.\n\nThe human is already expecting some amount of waiting.\n\nSuppose the first result isn't perfect.\n\nThe user can look at the second result, then the third. They can change the query, scan the surrounding text, or figure out what is actually useful themselves.\n\nThe human is still doing part of the retrieval job.\n\nThat makes enterprise search surprisingly tolerant of imperfect ranking.\n\nEnterprise knowledge is rarely a beautiful collection of perfectly written documents.\n\nIt is usually some combination of wikis, tickets, contracts, PDFs, slide decks, old documentation and spreadsheets, all written by different people with different vocabularies over a period of years.\n\nSo retrieval has to deal with a large, heterogeneous and constantly changing corpus.\n\nYou may need metadata filters → access control → query rewriting → hybrid search → reranking → vector similarity.\n\nVector search is just one part of the problem.\n\nA new document doesn't necessarily need to be searchable milliseconds after it gets uploaded.\n\nYou can process it asynchronously, embed it in a batch, retry it later, or rebuild an index overnight.\n\nThe user usually doesn't care what happens behind the scenes.\n\nThere is no hard requirement that every change becomes searchable immediately.\n\nThe system gives you the wrong result.\n\nYou search again.\n\nThat's basically it.\n\nThere is a human sitting between the retrieval system and the final action.\n\nAnd that changes everything.\n\nTake the same retrieval problem and put it inside a conversation.\n\nNow the AI has to listen to someone → understand what they said → figure out what information it needs → retrieve that information → generate a response → speak it back.\n\nAnd it has to do all of that while the caller is sitting there waiting.\n\nSuddenly, things get much more interesting.\n\nA phone conversation has a rhythm.\n\nA short pause is normal. A long pause feels weird, and a really long pause feels like someone has put you on hold.\n\nRetrieval isn't the only thing consuming that time either.\n\nYou have speech-to-text → retrieval → LLM → text-to-speech → audio delivery.\n\nSo if retrieval takes 500ms, those aren't just \"500ms of retrieval latency.\" That's 500ms taken from the total time the caller waits before hearing the next response.\n\nThis is why voice systems force you to think about latency differently.\n\nA search page can afford to wait.\n\nA conversation can't.\n\nIn a UI, a spinner is just a UI state.\n\nOn a phone call, silence is a social interaction.\n\nNobody sees your loading indicator or a progress bar saying \"Retrieving relevant context...\"\n\nThe caller just hears nothing.\n\nAnd after a couple of seconds, the human brain starts filling in the blanks: \"Did it hear me?\", \"Did it crash?\", \"Should I say something?\", \"Is this thing alive?\"\n\nWe used to think about latency as a performance problem.\n\nIn voice, it becomes a UX problem.\n\nThis is where the architecture starts changing.\n\nIn enterprise search, it is perfectly reasonable to do work when the request arrives: embed the query → rewrite it → run retrieval → rerank → fetch metadata → process the result.\n\nIn a voice agent, every extra operation in the live path costs you time, and time is exactly what you don't have.\n\nSo the obvious question becomes:\n\n**What can we compute before the call starts?**\n\nThe answer is: almost everything.\n\nDocument parsing → chunking → verification → embeddings → index construction → metadata preparation.\n\nAnything expensive should ideally happen before a real human is waiting on the other end of the phone.\n\n**The live path should be boring. Really boring.**\n\nThis sounds strange if you're coming from enterprise search.\n\nWhy not give the voice agent the entire company knowledge base?\n\nBecause you usually don't need to.\n\nA voice agent handling billing questions for a particular product does not need every document the company has ever created. It needs the billing policies, relevant FAQs, product information and the things it is actually allowed to talk about.\n\nThat smaller corpus is a feature, not a limitation.\n\nYou're trading breadth for speed and reliability.\n\nAnd when your latency budget is tiny, that trade is often worth a lot.\n\nThis one sounds obvious, but it changes the stakes.\n\nImagine a search system retrieves a slightly outdated policy. You see the result, decide it looks wrong, and click something else.\n\nNow imagine a voice agent retrieving that same policy and confidently saying it to a customer.\n\nThe customer doesn't see the source document or a confidence score.\n\nThey just heard the company's AI tell them something that may not be true.\n\nIn a search interface, humans naturally filter results.\n\nIn a voice call, the AI is often the interface.\n\nThat means the tolerance for bad retrieval gets much lower.\n\nYou can make all of these arguments on a whiteboard.\n\nWe did.\n\nBut two incidents made them much more real.\n\nOne of our early retrieval implementations had a simple approach to extracting text from a document: take the relevant document, grab the first couple thousand characters, and send that into the retrieval pipeline.\n\nIt worked surprisingly well.\n\nUntil it didn't.\n\nA caller asked about something described much further down in a long document. The information existed, the search system found the document, but the relevant text never made it into the context window.\n\nWe had effectively told the agent: **this document exists, but only the first part of it is real.**\n\nThe agent obviously had no idea the answer was sitting thousands of characters away.\n\nIn a search UI, this would probably be a minor annoyance.\n\nThe user tries a different query and keeps going.\n\nOn a call, it is much worse.\n\nThe agent confidently starts answering around something it doesn't know.\n\nNot because the model hallucinated.\n\nNot because the vector database failed.\n\nBecause the correct information was never a retrieval candidate in the first place.\n\nThe fix was fairly boring: instead of always starting extraction at character zero, we centered the window around the actual keyword match and widened it.\n\nThe implementation was simple.\n\nThe lesson wasn't.\n\n**Retrieval correctness isn't just about finding the right document. It's about making sure the right piece of that document actually reaches the model.**\n\nA search technically running does not mean retrieval succeeded.\n\nThe other incident was much less exciting.\n\nWhich is exactly why it matters.\n\nWe had an offline job responsible for periodically regenerating embeddings across knowledge bases. It ran for a while, then hit its execution timeout halfway through the batch.\n\nNo big deal.\n\nWe retry.\n\nExcept the retry then ran into a second problem: a long-idle database connection had already been dropped by the server underneath it.\n\nNothing exotic.\n\nNo distributed systems paper required.\n\nJust a long-running background job doing what long-running background jobs eventually do.\n\nAnd honestly, this is where a lot of the real engineering in voice RAG lives.\n\nNot in the glamorous part.\n\nNot in the live request.\n\nThe live request should barely have any work left to do.\n\nThe hard part is building an offline pipeline that can reliably prepare everything the live system will need.\n\nIt needs to survive large batches → retries → timeouts → database failures → partial progress → model changes → reprocessing.\n\nAnd it needs to do all of that without requiring the voice agent to suddenly become clever when a caller asks a question.\n\nThere is another interesting side effect of this architecture.\n\nBecause the expensive work is pushed offline, you can change that pipeline without touching the live call path.\n\nLater, we switched the embedding model to one with better multilingual coverage. That meant re-embedding the existing knowledge bases, while also dealing with a tighter input-length limit than the previous model supported.\n\nSo we had a migration problem.\n\nBut we didn't have a live-call problem.\n\nThe offline pipeline did its job → rebuilt the data → made the new vectors available.\n\nThe live system kept doing what it was supposed to do.\n\nThat separation turned out to be incredibly valuable.\n\n| Dimension | Enterprise Search RAG | Voice Agent on a Live Call | \n|---|---|---|\n| Acceptable retrieval latency | Seconds | Single-digit milliseconds preferred | \n| Embeddings | On demand or batch | Precomputed before the call | \n| Corpus | Large, heterogeneous, evolving | Smaller, curated, purpose-built | \n| Retrieval strategy | Broad recall is useful | Precision and predictability matter more | \n| Slow retrieval | Slightly slower UI | Audible dead air | \n| Wrong retrieval | User searches again | Wrong answer spoken to a customer | \n| Verification | Often human-in-the-loop | Needs to happen upstream | \n| Indexing | Scheduled or on upload | Must be ready before the agent needs it | \n| Live computation | Often acceptable | Minimize as much as possible | \n\nIt is tempting to take an enterprise-search RAG system and say:\n\n**\"Okay, now let's just make it faster.\"**\n\nBut that's not really the problem.\n\nThe two systems are operating under fundamentally different constraints.\n\nEnterprise search can trade time for accuracy.\n\nVoice cannot.\n\nA search system can afford to retrieve a bunch of things and let the user figure out what matters.\n\nA voice agent needs to decide much faster what it should know and what it should say.\n\nA search system can perform work when the question arrives.\n\nA voice system needs as much work as possible to happen before the question even exists.\n\nThat changes where the engineering effort goes.\n\nYou spend more time thinking about:\n\nDocuments → preparation → chunking → relevance → verification → embeddings → indexing → retries → freshness.\n\nThe live path then becomes almost boring.\n\nAnd that's exactly what you want.\n\nWhen someone is on a phone call, you don't want your infrastructure trying to discover a new solution to vector search.\n\nYou want it to already know the answer.\n\nRAG is often described as one pipeline.\n\nIn practice, the environment around that pipeline determines almost everything.\n\nEnterprise search can be patient.\n\nEnterprise search can put the human back in the loop.\n\nVoice often can't.\n\nEnterprise search can perform some work when the question arrives.\n\nVoice has to do much of that work beforehand.\n\nOnce you take that seriously, the architecture starts looking very different.\n\n**The goal isn't to make retrieval fast enough for a voice call.**\n\n**The goal is to make sure retrieval has almost nothing left to do when the call starts.**", "url": "https://wpnews.pro/news/rag-is-not-one-problem-enterprise-search-vs-a-voice-agent-mid-call", "canonical_source": "https://dev.to/mohammad_palla/rag-is-not-one-problem-enterprise-search-vs-a-voice-agent-mid-call-68i", "published_at": "2026-09-07 17:23:56+00:00", "updated_at": "2026-09-07 17:32:29.527995+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-products", "ai-infrastructure", "developer-tools"], "entities": [], "alternates": {"html": "https://wpnews.pro/news/rag-is-not-one-problem-enterprise-search-vs-a-voice-agent-mid-call", "markdown": "https://wpnews.pro/news/rag-is-not-one-problem-enterprise-search-vs-a-voice-agent-mid-call.md", "text": "https://wpnews.pro/news/rag-is-not-one-problem-enterprise-search-vs-a-voice-agent-mid-call.txt", "jsonld": "https://wpnews.pro/news/rag-is-not-one-problem-enterprise-search-vs-a-voice-agent-mid-call.jsonld"}}