RAG - Async Pipelines, MCP

A developer explains how asynchronous pipelines can reduce latency in retrieval-augmented generation (RAG) systems, cutting context retrieval time from 17 seconds to 10 seconds by running vector search, text search, and semantic caching concurrently. The post also introduces Model Context Protocol (MCP) as a standardized interface for exposing RAG functionality to other applications and LLMs, avoiding duplicate tool code and reducing HTTP latency.

Everything goes sequentially. P1, P2, and P3 are the three processes. Synchronous means P2 will start when P1 starts and finishes its job. P3 will start when P2 starts and finishes its job. It is not required all the time. During retrieval, in the case of hybrid search , first we will do vector search, then text search, and then any work related to semantic caching / context search. The goal here is to give more relevant context to the LLM. Suppose: In the case of a sequential process, we need to wait 17 seconds to get the context. So, is there any way we can reduce the waiting time? The answer is Async Pipelines . Here, we will start 3 threads at the same time. Their finishing times may be different. At most, we have to wait only 10 seconds to get the context. Through this async pipeline, we are reducing latency. A single thread switches between multiple tasks when needed. It is also called parallel processing . Each task will be allotted a separate thread. So, it can finish on its own time. These threads are limited to the available CPU cores. We can see the real difference in latency when we do this at the production level. If we ask the question "What happened today?" , an LLM cannot answer. If we ask the same question to ChatGPT or Gemini, it gives a response. How? Because of tool calling , which means attaching some tools or functionality to the LLM. The result of the tool will be fed into the LLM. The application developer will be responsible for writing the code for the tool/functionality. Suppose the idea of the application is to fetch weather details. Every developer may write their own code to fetch the weather details. At the end, all users/developers will get more or less the same result. This is not the right approach. So, here comes the concept of MCP . The responsible body/owner that provides the weather details will write the common tool or protocol a set of rules . The consumer/developer will use the common tool/protocol to fetch the result. Here, the developer does not need to write their own code. The common tool/protocol is called MCP . MCP will have functions and their descriptions. The LLM will decide which function to call. We can call the tool either by: MCP is within the same machine. We can also use the HTTP method to call the MCP, such as calling an API. However, we are introducing latency when using HTTP. Create an MCP for a RAG system. For example, suppose we have built a RAG system using documents that we gathered. Since you have the database and RAG system, you are the only person who can invoke the RAG system directly. Instead of keeping the RAG functionality restricted to your application, we can expose the RAG functionality through MCP . We can segregate the functionality according to different responsibilities or divisions. For example: Now, anyone who has access to the MCP can use the RAG functionality without directly accessing the underlying database or writing their own retrieval code. In this way, MCP acts as a common interface between the RAG system and different applications or LLMs . The RAG system continues to handle document retrieval, while MCP provides a standardized way for other applications or agents to access that functionality.