RAG as an MCP Server: How I Let Claude Code Search My Own Documents

A developer built a small MCP (Model Context Protocol) server that exposes an existing RAG pipeline's document search to Claude Code, letting the agent retrieve matching passages from a folder of PDFs, TXT and Markdown files instead of opening whole files and burning tokens. The server runs over Streamable HTTP so multiple clients can share one instance, calls no LLM itself, and exposes just two tools, search_documents(query) and list_documents(), backed by a Chroma vector store with roughly 500-character chunks and the same search_vectorstore function used by the original chatbot.

This article is for AI developers who want Claude Code to answer questions from their own documents. Claude Code can already read files in your project, but it opens them one by one and reads them as a whole. If you have a folder with many long PDFs, for example reports or manuals, that costs a lot of tokens, and Claude has to guess which file to open first. A search over all documents that only returns the matching passages works much better. I already had this search in my RAG chatbot. I mainly built the MCP server to understand how MCP works in practice, and I was surprised how little I had to change. Below, I show how I made the search of my RAG pipeline available to Claude Code with a small MCP server, how to connect it and how I test it. MCP Model Context Protocol is an open protocol that lets AI agents use tools from a server. The server describes its tools, and the client, for example Claude Code, decides when to call them. A good example is a Gmail MCP server. When Claude is connected to it, it can search your emails or create drafts in your Gmail account for you. The big advantage over your own REST API is that you build the server once and every MCP client can use it, like Claude Code, Claude Desktop or Cursor. With a REST API, you would have to write a separate integration for every client. My server uses the transport Streamable HTTP instead of stdio. With stdio, every client starts its own copy of the server. With HTTP, the server runs separately, and several clients can connect to the same server. Before writing any code, I made four decisions. The server calls no LLM. It only returns the passages it found. The client already has a model, so the user decides which model answers the question. This also means the server needs no API keys and costs nothing per request. It has its own knowledge base. In my chatbot, users upload files for each chat. The MCP server should always be ready, so it reads a folder of documents once and stores them on disk with Chroma. Two tools are enough. One tool searches the documents, and the other one shows which files are in the knowledge base. These are the two things a client really needs, so I didn't add more. | Tool | Returns | |---|---| | search documents query | the most relevant passages with file name and page | | list documents | the files in the knowledge base | The search code is the same as in the chatbot. The agent and the MCP server both call the same function, search vectorstore . So both return the same results, and I only have to test the search once. The ingest script reads all PDF, TXT and Markdown files in a folder, splits them into chunks, small pieces of about 500 characters, and stores them in the knowledge base. php def load folder folder: str - list Document : """Load all PDF, TXT and Markdown files in the folder, tagged with their file name.""" documents = PyPDFDirectoryLoader folder, recursive=True .load documents += DirectoryLoader folder, glob= " .txt", " .md" , recursive=True, loader cls=TextLoader .load for page in documents: page.metadata "source" = Path page.metadata "source" .name return documents if name == " main ": pages = load folder sys.argv 1 shutil.rmtree KNOWLEDGE BASE DIR, ignore errors=True Chroma.from documents RecursiveCharacterTextSplitter chunk size=CHUNK SIZE, chunk overlap=CHUNK OVERLAP .split documents pages , HuggingFaceEmbeddings model name=EMBEDDING MODEL , persist directory=KNOWLEDGE BASE DIR, collection metadata={"hnsw:space": "cosine"}, print f"Ingested {sorted {p.metadata 'source' for p in pages} }" uv run python -m mcp server.ingest path/to/documents load folder loads the PDFs page by page, so every chunk keeps its page number. I only keep the file name instead of the whole path. That way, the source in the answer stays short, and the client doesn't see any paths from my computer. Every run deletes the old knowledge base with shutil.rmtree and builds it again from scratch. So files you removed from the folder also disappear from the search. This is much simpler than comparing which files changed. The embeddings come from BAAI/bge-base-en-v1.5 , which runs locally with Hugging Face, so ingesting the documents doesn't need an API key either. The whole server has less than 40 lines. Without the imports, it looks like this. mcp = FastMCP "knowledge-base", host=MCP HOST, port=MCP PORT @cache def get knowledge base - Chroma: """Open the knowledge base built by mcp server.ingest on the first tool call.""" return Chroma persist directory=KNOWLEDGE BASE DIR, embedding function=HuggingFaceEmbeddings model name=EMBEDDING MODEL , collection metadata={"hnsw:space": "cosine"}, @mcp.tool def search documents query: str - str: """Search the company knowledge base and return the most relevant passages with file name and page.""" return search vectorstore get knowledge base , query @mcp.tool def list documents - list str : """List the files that are in the knowledge base.""" sources = {m "source" for m in get knowledge base .get "metadatas" } return sorted sources if name == " main ": mcp.run transport="streamable-http" FastMCP from the official MCP Python SDK does most of the work. @mcp.tool turns a normal Python function into an MCP tool. The type hints become the input schema, and the docstring becomes the description the model sees. The docstring is more important than it looks. Claude only sees the name and the description of a tool and decides based on them when to call it. "Search the company knowledge base …" tells it clearly that this tool is meant for questions about documents. @cache on get knowledge base makes sure that the embedding model and the database are only loaded on the first tool call and reused after that. So the server starts quickly, and later searches don't load the model again. This is the search function that the chatbot and the MCP server share. php def search vectorstore vectorstore: VectorStore, query: str - str: """Return the most relevant chunks with file and page, shared by the agent and the MCP server.""" results = vectorstore.similarity search with relevance scores query, k=5 relevant = doc for doc, score in results if score = SIMILARITY THRESHOLD if not relevant: return NO DOCUMENTS FOUND output = for doc in relevant: source = doc.metadata.get "source", "Unknown" page = doc.metadata.get "page", None ref = f" {source}" + f", page {page + 1} " if page is not None else " " output.append f"{ref}\n{doc.page content}" return "Found in documents:\n\n" + "\n\n".join output It takes the 5 most similar chunks, drops every chunk with a similarity score below 0.2 and adds the file name and page to each passage. The score shows how close the meaning of the chunk is to the question, and 1.0 would mean the same meaning. PDF loaders count pages from 0, so page + 1 turns them into the page numbers a reader sees. If nothing is left, it returns NO DOCUMENTS FOUND . Start the server and add it to Claude Code with one command. uv run python -m mcp server.server serves http://127.0.0.1:8001/mcp claude mcp add --transport http knowledge-base http://127.0.0.1:8001/mcp For the example, I ingested a sample report about a solar park, the same PDF I use for the evals of my chatbot, and asked Claude Code which bank provided the loan. At first, Claude didn't call the tool and offered to search the web instead. It only sees the description "Search the company knowledge base …", so it doesn't know that this report is in there. When I added "Search my documents." to the question, it worked. ❯ Search my documents. Which bank provided the loan for the Nordheide Solar Park and at what interest rate? Called knowledge-base ⏺ Regionalbank Lüneburg provided the loan, at a fixed interest rate of 3.2% over an 18-year term covering 60% of the €14.5M total investment . Claude called search documents and got five passages with file name and page. Only the one from page 2 contains the loan, and that's the one Claude used. So the tool description decides when Claude searches your documents, and a description that says which documents are in the knowledge base would probably help. I test the server through a real MCP client session instead of calling the Python functions directly. That way, the tests also check that the tools are registered and that everything works over the protocol. php async def call tool: str, arguments: dict - str: """Call a tool through a real MCP client session and return its text output.""" async with create connected server and client session mcp, raise exceptions=True as client: result = await client.call tool tool, arguments assert not result.isError return "\n".join c.text for c in result.content if isinstance c, TextContent async def test search returns passage with file and page server with : """A matching passage should come back with its file name and page number.""" server with.add texts LOAN TEXT , metadatas= {"source": "report.pdf", "page": 2} output = await call "search documents", {"query": LOAN TEXT} assert " report.pdf, page 3 " in output assert "Sparkasse Nordheide" in output create connected server and client session from the MCP SDK connects a client to the server in memory, so I don't have to start an HTTP server for the tests. The fixture server with replaces the real knowledge base with an in-memory Chroma that uses fake embeddings DeterministicFakeEmbedding . So no model has to be downloaded, and the five tests run in a few seconds. The tests check that the client sees exactly the two tools, that a search returns the passage with file name and page, that an empty knowledge base returns NO DOCUMENTS FOUND , that every file is listed only once and that the ingest only reads PDF, TXT and Markdown files. This is a demo project, so I deliberately left out a few things a real company would need. 127.0.0.1 , so nobody else in the network can search the documents. @cache , the server has to be restarted after every ingest to see the new documents. Turning an existing RAG pipeline into an MCP server was much easier than I expected. I didn't change the search at all. I only wrapped it in two tools with FastMCP, and Claude Code could use it right away. The most important part is the description of the tools, because the model decides based on it when to search your documents. In my test, Claude only searched after I told it to. And because the server calls no LLM, it needs no API keys and works with any MCP client and any model. You can find the full code in my repo langgraph-rag-agent https://github.com/Alex-Resch/langgraph-rag-agent . AI helped me with the English, the wording and some parts of the text. I reviewed and edited everything myself.