# I Built RAG From Scratch Before Touching LangChain. Here’s What I Learned. > Source: > Published: 2026-07-31 09:12:07+00:00 When I started learning RAG, almost every tutorial looked the same. Load a document. Create embeddings. Store them in a vector database. Create a retriever. Connect everything using LangChain. A few lines of code later, I had a chatbot answering questions from my documents. It worked. But something didn’t feel right. I had built a working RAG application, but if someone had asked me what actually happened between the user’s question and the final answer, I wouldn’t have been able to explain it. I felt like I was missing something. So I stopped using LangChain for a while. Instead, I tried to understand every piece of the pipeline myself. What problem is chunking solving? Why do we need embeddings? What does a vector database actually store? What is the retriever retrieving? And where does the LLM fit into all of this? That decision completely changed how I think about RAG. In this article, I’ll walk through the same journey — starting from the problem RAG solves, building a simple RAG pipeline, and finally seeing how LangChain connects all of those pieces together. Let’s say you ask an AI assistant: **“What’s our company’s leave policy?”** or *“*How do I claim travel expenses?” The LLM has never seen your company’s internal documents or private data. So instead of answering from your company knowledge, it’ll generate an answer that **sounds** right. The tricky part is that it doesn’t sound like a guess. It sounds confident. Why does this happen? Because an LLM can only answer using what it learned during training. Your company’s documents, internal knowledge base, or private data were never part of that training. So when you ask about them, the model fills in the missing information with what it thinks is the most likely answer. That’s what we call a **hallucination**. At this point, you might be thinking: **“Why not just retrain the model with our company data?”** The problem is that company data keeps changing. New documents are added. Policies get updated. Knowledge bases keep growing. Retraining a large language model every time something changes would be slow, expensive, and impractical. So we need a different approach. Instead of expecting the model to already know everything, what if we could simply give it the right information when someone asks a question? That’s exactly the problem RAG solves. At a high level, RAG is actually pretty simple. A user asks a question. Instead of immediately sending that question to the LLM, the system first looks for information related to it. That information is then added to the prompt before the LLM generates the answer. **But how does the system actually find that information?** Let’s walk through it. First, we need the documents we want the LLM to use. These could be company policies, product documentation, PDFs, internal guides, or anything else we want the system to answer questions about. We can’t just send the entire document to the LLM every time someone asks a question. It would waste tokens, increase the cost, and could eventually run into the model’s context-window limit. So instead, we split the documents into smaller pieces called **chunks**. For example, a company handbook might contain sections about leave, travel reimbursement, and work-from-home policies. Instead of treating the whole handbook as one large piece of text, we split it into smaller chunks. Now we have smaller pieces of information. But how do we know which chunk contains the answer? Let’s say someone asks: **“How do I claim travel expenses?”** We need a way to find the chunk that talks about travel reimbursement. This is where **embeddings** come in. An embedding converts a piece of text into numbers that represent its meaning. So our chunks become something like: We then store these embeddings in a **vector database**. The important part is that the vector database lets us search based on meaning, not just exact words. Now the user asks: **“How do I claim travel expenses?”** We convert the question into an embedding as well. The system then compares the question’s embedding with the stored embeddings and finds the chunks that are most similar. A common way to measure this similarity is **cosine similarity**. In simple terms, the closer two embeddings are, the more similar their meaning is considered to be. In our example, the **Travel Policy** chunk should be one of the closest matches. Finally, we send both the user’s question and the retrieved chunks to the LLM. Something like: Now the LLM has the context it needs to answer the question accurately. That’s RAG at a high level. The important thing to notice is that **the LLM isn’t being retrained with your documents.** We’re simply finding the relevant information and giving it to the model when it’s needed. Now that we understand how RAG works, let’s build it ourselves. This time, we’re going to skip LangChain and build the pipeline from scratch. That way, we’ll understand what each piece is actually doing before using a framework that abstracts it away. The pipeline we’ll build looks like this: For this example, we’ll use: The goal isn’t to build a production-ready RAG system. I just want to understand what’s happening behind the scenes. Once this works, we’ll bring in LangChain and see what it actually simplifies. Let’s build the same company-policy example we’ve been using throughout this article. ``` python import faissimport numpy as npimport ollama# Create embeddings using Ollamadef get_embedding(text): response = ollama.embed( model="nomic-embed-text", input=text ) return response.embeddings[0]# Load the company documentwith open("company_policy.txt", "r", encoding="utf-8") as file: document = file.read()# Split the document into smaller chunksdef chunk_text(document, chunk_size=500, overlap=100): if overlap >= chunk_size: raise ValueError("overlap must be smaller than chunk_size") chunks = [] start = 0 while start < len(document): end = start + chunk_size chunks.append(document[start:end]) start += chunk_size - overlap return chunkschunks = chunk_text(document)# Create an embedding for every chunkknowledge_base = []for chunk in chunks: embedding = get_embedding(chunk) knowledge_base.append({ "text": chunk, "embedding": embedding })# Create the FAISS indexembeddings = np.array( [item["embedding"] for item in knowledge_base], dtype=np.float32)# nomic-embed-text produces 768-dimensional embeddingsindex = faiss.IndexFlatL2(768)# Add document embeddings to the indexindex.add(embeddings)# Get the user's questionquestion = input("Ask a question: ")query_embedding = get_embedding(question)query_vector = np.array( [query_embedding], dtype=np.float32)# Retrieve the most relevant chunksdistances, indices = index.search( query_vector, k=2)top_chunks = [ knowledge_base[index]["text"] for index in indices[0]]context = "\n\n".join(top_chunks)# Build the prompt using the retrieved contextprompt = f"""Answer the question using only the company information below.Company Information:{context}Question:{question}If the answer is not present in the company information,say that you don't have enough information to answer."""# Send the prompt to the LLMresponse = ollama.chat( model="llama3", messages=[ { "role": "user", "content": prompt } ])# Print the final answerprint("\nAnswer:")print(response["message"]["content"]) ``` If you compare the code with the diagram from the previous section, you’ll notice that every step maps directly to the RAG pipeline. Nothing magical happened. We simply took a document, found the pieces that were relevant to the user’s question, and gave those pieces to the LLM as context. That’s the core idea behind RAG. But while building this, I noticed something. I was writing a lot of code that wasn’t really specific to my application. Loading documents. Creating embeddings. Managing the vector database. Searching for relevant chunks. Building prompts. It works, and it’s a great way to understand how RAG works under the hood. But if you were building a real application, you’d probably want a cleaner way to connect all of these pieces together. That’s exactly where **LangChain** comes in. Now that we’ve built the pipeline ourselves, it’s much easier to see what LangChain is actually doing. It isn’t replacing RAG. It’s giving us ready-made components for the pieces we built manually. Here’s the rough mapping: Let’s build the same company-policy example again using LangChain. ``` python from langchain_community.document_loaders import TextLoaderfrom langchain_text_splitters import RecursiveCharacterTextSplitterfrom langchain_ollama import OllamaEmbeddings, ChatOllamafrom langchain_community.vectorstores import FAISSfrom langchain_core.prompts import ChatPromptTemplate# 1. Load the company policy documentloader = TextLoader( "company_policy.txt", encoding="utf-8")documents = loader.load()# 2. Split the document into chunkstext_splitter = RecursiveCharacterTextSplitter( chunk_size=500, chunk_overlap=100)chunks = text_splitter.split_documents(documents)# 3. Create embeddingsembeddings = OllamaEmbeddings( model="nomic-embed-text")# 4. Create the vector storevectorstore = FAISS.from_documents( chunks, embeddings)# 5. Create a retrieverretriever = vectorstore.as_retriever( search_kwargs={"k": 2})# 6. Get the user's questionquestion = input("Question": )# 7. Retrieve relevant chunksrelevant_chunks = retriever.invoke(question)context = "\n\n".join( document.page_content for document in relevant_chunks)# 8. Build the promptprompt = ChatPromptTemplate.from_template( """ Answer the question using only the company information below. Company Information: {context} Question: {question} If the answer is not present in the company information, say that you don't have enough information to answer. """)# 9. Create the LLMllm = ChatOllama( model="llama3")# 10. Generate the answermessages = prompt.format_messages( context=context, question=question)response = llm.invoke(messages)print(response.content) ``` If you compare it with our manual implementation, you’ll notice that the overall flow hasn’t changed. Instead of writing our own chunking function, we use: ``` text_splitter = RecursiveCharacterTextSplitter( chunk_size=500, chunk_overlap=100) ``` Instead of calling ollama.embed() ourselves, we use: ``` embeddings = OllamaEmbeddings( model="nomic-embed-text") ``` Instead of creating and managing the FAISS index manually, we simply write: ``` vectorstore = FAISS.from_documents( chunks, embeddings) ``` And instead of searching the vector store ourselves, we use a retriever: ``` retriever = vectorstore.as_retriever( search_kwargs={"k": 2}) ``` Under the hood, nothing has really changed. We’re still loading documents, creating embeddings, retrieving the most relevant chunks, and sending them to the LLM. The difference is that LangChain gives us reusable components instead of making us build every piece ourselves. Building RAG from scratch completely changed the way I look at it. Before this, I thought RAG was something you built with LangChain. Now I see it differently. **RAG isn’t tied to LangChain.** **It’s simply a workflow for retrieving relevant information and giving it to an LLM before it generates an answer.** **LangChain is just one of the tools that helps you build that workflow.** That was the biggest shift for me. Once I understood what was happening behind the scenes, reading LangChain code became much easier. I wasn’t just copying classes and methods anymore — I actually knew what each component was doing and why it was there. Will I still use LangChain? Absolutely. If I were building a real application today, I’d still use it because it saves a lot of time. But I’m really glad I built the pipeline myself first. It helped me understand what was happening under the hood instead of treating everything like a black box. For me, that was the biggest takeaway from this whole journey. [I Built RAG From Scratch Before Touching LangChain. Here’s What I Learned.](https://blog.devgenius.io/i-built-rag-from-scratch-before-touching-langchain-heres-what-i-learned-73a81288d87c) was originally published in [Dev Genius](https://blog.devgenius.io) on Medium, where people are continuing the conversation by highlighting and responding to this story.