{"slug": "i-built-rag-from-scratch-before-touching-langchain-heres-what-i-learned", "title": "I Built RAG From Scratch Before Touching LangChain. Here’s What I Learned.", "summary": "A developer recounts building a Retrieval-Augmented Generation (RAG) pipeline from scratch before using LangChain, learning the underlying mechanics of chunking, embeddings, vector databases, and retrieval. The article explains that RAG solves LLM hallucinations by retrieving relevant document chunks and adding them to the prompt, avoiding costly retraining. The author emphasizes that understanding each component is crucial for effective RAG implementation.", "body_md": "When I started learning RAG, almost every tutorial looked the same.\n\nLoad a document.\n\nCreate embeddings.\n\nStore them in a vector database.\n\nCreate a retriever.\n\nConnect everything using LangChain.\n\nA few lines of code later, I had a chatbot answering questions from my documents.\n\nIt worked.\n\nBut something didn’t feel right.\n\nI had built a working RAG application, but if someone had asked me what actually happened between the user’s question and the final answer, I wouldn’t have been able to explain it.\n\nI felt like I was missing something.\n\nSo I stopped using LangChain for a while.\n\nInstead, I tried to understand every piece of the pipeline myself.\n\nWhat problem is chunking solving?\n\nWhy do we need embeddings?\n\nWhat does a vector database actually store?\n\nWhat is the retriever retrieving?\n\nAnd where does the LLM fit into all of this?\n\nThat decision completely changed how I think about RAG.\n\nIn this article, I’ll walk through the same journey — starting from the problem RAG solves, building a simple RAG pipeline, and finally seeing how LangChain connects all of those pieces together.\n\nLet’s say you ask an AI assistant:\n\n**“What’s our company’s leave policy?”**\n\nor\n\n*“*How do I claim travel expenses?”\n\nThe LLM has never seen your company’s internal documents or private data.\n\nSo instead of answering from your company knowledge, it’ll generate an answer that **sounds** right.\n\nThe tricky part is that it doesn’t sound like a guess.\n\nIt sounds confident.\n\nWhy does this happen?\n\nBecause an LLM can only answer using what it learned during training.\n\nYour company’s documents, internal knowledge base, or private data were never part of that training.\n\nSo when you ask about them, the model fills in the missing information with what it thinks is the most likely answer.\n\nThat’s what we call a **hallucination**.\n\nAt this point, you might be thinking:\n\n**“Why not just retrain the model with our company data?”**\n\nThe problem is that company data keeps changing.\n\nNew documents are added.\n\nPolicies get updated.\n\nKnowledge bases keep growing.\n\nRetraining a large language model every time something changes would be slow, expensive, and impractical.\n\nSo we need a different approach.\n\nInstead of expecting the model to already know everything, what if we could simply give it the right information when someone asks a question?\n\nThat’s exactly the problem RAG solves.\n\nAt a high level, RAG is actually pretty simple.\n\nA user asks a question.\n\nInstead of immediately sending that question to the LLM, the system first looks for information related to it.\n\nThat information is then added to the prompt before the LLM generates the answer.\n\n**But how does the system actually find that information?**\n\nLet’s walk through it.\n\nFirst, we need the documents we want the LLM to use.\n\nThese could be company policies, product documentation, PDFs, internal guides, or anything else we want the system to answer questions about.\n\nWe can’t just send the entire document to the LLM every time someone asks a question.\n\nIt would waste tokens, increase the cost, and could eventually run into the model’s context-window limit.\n\nSo instead, we split the documents into smaller pieces called **chunks**.\n\nFor example, a company handbook might contain sections about leave, travel reimbursement, and work-from-home policies.\n\nInstead of treating the whole handbook as one large piece of text, we split it into smaller chunks.\n\nNow we have smaller pieces of information.\n\nBut how do we know which chunk contains the answer?\n\nLet’s say someone asks:\n\n**“How do I claim travel expenses?”**\n\nWe need a way to find the chunk that talks about travel reimbursement.\n\nThis is where **embeddings** come in.\n\nAn embedding converts a piece of text into numbers that represent its meaning.\n\nSo our chunks become something like:\n\nWe then store these embeddings in a **vector database**.\n\nThe important part is that the vector database lets us search based on meaning, not just exact words.\n\nNow the user asks:\n\n**“How do I claim travel expenses?”**\n\nWe convert the question into an embedding as well.\n\nThe system then compares the question’s embedding with the stored embeddings and finds the chunks that are most similar.\n\nA common way to measure this similarity is **cosine similarity**.\n\nIn simple terms, the closer two embeddings are, the more similar their meaning is considered to be.\n\nIn our example, the **Travel Policy** chunk should be one of the closest matches.\n\nFinally, we send both the user’s question and the retrieved chunks to the LLM.\n\nSomething like:\n\nNow the LLM has the context it needs to answer the question accurately.\n\nThat’s RAG at a high level.\n\nThe important thing to notice is that **the LLM isn’t being retrained with your documents.**\n\nWe’re simply finding the relevant information and giving it to the model when it’s needed.\n\nNow that we understand how RAG works, let’s build it ourselves.\n\nThis time, we’re going to skip LangChain and build the pipeline from scratch.\n\nThat way, we’ll understand what each piece is actually doing before using a framework that abstracts it away.\n\nThe pipeline we’ll build looks like this:\n\nFor this example, we’ll use:\n\nThe goal isn’t to build a production-ready RAG system.\n\nI just want to understand what’s happening behind the scenes.\n\nOnce this works, we’ll bring in LangChain and see what it actually simplifies.\n\nLet’s build the same company-policy example we’ve been using throughout this article.\n\n``` python\nimport faissimport numpy as npimport ollama# Create embeddings using Ollamadef get_embedding(text):    response = ollama.embed(        model=\"nomic-embed-text\",        input=text    )    return response.embeddings[0]# Load the company documentwith open(\"company_policy.txt\", \"r\", encoding=\"utf-8\") as file:    document = file.read()# Split the document into smaller chunksdef chunk_text(document, chunk_size=500, overlap=100):    if overlap >= chunk_size:        raise ValueError(\"overlap must be smaller than chunk_size\")    chunks = []    start = 0    while start < len(document):        end = start + chunk_size        chunks.append(document[start:end])        start += chunk_size - overlap    return chunkschunks = chunk_text(document)# Create an embedding for every chunkknowledge_base = []for chunk in chunks:    embedding = get_embedding(chunk)    knowledge_base.append({        \"text\": chunk,        \"embedding\": embedding    })# Create the FAISS indexembeddings = np.array(    [item[\"embedding\"] for item in knowledge_base],    dtype=np.float32)# nomic-embed-text produces 768-dimensional embeddingsindex = faiss.IndexFlatL2(768)# Add document embeddings to the indexindex.add(embeddings)# Get the user's questionquestion = input(\"Ask a question: \")query_embedding = get_embedding(question)query_vector = np.array(    [query_embedding],    dtype=np.float32)# Retrieve the most relevant chunksdistances, indices = index.search(    query_vector,    k=2)top_chunks = [    knowledge_base[index][\"text\"]    for index in indices[0]]context = \"\\n\\n\".join(top_chunks)# Build the prompt using the retrieved contextprompt = f\"\"\"Answer the question using only the company information below.Company Information:{context}Question:{question}If the answer is not present in the company information,say that you don't have enough information to answer.\"\"\"# Send the prompt to the LLMresponse = ollama.chat(    model=\"llama3\",    messages=[        {            \"role\": \"user\",            \"content\": prompt        }    ])# Print the final answerprint(\"\\nAnswer:\")print(response[\"message\"][\"content\"])\n```\n\nIf you compare the code with the diagram from the previous section, you’ll notice that every step maps directly to the RAG pipeline.\n\nNothing magical happened.\n\nWe simply took a document, found the pieces that were relevant to the user’s question, and gave those pieces to the LLM as context.\n\nThat’s the core idea behind RAG.\n\nBut while building this, I noticed something.\n\nI was writing a lot of code that wasn’t really specific to my application.\n\nLoading documents.\n\nCreating embeddings.\n\nManaging the vector database.\n\nSearching for relevant chunks.\n\nBuilding prompts.\n\nIt works, and it’s a great way to understand how RAG works under the hood.\n\nBut if you were building a real application, you’d probably want a cleaner way to connect all of these pieces together.\n\nThat’s exactly where **LangChain** comes in.\n\nNow that we’ve built the pipeline ourselves, it’s much easier to see what LangChain is actually doing.\n\nIt isn’t replacing RAG.\n\nIt’s giving us ready-made components for the pieces we built manually.\n\nHere’s the rough mapping:\n\nLet’s build the same company-policy example again using LangChain.\n\n``` python\nfrom langchain_community.document_loaders import TextLoaderfrom langchain_text_splitters import RecursiveCharacterTextSplitterfrom langchain_ollama import OllamaEmbeddings, ChatOllamafrom langchain_community.vectorstores import FAISSfrom langchain_core.prompts import ChatPromptTemplate# 1. Load the company policy documentloader = TextLoader(    \"company_policy.txt\",    encoding=\"utf-8\")documents = loader.load()# 2. Split the document into chunkstext_splitter = RecursiveCharacterTextSplitter(    chunk_size=500,    chunk_overlap=100)chunks = text_splitter.split_documents(documents)# 3. Create embeddingsembeddings = OllamaEmbeddings(    model=\"nomic-embed-text\")# 4. Create the vector storevectorstore = FAISS.from_documents(    chunks,    embeddings)# 5. Create a retrieverretriever = vectorstore.as_retriever(    search_kwargs={\"k\": 2})# 6. Get the user's questionquestion = input(\"Question\": )# 7. Retrieve relevant chunksrelevant_chunks = retriever.invoke(question)context = \"\\n\\n\".join(    document.page_content    for document in relevant_chunks)# 8. Build the promptprompt = ChatPromptTemplate.from_template(    \"\"\"    Answer the question using only the company information below.    Company Information:    {context}    Question:    {question}    If the answer is not present in the company information,    say that you don't have enough information to answer.    \"\"\")# 9. Create the LLMllm = ChatOllama(    model=\"llama3\")# 10. Generate the answermessages = prompt.format_messages(    context=context,    question=question)response = llm.invoke(messages)print(response.content)\n```\n\nIf you compare it with our manual implementation, you’ll notice that the overall flow hasn’t changed.\n\nInstead of writing our own chunking function, we use:\n\n```\ntext_splitter = RecursiveCharacterTextSplitter(    chunk_size=500,    chunk_overlap=100)\n```\n\nInstead of calling ollama.embed() ourselves, we use:\n\n```\nembeddings = OllamaEmbeddings(    model=\"nomic-embed-text\")\n```\n\nInstead of creating and managing the FAISS index manually, we simply write:\n\n```\nvectorstore = FAISS.from_documents(    chunks,    embeddings)\n```\n\nAnd instead of searching the vector store ourselves, we use a retriever:\n\n```\nretriever = vectorstore.as_retriever(    search_kwargs={\"k\": 2})\n```\n\nUnder the hood, nothing has really changed.\n\nWe’re still loading documents, creating embeddings, retrieving the most relevant chunks, and sending them to the LLM.\n\nThe difference is that LangChain gives us reusable components instead of making us build every piece ourselves.\n\nBuilding RAG from scratch completely changed the way I look at it.\n\nBefore this, I thought RAG was something you built with LangChain.\n\nNow I see it differently.\n\n**RAG isn’t tied to LangChain.**\n\n**It’s simply a workflow for retrieving relevant information and giving it to an LLM before it generates an answer.**\n\n**LangChain is just one of the tools that helps you build that workflow.**\n\nThat was the biggest shift for me.\n\nOnce I understood what was happening behind the scenes, reading LangChain code became much easier. I wasn’t just copying classes and methods anymore — I actually knew what each component was doing and why it was there.\n\nWill I still use LangChain?\n\nAbsolutely.\n\nIf I were building a real application today, I’d still use it because it saves a lot of time.\n\nBut I’m really glad I built the pipeline myself first.\n\nIt helped me understand what was happening under the hood instead of treating everything like a black box.\n\nFor me, that was the biggest takeaway from this whole journey.\n\n[I Built RAG From Scratch Before Touching LangChain. Here’s What I Learned.](https://blog.devgenius.io/i-built-rag-from-scratch-before-touching-langchain-heres-what-i-learned-73a81288d87c) was originally published in [Dev Genius](https://blog.devgenius.io) on Medium, where people are continuing the conversation by highlighting and responding to this story.", "url": "https://wpnews.pro/news/i-built-rag-from-scratch-before-touching-langchain-heres-what-i-learned", "canonical_source": "https://blog.devgenius.io/i-built-rag-from-scratch-before-touching-langchain-heres-what-i-learned-73a81288d87c?source=rss----4e2c1156667e---4", "published_at": "2026-07-31 09:12:07+00:00", "updated_at": "2026-07-31 09:29:31.807615+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-tools", "ai-infrastructure"], "entities": ["LangChain", "RAG"], "alternates": {"html": "https://wpnews.pro/news/i-built-rag-from-scratch-before-touching-langchain-heres-what-i-learned", "markdown": "https://wpnews.pro/news/i-built-rag-from-scratch-before-touching-langchain-heres-what-i-learned.md", "text": "https://wpnews.pro/news/i-built-rag-from-scratch-before-touching-langchain-heres-what-i-learned.txt", "jsonld": "https://wpnews.pro/news/i-built-rag-from-scratch-before-touching-langchain-heres-what-i-learned.jsonld"}}