{"slug": "building-a-rag-system-from-scratch-four-components-one-working-pipeline", "title": "Building a RAG System From Scratch — Four Components, One Working Pipeline", "summary": "A developer built a complete retrieval-augmented generation (RAG) pipeline from scratch using LangChain, ChromaDB, and a local LLM via LM Studio, avoiding cloud costs and API keys. The system splits HR documents into chunks, embeds them with nomic-embed-text, stores vectors in ChromaDB, and retrieves relevant passages for a Qwen 9B model to answer employee questions. The developer highlighted key pitfalls, such as the need to use the same embedding model for indexing and querying and the tight coupling between chunk size and embeddings.", "body_md": "Most RAG tutorials explain the concept. This one shows the code — a complete working pipeline using LangChain, ChromaDB, and a local LLM via LM Studio. No OpenAI API key. No cloud costs.\n\n**The business problem:** A company has hundreds of pages of HR documentation. Employees ask questions. An AI answers accurately from the actual documents in seconds.\n\n**Stack:** Python, LangChain, ChromaDB, nomic-embed-text, Qwen 9B via LM Studio\n\n```\npip install langchain langchain-community langchain-chroma langchain-openai langchain-text-splitters langchain-core openai requests\n```\n\nMake sure LM Studio is running with both models loaded before running the code.\n\nSplits raw text into chunks. Chunk size and overlap determine everything downstream — cut a paragraph wrong and retrieval suffers.\n\n``` python\nfrom langchain_text_splitters import RecursiveCharacterTextSplitter\nfrom langchain_core.documents import Document\nfrom typing import List\n\nclass DocumentProcessor:\n    def __init__(self, chunk_size: int = 500, chunk_overlap: int = 50):\n        self.splitter = RecursiveCharacterTextSplitter(\n            chunk_size=chunk_size,\n            chunk_overlap=chunk_overlap,\n            separators=[\"\\n\\n\", \"\\n\", \". \", \" \", \"\"]\n        )\n\n    def process(self, text: str, source: str = \"\") -> List[Document]:\n        chunks = self.splitter.split_text(text)\n        return [\n            Document(\n                page_content=chunk,\n                metadata={\"source\": source, \"chunk_index\": i}\n            )\n            for i, chunk in enumerate(chunks)\n        ]\n```\n\n⚠️ Changing chunk size later means reembedding everything from scratch. Your chunks and embeddings are tightly coupled. Choose carefully the first time.\n\nConverts text into numerical vectors. LM Studio needs a direct requests call — the standard LangChain OpenAI wrapper sends the wrong input format and throws a 400 error.\n\n``` python\nfrom langchain_core.embeddings import Embeddings\nimport requests\n\nLM_STUDIO_URL = \"http://localhost:1234/v1\"\nEMBED_MODEL   = \"nomic-embed-text\"\n\nclass EmbeddingService(Embeddings):\n    def __init__(self):\n        self.url   = f\"{LM_STUDIO_URL}/embeddings\"\n        self.model = EMBED_MODEL\n\n    def _embed(self, text: str) -> List[float]:\n        response = requests.post(\n            self.url,\n            json={\"model\": self.model, \"input\": text},\n            headers={\"Authorization\": \"Bearer lm-studio\"}\n        )\n        response.raise_for_status()\n        return response.json()[\"data\"][0][\"embedding\"]\n\n    def embed_documents(self, texts: List[str]) -> List[List[float]]:\n        return [self._embed(text) for text in texts]\n\n    def embed_query(self, text: str) -> List[float]:\n        return self._embed(text)\n```\n\n⚠️ Same model must be used for both indexing and querying. They must share the same vector space. Switching models means rebuilding the entire index.\n\nStores embeddings and retrieves closest matches by meaning — not by keyword. ChromaDB persists to disk so your index survives restarts without reprocessing documents.\n\n``` python\nfrom langchain_chroma import Chroma\nfrom typing import Tuple\n\nCHROMA_DIR      = \"./chroma_db\"\nTOP_K           = 4\nSCORE_THRESHOLD = 0.3\n\nclass VectorStore:\n    def __init__(self, embedding_service: EmbeddingService):\n        self.store = Chroma(\n            collection_name=\"rag_collection\",\n            embedding_function=embedding_service,\n            persist_directory=CHROMA_DIR\n        )\n\n    def add_documents(self, documents: List[Document]) -> None:\n        self.store.add_documents(documents)\n\n    def similarity_search(self, query: str) -> List[Tuple[Document, float]]:\n        results = self.store.similarity_search_with_relevance_scores(\n            query=query,\n            k=TOP_K,\n        )\n        return [(doc, score) for doc, score in results if score >= SCORE_THRESHOLD]\n```\n\n💡 Score threshold of 0.3 works well for small document sets. In production with larger knowledge bases tune this to 0.6–0.75 to filter out loosely relevant chunks.\n\nOrchestrates the full flow. Query in, embed it, retrieve closest chunks, build context, generate grounded answer, return sources.\n\nThe most important line in the entire pipeline is the system prompt. Without the strict instruction to answer only from context, the LLM falls back on its general training knowledge — defeating the entire purpose of RAG.\n\n``` python\nfrom langchain_openai import ChatOpenAI\nfrom langchain_core.messages import HumanMessage, SystemMessage\nfrom typing import Dict\n\nCHAT_MODEL = \"qwen/qwen3.5-9b\"\n\nclass RAGPipeline:\n    def __init__(self, vector_store: VectorStore):\n        self.vector_store = vector_store\n        self.llm = ChatOpenAI(\n            model=CHAT_MODEL,\n            openai_api_base=LM_STUDIO_URL,\n            openai_api_key=\"lm-studio\",\n            max_tokens=1000,\n            temperature=0.1\n        )\n\n    def _build_context(self, documents: List[Tuple[Document, float]]) -> str:\n        parts = []\n        for doc, score in documents:\n            parts.append(\n                f\"Source: {doc.metadata.get('source', 'unknown')}\\n\"\n                f\"Relevance: {score:.2f}\\n\"\n                f\"Content: {doc.page_content}\"\n            )\n        return \"\\n\\n---\\n\\n\".join(parts)\n\n    def query(self, question: str) -> Dict:\n        results = self.vector_store.similarity_search(question)\n\n        if not results:\n            return {\n                \"answer\": \"I could not find relevant information to answer this question.\",\n                \"sources\": [],\n                \"chunks_used\": 0\n            }\n\n        context = self._build_context(results)\n\n        messages = [\n            SystemMessage(content=(\n                \"You are a precise assistant that answers questions \"\n                \"based solely on the provided context. \"\n                \"If the answer is not in the context, say so clearly. \"\n                \"Do not use your general knowledge to supplement the context.\"\n            )),\n            HumanMessage(content=(\n                f\"Context:\\n{context}\\n\\n\"\n                f\"Question: {question}\\n\\n\"\n                f\"Answer based only on the context above:\"\n            ))\n        ]\n\n        response = self.llm(messages)\n\n        return {\n            \"answer\": response.content,\n            \"sources\": list(set(doc.metadata.get(\"source\") for doc, _ in results)),\n            \"chunks_used\": len(results)\n        }\n```\n\n💡 temperature=0.1 keeps answers factual and consistent. Low temperature means the model stays close to what the context says rather than being creative with it.\n\n```\nif __name__ == \"__main__\":\n\n    SAMPLE_DOCUMENT = \"\"\"\n    Employee Leave Policy\n\n    Sick Leave:\n    Employees are entitled to 10 days of paid sick leave per calendar year.\n    Sick leave resets on January 1st each year.\n    Unused sick leave cannot be carried over to the next year.\n    To apply for sick leave, submit a request through the HR portal.\n    A medical certificate is required for sick leave exceeding 3 consecutive days.\n\n    Annual Leave:\n    Employees receive 25 days of annual leave per year.\n    Annual leave must be approved by the line manager at least 2 weeks in advance.\n    Up to 5 unused annual leave days can be carried over to the following year.\n\n    Remote Work Policy:\n    Employees may work remotely up to 3 days per week.\n    Remote work requires a stable internet connection and a dedicated workspace.\n    Core hours of 10:00 to 16:00 must be maintained regardless of location.\n\n    Expense Policy:\n    Business travel expenses must be approved before travel.\n    Receipts are required for all expenses above 25 euros.\n    Expense reports must be submitted within 30 days of the expense.\n    Maximum meal allowance is 50 euros per day during business travel.\n    \"\"\"\n\n    # Initialise all four components\n    processor  = DocumentProcessor()\n    embeddings = EmbeddingService()\n    store      = VectorStore(embedding_service=embeddings)\n    pipeline   = RAGPipeline(vector_store=store)\n\n    # Index documents — happens once\n    documents = processor.process(SAMPLE_DOCUMENT, source=\"company_policy\")\n    store.add_documents(documents)\n\n    # Query — happens live for every user question\n    questions = [\n        \"How many sick days am I entitled to per year?\",\n        \"Can I carry over unused annual leave?\",\n        \"How many days can I work remotely?\",\n        \"What is the maximum meal allowance during business travel?\",\n        \"Do I need a medical certificate for sick leave?\",\n    ]\n\n    for q in questions:\n        result = pipeline.query(q)\n        print(f\"Q: {q}\")\n        print(f\"A: {result['answer']}\")\n        print(f\"   Sources: {result['sources']}\")\n        print(f\"   Chunks used: {result['chunks_used']}\")\n        print(\"-\" * 60)\n```\n\nThis is the actual output from running this pipeline locally:\n\n```\nQ: How many sick days am I entitled to per year?\nA: Based on the provided context, employees are entitled to 10 days\n   of paid sick leave per calendar year.\n   Sources: ['company_policy'] · Chunks used: 4\n\nQ: Can I carry over unused annual leave?\nA: Yes, up to 5 unused annual leave days can be carried over to\n   the following year.\n   Sources: ['company_policy'] · Chunks used: 4\n\nQ: How many days can I work remotely?\nA: Based on the provided context, employees may work remotely up\n   to 3 days per week.\n   Sources: ['company_policy'] · Chunks used: 4\n\nQ: What is the maximum meal allowance during business travel?\nA: The maximum meal allowance during business travel is 50 euros\n   per day.\n   Sources: ['company_policy'] · Chunks used: 4\n\nQ: Do I need a medical certificate for sick leave?\nA: A medical certificate is required for sick leave exceeding\n   3 consecutive days.\n   Sources: ['company_policy'] · Chunks used: 4\n```\n\nEvery answer accurate. Every answer sourced. No hallucination. No API key. No cloud cost.\n\nReplace `SAMPLE_DOCUMENT`\n\nwith your own content and you have a working RAG system on your documents in minutes.\n\nEvery component maps directly to an Azure service. The pipeline logic stays identical — only the initialisation changes.\n\n| Local | Azure Equivalent |\n|---|---|\n| nomic-embed-text via LM Studio | AzureOpenAIEmbeddings |\n| Qwen 9B via LM Studio | AzureChatOpenAI |\n| ChromaDB | Azure AI Search |\n\nThis implementation is intentionally minimal. In production add:\n\nThese are the patterns covered in the next post — where RAG goes from working to production-ready.", "url": "https://wpnews.pro/news/building-a-rag-system-from-scratch-four-components-one-working-pipeline", "canonical_source": "https://dev.to/rit_the_coder/building-a-rag-system-from-scratch-four-components-one-working-pipeline-5ad8", "published_at": "2026-08-12 11:40:08+00:00", "updated_at": "2026-08-12 11:47:20.641090+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "developer-tools"], "entities": ["LangChain", "ChromaDB", "LM Studio", "nomic-embed-text", "Qwen 9B"], "alternates": {"html": "https://wpnews.pro/news/building-a-rag-system-from-scratch-four-components-one-working-pipeline", "markdown": "https://wpnews.pro/news/building-a-rag-system-from-scratch-four-components-one-working-pipeline.md", "text": "https://wpnews.pro/news/building-a-rag-system-from-scratch-four-components-one-working-pipeline.txt", "jsonld": "https://wpnews.pro/news/building-a-rag-system-from-scratch-four-components-one-working-pipeline.jsonld"}}