{"slug": "designing-an-end-to-end-rag-architecture-from-scratch", "title": "Designing an End-to-End RAG Architecture from Scratch", "summary": "An engineer built Guidely, an internal knowledge assistant, and detailed its end-to-end Retrieval-Augmented Generation (RAG) architecture. The system separates document ingestion, parsing, chunking, embeddings, vector storage, semantic search, and response generation into distinct, testable stages. The design emphasizes clear component responsibilities, enabling reliable retrieval and citation-backed answers.", "body_md": "Building an AI-powered application is easy to describe.\n\nUpload documents → ask a question → get an answer.\n\nActually building that flow is a different story.\n\nWhile working on **Guidely**, an internal knowledge assistant, I wanted to understand what happens between those two endpoints. More importantly, I wanted to design the system so that each part had a clear responsibility and could be tested independently.\n\nThe result was an end-to-end Retrieval-Augmented Generation (RAG) system built around a simple pipeline:\n\n```\nDocuments\n    ↓\nParsing\n    ↓\nChunking\n    ↓\nEmbeddings\n    ↓\nVector Store\n    ↓\nSemantic Search\n    ↓\nResponse Generation\n    ↓\nCitations\n    ↓\nReact Frontend\n```\n\nThe interesting part wasn't simply getting an LLM to answer questions.\n\nIt was designing the architecture that makes the entire pipeline work reliably.\n\nGuidely is an internal knowledge assistant that allows users to ask questions about a collection of organizational documents.\n\nInstead of expecting an AI model to already know everything about an organization's internal knowledge, Guidely retrieves relevant information from the organization's documents and uses that information to construct an answer.\n\nFor example, a user might ask:\n\n\"What is TrustLayer?\"\n\nGuidely searches the organization's knowledge base, retrieves the most relevant sections, and uses those sections as context for generating the answer.\n\nThe response is then presented together with the sources that support it.\n\nThis is the basic idea behind RAG.\n\nBut I wanted the architecture to make the flow explicit rather than hiding everything inside one large function.\n\nThe first major decision was to separate the system into distinct stages.\n\n```\n                    ┌─────────────────┐\n                    │    Documents    │\n                    └────────┬────────┘\n                             ↓\n                    ┌─────────────────┐\n                    │     Parser      │\n                    └────────┬────────┘\n                             ↓\n                    ┌─────────────────┐\n                    │     Chunker     │\n                    └────────┬────────┘\n                             ↓\n                    ┌─────────────────┐\n                    │   Embeddings    │\n                    └────────┬────────┘\n                             ↓\n                    ┌─────────────────┐\n                    │   Vector Store  │\n                    └────────┬────────┘\n                             ↓\n                         User Query\n                             ↓\n                    ┌─────────────────┐\n                    │ Semantic Search │\n                    └────────┬────────┘\n                             ↓\n                    ┌─────────────────┐\n                    │    Response     │\n                    └────────┬────────┘\n                             ↓\n                    ┌─────────────────┐\n                    │ React Frontend  │\n                    └─────────────────┘\n```\n\nEach component answers a different question.\n\nThat separation became one of the most important architectural decisions in the project.\n\nThe first stage is getting documents into the system.\n\nGuidely supports documents such as:\n\nThe upload API receives a document and stores it in the document directory.\n\nBut uploading a file isn't the same thing as making it searchable.\n\nThe document needs to go through the ingestion pipeline.\n\n```\nUploaded document\n       ↓\nDetermine file type\n       ↓\nParse document\n       ↓\nExtract text\n       ↓\nChunk text\n       ↓\nGenerate embeddings\n       ↓\nStore vectors + metadata\n```\n\nThis separation means the upload layer doesn't need to understand embeddings or semantic search.\n\nIts responsibility is simply:\n\nGet the document into the system.\n\nOnce a document exists, Guidely needs to extract its text.\n\nThe parser provides a common interface:\n\n```\nparse_document(file_path)\n```\n\nInternally, the appropriate parser can be selected depending on the file type.\n\nFor example:\n\n```\n.txt   → text parser\n.pdf   → PDF parser\n.docx  → DOCX parser\n```\n\nThe important architectural idea here is that the rest of the pipeline doesn't need to care where the text came from.\n\nOnce parsing is complete, everything downstream works with:\n\n```\ntext: str\n```\n\nThis keeps the pipeline format-independent.\n\nA document can be thousands of words long.\n\nSending an entire document into a retrieval system isn't ideal.\n\nInstead, Guidely breaks the extracted text into smaller chunks.\n\nThe chunking strategy uses tokens rather than simply splitting every N characters.\n\nFor example:\n\n```\nDocument\n────────────────────────────\n\nParagraph 1\nParagraph 2\nParagraph 3\nParagraph 4\nParagraph 5\n...\n\nChunk 1\n────────────────\nParagraph 1\nParagraph 2\n\nChunk 2\n────────────────\nParagraph 2\nParagraph 3\n\nChunk 3\n────────────────\nParagraph 3\nParagraph 4\n```\n\nThe overlap is intentional.\n\nIf a relevant sentence happens to sit near a chunk boundary, overlap reduces the chance that important context gets separated.\n\nThe chunking function therefore has two important parameters:\n\n```\nchunk_size = 800\noverlap = 100\n```\n\nThe exact values can be tuned later.\n\nThe important design decision was making chunking its own service rather than embedding the logic inside document ingestion.\n\nThis is where the system starts moving from traditional text processing into semantic search.\n\nEach chunk is converted into a vector representation.\n\nConceptually:\n\n```\n\"TrustLayer is a decentralized protocol...\"\n                    ↓\n              Embedding Model\n                    ↓\n        [0.021, -0.143, 0.782, ...]\n```\n\nThe same process happens when a user asks a question.\n\n```\n\"What is TrustLayer?\"\n          ↓\n     Embedding Model\n          ↓\n    Query Vector\n```\n\nNow the system can compare the query vector with document vectors.\n\nThis is the foundation of semantic retrieval.\n\nOne challenge I encountered here was model selection.\n\nI initially explored hosted embedding APIs but ran into API limitations. I eventually moved toward a local Sentence Transformers model.\n\nThat decision had an architectural benefit beyond simply solving the immediate problem:\n\n**the embedding layer became independent from the rest of the application.**\n\nIf I change the embedding model later, the search API doesn't need to change.\n\nOnce chunks have embeddings, the vectors need to be stored somewhere.\n\nFor Guidely, I used FAISS.\n\nThe basic relationship looks like:\n\n```\nVector\n   │\n   ├── FAISS index\n   │\n   └── Metadata\n         ├── filename\n         ├── chunk information\n         └── original text\n```\n\nThe vector index handles similarity search.\n\nThe metadata provides the information needed to understand what a vector represents.\n\nThis separation is important.\n\nFAISS answers:\n\nWhich vectors are closest to this query?\n\nThe metadata answers:\n\nWhat do those vectors actually represent?\n\nWhen a user submits a question, the query follows a shorter path:\n\n```\nUser question\n     ↓\nCreate embedding\n     ↓\nFAISS similarity search\n     ↓\nTop K results\n     ↓\nRelevant document chunks\n```\n\nThe search service looks roughly like this:\n\n```\nquery_embedding = create_embeddings([query])[0]\n\ndistances, indices = index.search(\n    query_vector,\n    top_k\n)\n```\n\nThe returned vector IDs are then mapped back to document metadata.\n\nOne useful property of this architecture is that the search service doesn't need to know anything about the frontend.\n\nIt simply returns structured results.\n\nFor example:\n\n```\n{\n  \"filename\": \"faq.txt\",\n  \"text\": \"TrustLayer is a decentralized protocol...\"\n}\n```\n\nOne of the more important problems I encountered was handling irrelevant questions.\n\nA vector database will usually return *something*.\n\nEven if the user asks a question completely unrelated to the knowledge base, FAISS can still return the nearest vectors.\n\nThat creates a dangerous situation:\n\n```\nIrrelevant question\n       ↓\nSimilarity search\n       ↓\nSome vaguely similar chunks\n       ↓\nAI generates an answer\n       ↓\nCitation appears\n```\n\nThe system can therefore look confident even when it shouldn't be answering.\n\nThis led to an important architectural requirement:\n\nRetrieval needs a relevance boundary.\n\nInstead of blindly accepting the top K results, the system needs to determine whether the retrieved results are actually relevant enough to support an answer.\n\nThis is also where citations need to be handled carefully.\n\nA citation should not appear simply because a document happened to be returned by FAISS.\n\nAfter retrieval, the relevant chunks become context for the response layer.\n\nConceptually:\n\n```\nUser Question\n      +\nRetrieved Context\n      ↓\nResponse Generator\n      ↓\nAnswer + Sources\n```\n\nThe backend returns a structured response rather than exposing internal implementation details.\n\nFor example:\n\n```\n{\n  \"answer\": \"TrustLayer is a decentralized protocol on Solana...\",\n  \"citations\": [\n    {\n      \"source\": \"faq.txt\",\n      \"snippet\": \"It allows clients and talent to collaborate directly...\"\n    }\n  ]\n}\n```\n\nThis distinction matters.\n\nThe backend can contain things such as:\n\nCitations became an interesting part of the project.\n\nInitially, returning the entire retrieved chunk produced poor results.\n\nA citation could contain an entire section of a document even when only one sentence supported the answer.\n\nFor example:\n\n```\nfaq.txt\n\nTRUSTLAYER FREQUENTLY ASKED QUESTIONS\n...\nTABLE OF CONTENTS\n...\nQ1...\nQ2...\nQ3...\n```\n\nThat's technically a citation, but it isn't particularly useful to a human.\n\nThe goal became:\n\n```\nfaq.txt\n\n\"It allows clients and talent to collaborate directly...\"\n```\n\nThe citation should answer:\n\n\"Where did this information come from?\"\n\nnot:\n\n\"Here is a large portion of the document.\"\n\nThis led to a query-aware citation strategy where the snippet is selected based on the information relevant to the user's question.\n\nThe frontend is intentionally separated from the retrieval system.\n\nI built the interface with React.\n\nThe user sees:\n\n```\n                 Guidely\n\n      Ask your organization's knowledge\n\n ┌──────────────────────────────────────────┐\n │ Ask a question...                    →  │\n └──────────────────────────────────────────┘\n\n Answer\n\n TrustLayer is a decentralized protocol...\n\n Sources\n\n ┌──────────────────────────────────────────┐\n │ 📄 faq.txt                               │\n │    It allows clients and talent...       │\n └──────────────────────────────────────────┘\n```\n\nThe frontend doesn't need to know how embeddings work.\n\nIt doesn't know what FAISS is.\n\nIt doesn't need to understand chunking.\n\nIt simply consumes the response contract from the API.\n\nThe second major frontend surface is the knowledge-base administration page.\n\nThe admin interface allows documents to be uploaded and viewed.\n\nThe architecture looks like:\n\n```\nAdmin\n  ↓\nUpload document\n  ↓\nFastAPI\n  ↓\nDocument storage\n  ↓\nIngestion pipeline\n  ↓\nEmbeddings\n  ↓\nFAISS\n```\n\nThe interface intentionally hides implementation details.\n\nAn administrator doesn't need to know:\n\n\"Your document has been converted into a 768-dimensional vector and inserted at index 42.\"\n\nThey need to know:\n\n\"Your document has been uploaded and is available.\"\n\nThis distinction influenced a lot of the UI decisions.\n\nFastAPI became the boundary between the frontend and the internal services.\n\nThe application is organized around responsibilities such as:\n\n```\napp/\n├── routers/\n│   ├── search.py\n│   └── documents.py\n│\n├── services/\n│   ├── parser.py\n│   ├── chunker.py\n│   ├── embeddings.py\n│   ├── vector_store.py\n│   └── response.py\n│\n└── main.py\n```\n\nThis structure isn't about creating as many files as possible.\n\nIt's about making the data flow understandable.\n\nA search request can be traced through:\n\n```\nsearch router\n     ↓\nsearch service\n     ↓\nembedding service\n     ↓\nvector store\n     ↓\nresponse service\n```\n\nSimilarly, document ingestion has its own path.\n\nThat makes debugging considerably easier.\n\nThe biggest challenge wasn't writing the individual functions.\n\nIt was deciding where each responsibility belonged.\n\nFor example, it would have been possible to create one large function:\n\n``` python\ndef ask_question(query):\n    # create embedding\n    # search FAISS\n    # retrieve documents\n    # generate response\n    # format citations\n    # return result\n```\n\nIt would probably work.\n\nBut it would also become difficult to test and modify.\n\nInstead, Guidely separates those responsibilities.\n\nThat gives me the ability to change:\n\nwithout necessarily rewriting the entire system.\n\nAfter putting everything together, the final architecture looks like this:\n\n```\n                    DOCUMENT INGESTION\n\nDocument\n   ↓\nParser\n   ↓\nText\n   ↓\nChunker\n   ↓\nChunks\n   ↓\nEmbedding Model\n   ↓\nVectors\n   ↓\nFAISS + Metadata\n   │\n   │\n   │\n   ▼\n                    QUERY PIPELINE\n\nUser Question\n   ↓\nEmbedding Model\n   ↓\nQuery Vector\n   ↓\nFAISS Similarity Search\n   ↓\nRelevant Chunks\n   ↓\nRelevance Filtering\n   ↓\nResponse Generation\n   ↓\nAnswer + Query-Aware Citations\n   ↓\nReact UI\n```\n\nThe current architecture works, but there are several areas I would improve as the project evolves.\n\nSimilarity scores alone aren't enough.\n\nI'd like to build a proper evaluation dataset containing:\n\nThis would make retrieval quality measurable rather than something I evaluate manually.\n\nDifferent documents have different structures.\n\nA fixed token-based chunk size isn't necessarily optimal for:\n\nA future version could use structure-aware chunking.\n\nCitation snippets could be selected more intelligently based on the query and the generated answer.\n\nFAISS works well for a project like this, but a production deployment could benefit from a persistent vector database depending on scale and operational requirements.\n\nThe current admin functionality is primarily focused on document management.\n\n*A production knowledge assistant would also need authentication, authorization, document ownership, and potentially per-user or per-team knowledge bases.*\n\nThe most valuable part of this project wasn't getting an AI model to answer a question.\n\nIt was learning to think about the system as a collection of independent stages.\n\nA useful mental model is:\n\n```\nDon't start with:\n\n\"How do I make an AI answer questions?\"\n\nStart with:\n\n\"How does information move through the system?\"\n```\n\nOnce that question is answered, the architecture becomes much clearer.\n\nDocuments become text.\n\nText becomes chunks.\n\nChunks become vectors.\n\nVectors become searchable knowledge.\n\nSearch results become context.\n\nContext becomes an answer.\n\nAnd the answer becomes something a human can actually use.\n\nThat is the architecture behind Guidely: a small but complete end-to-end RAG system designed not just to work, but to make each stage understandable, replaceable, and testable.\n\nThe next challenge is measuring how well each stage works.", "url": "https://wpnews.pro/news/designing-an-end-to-end-rag-architecture-from-scratch", "canonical_source": "https://dev.to/odingaval/designing-an-end-to-end-rag-architecture-from-scratch-230i", "published_at": "2026-08-11 21:03:25+00:00", "updated_at": "2026-08-11 21:17:36.339617+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "ai-products", "developer-tools"], "entities": ["Guidely", "TrustLayer"], "alternates": {"html": "https://wpnews.pro/news/designing-an-end-to-end-rag-architecture-from-scratch", "markdown": "https://wpnews.pro/news/designing-an-end-to-end-rag-architecture-from-scratch.md", "text": "https://wpnews.pro/news/designing-an-end-to-end-rag-architecture-from-scratch.txt", "jsonld": "https://wpnews.pro/news/designing-an-end-to-end-rag-architecture-from-scratch.jsonld"}}