{"slug": "building-my-first-rag-system-from-components-to-knowledge-and-query-pipelines", "title": "Building My First RAG System: From Components to Knowledge and Query Pipelines - Part Two", "summary": "A developer detailed the architecture of a retrieval-augmented generation (RAG) system, breaking it into two main pipelines: the knowledge pipeline, which ingests and indexes source data, and the query pipeline, which retrieves relevant context and generates answers. The post explains how connectors, ingestion, processing, and indexing form the knowledge pipeline, while retrieval, ranking, and generation compose the query pipeline, drawing an analogy to search engines.", "body_md": "In the last [part](https://dev.to/timilehin-olusegun/building-my-first-rag-system-deriving-the-architecture-from-first-principles-part-one-n43), we worked up from the foundational problem to the key components of a RAG system. We examined eight components and how they fit together.\n\nThese eight components can be grouped into two main pipelines: **the knowledge pipeline** and **the query pipeline**.\n\nThis pipeline turns raw source information into knowledge the system can store and retrieve later. It runs when information enters or changes and aims to produce retrievable knowledge data. For example, when a new ADR document is created in Notion, we fetch it, parse its structure into a uniform format, convert it into a consistent form, store it, and index it so it can be retrieved efficiently later.\n\nIn essence, this pipeline prepares the knowledge that the retrieval part of the RAG system will eventually search. It works much like the write path in a database system.\n\nThis pipeline takes a question and finds enough relevant existing knowledge to provide as context for the LLM to answer it. It runs when you ask a question. For example, if you ask, \"Why is there a separate requery service besides the payout service?\", the system fetches relevant knowledge from the store populated by the knowledge pipeline, ranks and selects the most relevant information, packages it with the original question, and sends it to the LLM for an answer.\n\nIn short:\n\n**Query pipeline = Retrieval + Augmentation + Generation.**\n\nIt acts much like the read path of a database system.\n\nSplitting the system into these two halves, the flow looks like:\n\n```\n ------------------------------------------------------------------\n|                      KNOWLEDGE PIPELINE                          |\n| Sources → Connectors → Ingestion → Processing → Indexing/Storage |\n ------------------------------------------------------------------\n                              |\n                              |\n                              ↓\n -------------------------------------------------------------------\n|                       QUERY PIPELINE                              |\n|Question → (Retrieval+ Ranking) → Context Preparation → LLM →Answer|\n -------------------------------------------------------------------\n```\n\nSeparating these two parts solidifies a mental model: one prepares the knowledge, while the other consumes it.\n\nA good analogy is search engines. Google does not crawl and process the entire internet every time you search. A background process crawls the web, processes it, and indexes the data. When you search, another part of the system searches those indexes, retrieves matching results, ranks them, and shows you the answers.\n\nA RAG system follows a similar pattern but for your own knowledge and with an LLM at the end.\n\nLet's go deeper into the knowledge pipeline.\n\nThe connector lets us interact with knowledge sources. It's source-specific: we might have a Slack connector, a Notion connector, a Google Drive connector, etc.—like separate database drivers for different systems.\n\nFor a Slack connector, for example, we need to:\n\nThe output should remain close to the source, for example:\n\n```\nSourceRecord\n- source_type: slack\n- source_id: \"1712345678.1234\"\n- raw_content: \"We decided to use Redis...\"\n- source_metadata:\n    channel: engineering\n    thread_id: ...\n    author: ...\n    timestamp: ...\n```\n\nWhile the connector answers,\n\nHow do I talk to this source?\n\nThe ingestion layer answers:\n\nWhich source records should enter or re-enter my knowledge pipeline?\n\nWhen dealing with source changes, the ingestion layer decides:\n\nUsing the Slack example, suppose you have this in a Slack channel:\n\n```\n#engineering\n\nTimi:  \n\"We decided to use Redis locking because two workers could update the wallet concurrently.\"\n```\n\nThe flow becomes:\n\nSlack API → SlackConnector → SourceRecord `{id, message, channel, author, timestamp, thread}`\n\n→ Ingestion service.\n\nThe ingestion service asks:\n\n\"Have I seen this Slack message before?\"\n\nFor the knowledge base we’re building, I’m choosing to keep connector and ingestion logic as separate architectural responsibilities. While combining them seems simpler, as sources grow, two dimensions start changing independently:\n\nBecause these have different reasons to change, separating them is important.\n\n```\nReceive/discover source changes\n      ↓\nDetermine create/update/delete\n      ↓\nSubmit changes downstream\n      ↓\nPersist sync/change state if required\n```\n\nThe ingestion layer shouldn’t care about the transport; its job is deciding how new/updated/deleted data affects your knowledge store.\n\nA common ingestion engine can also handle:\n\nIn this part, we moved from seeing a RAG system as a single linear flow to separating it into two major pipelines: the knowledge pipeline and the query pipeline. This separation offers a clearer architectural picture—one side prepares knowledge, and the other consumes it when a question arrives.\n\nWe started breaking down the knowledge pipeline by exploring connector and ingestion layers. The connector is source-specific, while ingestion decides what to do with incoming, changing, or deleted data.\n\nThe key architectural takeaway: these are different responsibilities with different reasons to change. As knowledge sources grow, this separation should make the system easier to extend and manage.\n\nIn the next part, we’ll continue along the knowledge pipeline and examine what happens after data’s been ingested—how raw source records are processed into a form the rest of the system can use.", "url": "https://wpnews.pro/news/building-my-first-rag-system-from-components-to-knowledge-and-query-pipelines", "canonical_source": "https://dev.to/timilehin-olusegun/building-my-first-rag-system-from-components-to-knowledge-and-query-pipelines-part-two-jdj", "published_at": "2026-08-30 16:23:21+00:00", "updated_at": "2026-08-30 16:53:10.321955+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-infrastructure", "developer-tools"], "entities": ["RAG", "Notion", "Slack", "Google Drive", "LLM"], "alternates": {"html": "https://wpnews.pro/news/building-my-first-rag-system-from-components-to-knowledge-and-query-pipelines", "markdown": "https://wpnews.pro/news/building-my-first-rag-system-from-components-to-knowledge-and-query-pipelines.md", "text": "https://wpnews.pro/news/building-my-first-rag-system-from-components-to-knowledge-and-query-pipelines.txt", "jsonld": "https://wpnews.pro/news/building-my-first-rag-system-from-components-to-knowledge-and-query-pipelines.jsonld"}}