In the last part, we worked up from the foundational problem to the key components of a RAG system. We examined eight components and how they fit together.
These eight components can be grouped into two main pipelines: the knowledge pipeline and the query pipeline.
This pipeline turns raw source information into knowledge the system can store and retrieve later. It runs when information enters or changes and aims to produce retrievable knowledge data. For example, when a new ADR document is created in Notion, we fetch it, parse its structure into a uniform format, convert it into a consistent form, store it, and index it so it can be retrieved efficiently later.
In essence, this pipeline prepares the knowledge that the retrieval part of the RAG system will eventually search. It works much like the write path in a database system.
This pipeline takes a question and finds enough relevant existing knowledge to provide as context for the LLM to answer it. It runs when you ask a question. For example, if you ask, "Why is there a separate requery service besides the payout service?", the system fetches relevant knowledge from the store populated by the knowledge pipeline, ranks and selects the most relevant information, packages it with the original question, and sends it to the LLM for an answer.
In short:
Query pipeline = Retrieval + Augmentation + Generation.
It acts much like the read path of a database system.
Splitting the system into these two halves, the flow looks like:
------------------------------------------------------------------
| KNOWLEDGE PIPELINE |
| Sources → Connectors → Ingestion → Processing → Indexing/Storage |
------------------------------------------------------------------
|
|
↓
-------------------------------------------------------------------
| QUERY PIPELINE |
|Question → (Retrieval+ Ranking) → Context Preparation → LLM →Answer|
-------------------------------------------------------------------
Separating these two parts solidifies a mental model: one prepares the knowledge, while the other consumes it.
A good analogy is search engines. Google does not crawl and process the entire internet every time you search. A background process crawls the web, processes it, and indexes the data. When you search, another part of the system searches those indexes, retrieves matching results, ranks them, and shows you the answers.
A RAG system follows a similar pattern but for your own knowledge and with an LLM at the end.
Let's go deeper into the knowledge pipeline.
The connector lets us interact with knowledge sources. It's source-specific: we might have a Slack connector, a Notion connector, a Google Drive connector, etc.—like separate database drivers for different systems.
For a Slack connector, for example, we need to:
The output should remain close to the source, for example:
SourceRecord
- source_type: slack
- source_id: "1712345678.1234"
- raw_content: "We decided to use Redis..."
- source_metadata:
channel: engineering
thread_id: ...
author: ...
timestamp: ...
While the connector answers,
How do I talk to this source?
The ingestion layer answers:
Which source records should enter or re-enter my knowledge pipeline?
When dealing with source changes, the ingestion layer decides:
Using the Slack example, suppose you have this in a Slack channel:
#engineering
Timi:
"We decided to use Redis locking because two workers could update the wallet concurrently."
The flow becomes:
Slack API → SlackConnector → SourceRecord {id, message, channel, author, timestamp, thread}
→ Ingestion service.
The ingestion service asks:
"Have I seen this Slack message before?"
For the knowledge base we’re building, I’m choosing to keep connector and ingestion logic as separate architectural responsibilities. While combining them seems simpler, as sources grow, two dimensions start changing independently:
Because these have different reasons to change, separating them is important.
Receive/discover source changes
↓
Determine create/update/delete
↓
Submit changes downstream
↓
Persist sync/change state if required
The ingestion layer shouldn’t care about the transport; its job is deciding how new/updated/deleted data affects your knowledge store.
A common ingestion engine can also handle:
In this part, we moved from seeing a RAG system as a single linear flow to separating it into two major pipelines: the knowledge pipeline and the query pipeline. This separation offers a clearer architectural picture—one side prepares knowledge, and the other consumes it when a question arrives.
We started breaking down the knowledge pipeline by exploring connector and ingestion layers. The connector is source-specific, while ingestion decides what to do with incoming, changing, or deleted data.
The key architectural takeaway: these are different responsibilities with different reasons to change. As knowledge sources grow, this separation should make the system easier to extend and manage.
In the next part, we’ll continue along the knowledge pipeline and examine what happens after data’s been ingested—how raw source records are processed into a form the rest of the system can use.