I recently read an article about a VC who uses AI to boost his productivity. He described building a knowledge base using NotebookLM, and one point that stuck with me was:
Every time I read something online that I thought I wanted to remember, I'd copy and paste it into that repository. Whenever I wanted to write a blog post, I could query it and retrieve all the information I needed.
Like him, I have knowledge and resources scattered across Logseq, Gmail, Notion, ADR documents, Slack, project readmes, Markdown files, Twitter, and more. That made me wonder: how could I build my own system? Tools like NotebookLM exist, but I want a single knowledge layer across all my sources—not isolated, manually managed workspaces. NotebookLM’s model requires creating a workspace, adding sources, and asking questions about them, but separate notebooks mean separate contexts.
As an experienced engineer who’s never built a Retrieval-Augmented Generation (RAG) system, I saw this as an opportunity to learn and share. I’ll approach it from first principles, and in this series, we’ll:
Two years ago, I read an article about a man with ADHD. The post stayed with me, but for over a year I couldn’t find it again, even after searching bookmarks and Googling "article about a guy with ADHD". I finally found it because the author emailed it to his mailing list. Without that email, I might never have seen it again.
With a personal knowledge base (RAG system), I could have simply asked for "an article about a guy with ADHD" and quickly found it. Let’s dive into how such a system works.
Retrieval-Augmented Generation is the process of supplementing LLM (Large Language Model) prompts with relevant information retrieved from your data. Instead of just relying on general training data, an LLM can answer your questions with reference to your actual information. The typical steps:
The core problem:
Find the right knowledge at any given time from multiple, diverse sources and provide good enough context for AI to answer questions.
To solve this from first principles, what capabilities are required? At a minimum, the system must:
These lead straight to the major system components:
Retrieval: Finds the knowledge base items most relevant to the user's question. This is the heart of RAG. It ranks results by relevance. E.g., if you ask about an ADHD blog post:
Blog post whose title/content strongly matches the ADHD query - very relevant
Personal note discussing ADHD - relevant
A tweet about ADHD - probably relevant
Random tweet - probably irrelevant
For now, ranking can be bundled into retrieval; in larger systems, it might be a separate stage.
flowchart TD
A[Knowledge Sources]
B[Source Connectors]
C[Ingestion]
D[Processing]
E[Knowledge Storage / Indexing]
F[Retrieval]
G[Context Preparation]
H[LLM Generation]
I[Answer with References]
A --> B
B --> C
C --> D
D --> E
E --> F
F --> G
G --> H
H --> I
We’ve broken down the essential building blocks of a RAG system—without writing code yet. The aim was to clarify the problem so that the technical architecture emerges naturally.
If I want an AI system that can answer questions based on my dispersed personal knowledge, it must fetch that knowledge, ingest and process it into a standard form, store it accessibly, retrieve the most relevant parts, prepare them as LLM context, and use all this to generate useful responses.
Before now, I understood concepts like embeddings, vector databases, and indexing as isolated pieces. Taking a first-principles approach reframed it as a system design problem: how do I move knowledge from raw sources to usable LLM context?
Now, things like embeddings and indexing become tools for certain subproblems—not the whole system.
In the next part, we’ll look deeper at how these components work together, what the actual data flow looks like, and where important design decisions come in.