In Part 2, we learned why AI sometimes hallucinates. One of the biggest reasons is that an LLM can only answer based on what it learned during training and the information available in its context window. We also introduced grounding-providing the model with reliable information at runtime instead of expecting it to know everything.
But that raises an important question: Where does that information come from? Modern AI applications don't simply dump an entire database or a thousand-page PDF into the prompt. Instead, they first identify the most relevant pieces of information and only send those to the model. In this article, we'll learn how that works.
Let's continue building our AI-powered Travel Planner. So far, it can answer general travel questions using the knowledge it learned during training. Now we want to make it much smarter by up several documents into our application:
Together, these documents contain hundreds of pages. Now the user asks:
I'm staying near Tokyo Station. Which ramen restaurant from our travel guide is within walking distance and is known for vegetarian options?
Somewhere in those hundreds of pages is the answer. The challenge is no longer generating text-it's finding the right information first.
A common misconception is that AI applications simply send all their documents to the model. Imagine our travel guide contains 450 pages, thousands of restaurant listings, hotel descriptions, transportation details, and sightseeing recommendations. Sending all of that to the LLM every time someone asks "Where should I eat tonight?" creates several problems.
First, many documents are simply too large to fit inside the model's context window. Second, even if they did fit, making the model read hundreds of irrelevant pages just to answer a simple question is expensive, slow, and inefficient. Instead, we want to send only the information that matters. The question becomes: How do we find the right few paragraphs from hundreds of pages?
Your first instinct might be: "Why not just search the documents?" That's exactly what we'll explore next.
Before AI became popular, most applications relied on keyword search. The idea is simple: the user searches for a word, and the application finds documents containing that exact word.
For example, given this travel guide text:
Tokyo has several excellent vegetarian ramen restaurants...
Mount Fuji is one of Japan's most iconic landmarks...
The Shinkansen connects Tokyo and Kyoto...
If the user searches for "vegetarian ramen", the search engine finds the first paragraph because it contains both words. While traditional keyword search is simple, fast, and easy to understand, it falls short when matching meaning instead of exact text.
The real limitation is that keyword search compares text, while humans compare meaning. This is why search systems often return dozens of irrelevant results or miss the best result entirely, as you've likely experienced when searching documentation using a word that differs from the author's chosen terminology.
This is where AI changes everything. Instead of asking "Does this document contain the same words?", semantic search asks "Does this document mean the same thing?"
Let's return to our example. The user asks "Where can I eat meat-free ramen near Tokyo Station?" and the travel guide says "Ichiran offers vegetarian ramen a short walk from Tokyo Station." A semantic search understands that meat-free and vegetarian express nearly the same idea, allowing it to retrieve the correct passage even though the exact words are different.
But computers don't naturally understand meaning; they understand numbers. How can a computer determine that vegetarian and meat-free are closely related? That's where embeddings come in.
An embedding is a numerical representation of text that captures its meaning. Think of it as a fingerprint-not of the exact words, but of what those words represent.
For example, these three phrases all have different wording but express similar ideas, so their embeddings end up being very similar:
Conversely, phrases like Mount Fuji, Flight booking, and Currency exchange produce embeddings that are quite different because they represent completely unrelated concepts. The key takeaway is: embeddings represent meaning, not exact wording.
Let's go back to our Travel Planner. Suppose our travel guide contains:
Ichiran offers vegetarian ramen near Tokyo Station.
Now imagine two different users:
A computer comparing plain text sees very few matching words between User B's query and the guide. A traditional keyword search might decide they are unrelated. Embeddings solve this by converting both pieces of text into numerical representations where similar meanings end up close together.
Once text is converted into embeddings, comparing sentences becomes much easier. Instead of asking if they contain the same words, we ask: How similar are their embeddings? If two embeddings are close together, they are likely talking about similar ideas; if they are far apart, they are probably unrelated.
A useful mental model is to imagine every sentence placed on a giant map. Similar ideas appear close together, while unrelated concepts are placed farther apart:
Travel
●
Visit Japan
● ●
Vegetarian Ramen Tokyo Station
●
Meat-free Noodles
--------------------------------------------
●
JavaScript Closures
●
Docker Networking
Notice how the travel-related concepts naturally cluster together, while completely unrelated software concepts appear elsewhere. This is exactly what embeddings help achieve.
Now that every sentence has an embedding, we need a way to compare them mathematically. One of the most common techniques is cosine similarity.
Suppose our Travel Planner has these restaurant descriptions stored:
If the user asks for "Meat-free noodles near Tokyo Station", the embeddings might compare like this:
| Restaurant | Similarity |
|---|---|
| Vegetarian ramen near Tokyo Station | 0.96 |
| Traditional sushi restaurant | 0.41 |
| Italian pizza in Osaka | 0.09 |
The higher the score, the closer the meanings. Notice that the user never mentioned the word vegetarian, yet Restaurant A is still identified as the best match because the meaning is similar.
Mental ModelIn a library, keyword search works like looking up every book containing the word
ramen. Semantic search works by finding books discussing the same idea, even if they use completely different words.
Fortunately, you don't have to calculate these numbers yourself. Just as LLMs specialize in generating text, there are embedding models that specialize in generating embeddings. Instead of producing a paragraph, an embedding model takes text and produces a list of numbers representing its meaning:
User Text ("Vegetarian ramen...") ──► Embedding Model ──► [0.18, -0.42, 0.91, ...]
The resulting list (or vector) usually contains hundreds or thousands of numbers. You never need to interpret them yourself; their only purpose is to make mathematical similarity comparisons possible.
How do we generate embeddings for a 500-page travel guide? We don't feed the entire book into the embedding model at once. Embedding models have strict input limit constraints (typically measured in tokens), and comparing a whole book's meaning is too vague to be useful. If a user asks about a specific ramen restaurant, we want to retrieve the exact paragraph discussing it, not a 500-page book embedding.
So, we break the document down into smaller, logical pieces in a process called chunking. A chunk could be a single paragraph, a section of 100-200 words, or a sliding window of text (e.g., 200 words with a 20-word overlap to ensure context isn't cut off at boundaries).
Mental ModelThink of chunking like cutting a long movie reel into short, thematic scenes. If you want to find the scene with Mount Fuji, it is much easier to search and index short clips than searching the entire three-hour film.
💡 Developer's Takeaway
Choosing the right chunk size is a balancing act. If chunks are too small, they might lose context (e.g., a sentence saying "It is delicious" without mentioning what is delicious). If they're too large, the specific details get averaged out, and search accuracy drops.
Once we have broken our travel guide into thousands of chunks and generated an embedding for each, we need a place to store them. Traditional databases (like PostgreSQL or MySQL) are optimized for finding exact matches like numbers, dates, or strings. They aren't designed to compare thousands of high-dimensional lists of numbers to find which ones are "closest" in meaning.
That's where Vector Databases come in. A vector database is designed specifically to store and index high-dimensional vectors (embeddings) and search through them based on mathematical similarity (like cosine similarity) extremely fast.
Common vector database setups include:
pgvector
for PostgreSQL)💡 Developer's Takeaway
You don't need to write complex search algorithms yourself. A vector database handles the indexing and allows you to query it using an embedding, returning the most semantically similar chunks in milliseconds.
When you configure a vector database, it will ask you to specify the number of dimensions. Dimensions refer to the size of the list of numbers generated by the embedding model. For example:
text-embedding-3-small
) outputs Each dimension represents an abstract feature of meaning that the model learned during training. You don't need to know what each dimension represents; the database only cares that you are comparing vectors of the exact same size.
💡 Developer's Takeaway
An embedding model and a vector database must match: you cannot store a 1536-dimension embedding in a database index configured for 384 dimensions.
Now we have all the pieces to solve our Travel Planner's original challenge: answering questions using our travel guides without sending the entire book to the LLM. We use a pattern called RAG (Retrieval-Augmented Generation).
RAG combines Retrieval (finding relevant chunks from our vector database) and Generation (having the LLM write a response using those chunks). Here is the step-by-step workflow:
Ingestion (Happens once in advance):
Querying (Happens at runtime):
User Question
│
▼
Generate Embedding
│
▼
Query Vector DB ──► Stored Chunks
│
▼
Get Relevant Chunks
│
▼
Build Prompt (Question + Chunks)
│
▼
LLM Generates Text
│
▼
Grounded Response
💡 Developer's Takeaway
RAG is the most popular way to connect proprietary data to LLMs. It ensures the model's response is grounded in facts, reduces hallucinations, and works without needing to retrain the model.
While RAG is powerful, it requires setting up chunking pipelines, embedding models, and vector databases. This can be complex. With modern LLMs having massive context windows (often supporting 1 million to 2 million tokens), another approach has emerged: CAG (Cache-Augmented Generation).
Instead of retrieving matching chunks at runtime, CAG preloads the entire document set (e.g., our 500-page travel guide) directly into the model's context window. It leverages KV (Key-Value) Caching to precompute and cache the mathematical states of the documents once. When a user asks a question, the LLM reads the cached documents instantly, bypassing the retrieval pipeline entirely.
💡 Developer's Takeaway
CAG simplifies the architecture by eliminating vector databases and chunking. However, it is limited by the size of the context window and is best suited for relatively static datasets that fit comfortably within the model's limit. RAG remains essential for massive or constantly changing data.
In this article, we explored how to give AI knowledge beyond its training. We covered:
In ** Part 4**, we will answer the next logical question: