Your RAG bot just gave a confident, detailed answer. And it's completely wrong.
Here's the part that'll annoy you: the model did nothing wrong. It answered perfectly — using the text you handed it. The bug isn't in the AI. It's in the five steps before the AI.
Prefer to watch? Full 6-minute walkthrough with the "lost in the middle" animation:
RAG is simple on paper. You search your own documents, grab the best matches, paste them into the prompt, and the model answers from those. So when the answer's garbage, everyone blames the model.
Wrong suspect. There are five places this pipeline breaks — and four of them happen before the model even runs:
const chunks = split(docs) // 1 · chunking
const vectors = embed(chunks) // 2 · embedding
const hits = search(embed(query), vectors) // 3 · retrieval
const best = rerank(query, hits) // 4 · ranking
const answer = llm(prompt(best, query)) // 5 · generation
Five lines. Every bug I'm about to show you lives on one of them.
Your documents are long, so you cut them into pieces before you store them. Most people cut every few hundred characters and move on. That's where it starts to rot.
A fixed-length cut doesn't care about meaning. It'll slice a sentence in half. It'll split a question from its answer. Now the one chunk that held your answer is two halves — and each half looks irrelevant on its own. So retrieval never even finds it.
Fix: cut on structure, not length. Split on paragraphs and headers — where the meaning actually breaks. And let chunks overlap a little, so nothing gets stranded at the edge.
Every chunk goes through a model that turns text into a vector: a long list of numbers that captures its meaning. Similar meaning, similar numbers. That's the whole trick that makes search work.
But here's the trap. Your question gets embedded by the same model — and a question rarely looks like its answer.
You ask "how do I reset my password?" The doc says "account recovery procedure." Same thing to a human. Different words — and a weak embedder puts them far apart.
Fix: pick an embedding model built for retrieval, and test it on your data. A model that's great on legal text can be useless on code.
You take the question's vector and grab the closest chunks. This is the "vector search" everyone talks about. On its own, it's got a blind spot.
Vector search matches meaning, not exact words. Which is great — until someone searches for an error code. A product name. SKU-4417
. Those barely have a meaning to embed. They're just exact strings, and nothing is "close in meaning" to a serial number.
Fix: hybrid search. Run keyword search and vector search, then merge the results. Old-school keyword matching catches the exact strings embeddings miss.
Now you've got a pile of candidates — say the top twenty. But you can't paste twenty chunks into the prompt, so you keep the top five and drop the rest.
Here's the question nobody asks: what if the best chunk was ranked number eight? Then you just threw it away. Your answer was in the pile, and you cut it.
Fix: a re-ranker — a second, smarter model that reads the question and each chunk together and re-scores them. Cheap vector search casts a wide net; the re-ranker picks the real winners out of it. Now your top five are actually the top five.
Even the order you paste chunks in changes the answer. Models pay the most attention to the start and the end of the context. Put your most important chunk in the middle, and the model can skim right past it.
This isn't a hunch — it's measured. The same model gets more accurate just by moving the right chunk to the edges of the context (Liu et al., * Lost in the Middle*, 2023). So put your best chunk first, or last. Never buried in the middle.
Right chunks, right order. This is your last chance to ruin it, and people do it two ways:
Fix is basically one line in your prompt:
"Answer only from the context below. If it's not there, say you don't know."
That single instruction turns a confident liar into a system you can trust. A RAG bot that admits "I don't know" beats ten that guess.
Five failure points — and notice the model was the last one. Usually the innocent one.
So next time your RAG bot lies to you, don't touch the model first. Walk the five steps backward — prompt, ranking, retrieval, embedding, chunks. The bug is almost always upstream.
What's the worst answer your RAG bot has ever given you? Drop it in the comments — I actually want to read those.
I make Vlad's Stack https://www.youtube.com/channel/UCUO8Uo5LsEy1b9eRkrH6JNg — how the AI tools you use every day actually work, for people who write code. Full video walkthrough of this one is above.