You've got a local model running and a pile of your own material, notes, a codebase, a folder of PDFs, that you want it to actually know. You have two options. Stuff everything into the prompt and rely on a giant context window, or use RAG: retrieve only the relevant bits and hand the model just those. The first option is simple but, as our KV-cache guide showed, brutally expensive in memory. The second is how most people give a local model big knowledge on modest hardware.
This is the plain-English guide to retrieval-augmented generation through the local-hardware lens: what RAG actually is, the cheap second model it adds, why it often beats brute-force long context, and the honest reasons it's finicky.
What RAG actually is #
A language model has two kinds of memory. There's what's baked into its weights during training, its parametric memory, and there's whatever you put in the prompt right now. RAG adds a third thing: an external, searchable non-parametric memory, your documents, turned into a searchable index that the model consults on demand.
The idea comes from Lewis et al.'s "Retrieval-Augmented Generation" (2020): pair a generation model with a retriever over an external vector index, so the model can pull in facts it was never trained on, and you can update its knowledge by updating the index, not retraining the model. For local users that last part is the whole appeal: you keep a small, fast model and bolt your ever-changing knowledge onto the side.
The pipeline, in four steps #
Every RAG setup, however fancy, is the same four moves:
Chunk. Split your documents into bite-sized passages (a few hundred tokens each).Embed. Run each chunk through anembedding modelthat turns text into a vector, a list of numbers where similar meanings land near each other. This is the trick fromSentence-BERT(Reimers & Gurevych, 2019): text becomes geometry, so "similarity" becomes "distance."Store. Put those vectors in avector database(FAISS, Chroma, Qdrant, LanceDB) that can find nearest neighbors fast.** Retrieve & generate.**At query time, embed the question, grab the top-kmost similar chunks, paste them into the prompt, and let the model answer from them.
Why dense vectors instead of keyword search? Because meaning beats spelling. Karpukhin et al.'s Dense Passage Retrieval (2020) showed that learned embeddings beat a strong keyword system (BM25) by 9–19% on retrieval accuracy, the embedding finds the passage that means the same thing, even when it shares no words with your query.
The local-hardware angle: RAG is a memory-saving trick #
Here's the part most generic RAG tutorials miss, and the reason it matters on your hardware. RAG's real superpower for local LLMs isn't just accuracy, it's that it keeps your context small.
Instead of feeding the model a 100,000-token document and paying the full KV-cache and prompt-processing bill (the expensive phases from our prefill-vs-decode guide), you retrieve maybe five 500-token chunks, a ~2,500-token context, and answer from those. You traded an enormous, slow, VRAM-hungry context for a tiny, fast one. On a memory-constrained box, that's often the difference between "works" and "out of memory."
And the extra hardware cost is surprisingly small:
The embedding model is tiny. Popular local embedders, bge-small (~33M params), nomic-embed (~137M), e5-large (~335M), are a fraction of your main model's size and run happily on CPU or under a gigabyte of VRAM.The vector database is cheap. It mostly lives in RAM/on disk and leans on the CPU; nearest-neighbor search over even millions of chunks is millisecond-fast.
So you add a small model and a lightweight index, and in exchange you stop paying for giant context. For local setups, that's a great trade.
"Why not just use a long context window?" #
It's a fair question, modern models advertise 128k+ context, so why bother retrieving? Two reasons, one about quality and one about cost.
Quality: models are bad at using long contexts. Liu et al.'s aptly-named "Lost in the Middle" (2023) found a U-shaped curve: models reliably use information at the start and end of a long context but routinely miss what's buried in the middle, and overall accuracy keeps degrading as the context grows, even for models explicitly built for long context. Dumping everything in doesn't mean the model reads everything. Retrieving the few most-relevant chunks and placing them front-and-center often beats a giant haystack.
Cost: the giant haystack is also the expensive one, every one of those 100k tokens inflates your KV cache and your prompt-processing time. RAG sidesteps both. For a large or constantly-changing corpus, retrieval usually wins on both axes; for a single short document you'll read in full, long context is simpler and fine.
The honest part: RAG is finicky #
RAG is not magic, and anyone who's built one will tell you so. The quality lives or dies on details the tutorials gloss over: how you chunk (too small and you lose context, too big and retrieval gets noisy), which embedding model you pick, how many chunks you retrieve, and whether your documents even contain clean answers. Naive setups disappoint, as one r/LocalLLaMA builder put it bluntly while hunting for a local "memory" system:
"Tried summarization but it loses too much detail. Tried vector DB with embeddings but [it] didn't work well…", u/Independent_Plum_489, on the frustration of naive local retrieval
That's the realistic baseline: a first-pass RAG often underwhelms, and getting it good is iterative work (better chunking, a stronger embedder, re-ranking, hybrid keyword+dense search). Treat RAG as a technique you tune, not a switch you flip.
Making it actually good: re-ranking and hybrid search #
If a first-pass RAG disappoints, the fixes are well-trodden. Hybrid search combines dense embeddings with old-fashioned keyword matching (BM25), catching both meaning-based and exact-term hits, invaluable when your corpus is full of specific names, error codes, or part numbers that embeddings blur together. Re-ranking adds a second, more careful pass: a fast embedder casts a wide net (say, the top 50 chunks), then a slower cross-encoder re-scores them so the truly relevant passages land first, which matters because, per "lost in the middle," position is everything. Query rewriting has the model rephrase or expand your question before retrieval so it better matches how the documents are actually written. None of these are exotic; they're the standard escalation path from "RAG kinda works" to "RAG works," and each is just a small local model or a cheap CPU step bolted onto the pipeline you already have.
Choosing an embedding model #
The embedder is the most consequential small decision in a local RAG stack, it defines what "similar" even means. A few practical pointers. Match the domain: a general-purpose embedder (bge, e5, nomic) handles prose well, but code or highly technical text often benefits from a code- or domain-tuned model. Mind the dimensions: larger embedding vectors (1024+) capture more nuance but cost more storage and slightly slower search than compact ones (384–768), for most local corpora the smaller models are plenty. Check the leaderboard: the public MTEB benchmark ranks embedders on real retrieval tasks, and the open, locally-runnable models near the top are genuinely strong. And crucially, keep your embedder fixed: change embedding models and you must re-embed the entire corpus, because vectors from different models aren't comparable. Pick one that fits your domain and your box, and commit.
The decision cheat-sheet #
| Your situation | Reach for… |
|---|---|
| One short doc you'll use in full | Long context (simpler) |
| Big or growing corpus (notes, codebase, wiki) | RAG |
| Knowledge that changes often | RAG (update the index, not the model) |
| Memory-constrained box | RAG (keeps context small) |
| Need exact recall of a specific passage | RAG + hybrid (dense + keyword) |
RAG or fine-tuning? #
The other way to "teach" a local model your material is fine-tuning, actually adjusting its weights. They solve different problems, and the rule of thumb is clean: RAG adds knowledge; fine-tuning adds behavior. If you need the model to know facts, your docs, your product details, your codebase, reach for RAG: the knowledge stays in an index you can update in seconds, with full provenance (you can see exactly which chunk an answer came from). If you need the model to act differently, adopt a house tone, always output a specific format, speak niche jargon, or master a task pattern, that's fine-tuning's job, because you're reshaping how it responds, not what it can look up. For most local "chat with my stuff" use cases, RAG is the right and far cheaper tool: no training run, no GPU-hours, no risk of the model forgetting its general skills, and knowledge updates are a file drop. Fine-tuning is the specialist you call when behavior, not facts, is the gap, and the two compose well (a fine-tuned voice answering over a RAG-retrieved context).
What this means for your hardware #
RAG quietly changes the buying math. Because it shrinks the context the model actually processes, it lets a smaller, cheaper box punch above its weight: an 8B-class model on a modest machine, armed with a good retrieval index, can answer questions about gigabytes of your documents, something no amount of raw context would let that same box do. It's the local-AI equivalent of giving a sharp generalist a well-organized filing cabinet instead of trying to make them memorize the whole library.
So if your use case is "chat with my documents," don't over-buy hardware to fit a giant context. Buy enough to run a solid model comfortably, add a small embedder and a vector DB, and let retrieval do the heavy lifting. The knowledge lives in the index; the model just needs to be good at reasoning over the handful of chunks you hand it.
Sources & how we researched this #
This explainer synthesizes the primary retrieval literature, Lewis et al., "Retrieval-Augmented Generation" (2020); Karpukhin et al., Dense Passage Retrieval (2020); Reimers & Gurevych, Sentence-BERT (2019); and Liu et al., "Lost in the Middle" (2023), for the mechanisms and the retrieval-vs-long-context findings, which come from those papers. The practical "it's finicky" reality is an owner report from r/LocalLLaMA, linked so you can verify; we have not benchmarked these setups first-hand. Embedding-model parameter counts are approximate and rounded.
Related guides #
The KV cache, explained(why long context is expensive)Prompt processing vs generation(the cost RAG helps you dodge)Mixture-of-Experts, explained