📺 Prefer to watch?
[90-second YouTube Short]· 💬[Telegram] Originally published on software-engineer-blog.com.
The single most expensive misconception in applied AI right now is that fine-tuning teaches a model your documents. It doesn't — and entire GPU budgets get torched on this one mistake. The real split is clean, but almost nobody frames it this way: knowledge versus behaviour.
You have a set of internal documents. You want the model to answer questions about them. Two paths show up:
Most teams pick option 1, expecting the model to "know" the docs. Six weeks later, on GPU credits they can't get back, they realize the model still hallucinates. Then they find out about RAG. Then they argue about which one to use.
The argument ends when you stop conflating knowledge with behaviour.
RAG doesn't change the model at all. It changes the prompt.
Here's the flow:
Because the weights never change:
The cost of RAG is the pipeline itself. You now own:
The weak link is retrieval. Fetch the wrong chunk and the model answers confidently from it. This is operationally messier than it sounds: your monitoring has to watch for semantic drift in retrieval quality, not just model accuracy. A change to your embedding model can silently degrade recall.
Fine-tuning changes the model's weights. You feed it thousands of input-output pairs, and the model adjusts its parameters to predict those outputs given those inputs.
Fine-tuning teaches style. Examples:
These patterns — the structural regularities in your training data — get absorbed into the weights. The model learns to emit outputs that match the shape of your examples.
Fine-tuning does not teach new facts. This is the misconception that matters.
Why? Because facts are not patterns. A fact is a specific piece of information: "Our API rate limit is 1000 requests per minute." When you fine-tune on documents containing that fact, the model doesn't store the fact. It learns correlations: tokens near "rate limit" tend to be followed by numbers in a certain range. Those correlations smear across the weight matrix. The model still hallucinates. It still gets the limit wrong half the time. And when you change the limit to 2000 requests per minute, there is no weight to edit — you have to retrain.
Because weights are opaque, the model can't cite the source. It can't distinguish between what it learned during pre-training and what it learned during fine-tuning. Everything is probability.
| Dimension | RAG | Fine-tuning |
|---|---|---|
| What you're teaching | New knowledge (facts, data) | Consistent behavior (style, format, tone) |
| Does the model's weights change? | No | Yes |
| Can the model cite sources? | Yes (if retrieval includes source metadata) | No |
| How do you update when facts change? | Reindex (days, sometimes hours) | Retrain (weeks, GPU-intensive) |
| Hallucination risk if retrieval fails | High (wrong chunk, confident wrong answer) | High (weights encode fuzzy patterns, not facts) |
| Cost per inference | Higher (chunk tokens in prompt) | Lower (no retrieval overhead) |
| Operational complexity | Retrieval pipeline, embedding drift, chunk quality | Training infrastructure, data labeling, version control |
If you're running a serving system, RAG and fine-tuning hit your latency budget differently. RAG adds latency to time to first token (TTFT). The retrieval call (embedding the query, vector search, maybe re-ranking) happens before you send the prompt to the model. On a 100ms embedding latency + vector search, you're looking at 150–300ms added to TTFT before the model sees a token. Then chunks in the prompt increase the time per output token (TPOT) because the KV cache is larger.
Fine-tuning shifts latency to training time (offline). Inference is faster — shorter prompts, no retrieval. But you pay for retraining whenever behavior needs to change.
If you need both low latency and up-to-date facts, RAG is the only option. If you can tolerate retraining cycles, fine-tuning for behavior + a smaller RAG pipeline (for critical facts only) can reduce TTFT. Reach for RAG when: Your knowledge changes (documents, prices, policies, product specs). You need sources. You want to debug failures. You can afford the retrieval pipeline.
Reach for fine-tuning when: Your model's behaviour must stay consistent (output format, tone, terminology, refusal strategy). You're not adding facts; you're teaching a style.
If you need both: Fine-tune the behaviour first. Then wrap the fine-tuned model in a RAG pipeline that retrieves facts. Fine-tuning should not carry the burden of knowledge management — it will fail at that job, and you'll waste time and GPU budget figuring out why.
Watch the 90-second reel for the quick framing.