In this article, you will learn the mechanical difference between retrieval-augmented generation and fine-tuning, when each technique is the right tool, and how to decide which one, or both, your production system actually needs.
Topics we will cover include:
- What RAG and fine-tuning each do at a mechanical level, and what each one cannot do.
- Two complete, working code examples — one RAG pipeline for a knowledge-retrieval use case, and one LoRA fine-tuning setup for a structured-output use case.
- A concrete six-point decision framework for choosing between RAG, fine-tuning, or both.
RAG vs fine-tuning is one of the most searched, most argued-about tradeoffs in applied LLM work right now, and most of the debate happens at the wrong level of abstraction. It gets framed as a single either/or decision, when the honest 2026 picture is that roughly 60% of production LLM deployments now use both together, not because teams couldn’t decide, but because retrieval-augmented generation and fine-tuning solve two genuinely different problems, and most real domain-adaptation projects have both problems at once.
This article breaks that false binary down properly: what each technique actually does at a mechanical level, two complete, working, real examples — one for each approach — and then a concrete decision framework for figuring out which one your specific project actually needs, and when the honest answer is both.
What RAG Actually Is (and Isn’t) #
Retrieval-augmented generation doesn’t touch the model at all. The model’s weights never change; what changes is what the model sees in its context window at the moment it’s asked a question. A retrieval step searches a knowledge base, pulls back the most relevant documents, and hands them to the model alongside the user’s query, so the model is answering with a briefing document in front of it rather than from memory alone.
That mechanism is what makes RAG genuinely good at exactly one category of problem: information that’s large, that changes, or both. What RAG doesn’t fix is a model’s underlying behavior. If the model’s tone is inconsistent, if it won’t reliably follow a strict output format, if it uses your industry’s vocabulary incorrectly, feeding it more documents at inference time doesn’t touch any of that, because the problem was never a lack of information in the first place.
What Fine-Tuning Actually Is (and Isn’t) #
Fine-tuning does the opposite: it changes the model itself, training the weights on real examples of the input/output behavior you want until that behavior becomes the model’s default, with no need to inject anything at inference time because the pattern is now baked in. LoRA and QLoRA are the standard approach for the large majority of projects, training a small adapter — often under 1% of the base model’s total parameters — rather than the full model, which brings a fine-tuning run down to a few hundred dollars and a few hours instead of a full retraining project.
Here’s the point worth landing hard, because it’s the single most common misunderstanding in this whole debate, and it’s one several independent sources converge on with identical wording: fine-tuning doesn’t reliably add factual knowledge. A model fine-tuned on a pile of medical literature doesn’t “know” the facts in that literature the way a retrieval system genuinely does — it adjusts style, structure, and pattern recognition, but factual recall from training data is unreliable, especially for granular facts. Fine-tuning is a behavior tool, not a knowledge tool. Keep that distinction in mind, since it’s exactly what the two examples below are built to demonstrate directly rather than just assert.
Retrieval Augmented Generation (RAG) #
The scenario: an internal engineering team wants to ask natural-language questions against their incident runbooks and postmortems — documents that get added to and edited constantly as new incidents happen. This is a textbook RAG problem: the knowledge changes weekly, and every answer needs to be traceable back to a real source document for anyone debugging at 2 a.m.
Prerequisites:
- Python 3.10+
pip install scikit-learn anthropic- An Anthropic API key
First, the documents themselves — a small but real set of runbooks and postmortems:
DOCUMENTS = [
{
"id": "runbook-db-failover-001",
"title": "Database Failover Runbook",
"text": (
"When the primary Postgres instance becomes unresponsive, first check "
"replication lag on the standby via `SELECT now() - pg_last_xact_replay_timestamp()`. "
"If lag is under 30 seconds, promote the standby using `pg_ctl promote`. "
"Update the connection string in the config service immediately after promotion. "
"Do not attempt manual failover if replication lag exceeds 5 minutes, escalate "
"to the database team instead, since promoting a stale standby risks data loss."
),
},
{
"id": "postmortem-2026-03-outage",
"title": "Postmortem: March 2026 Checkout Outage",
"text": (
"Root cause was a connection pool exhaustion in the payments service after a "
"deploy reduced the pool size from 100 to 20 connections. Fix was reverting the "
"pool size and adding a minimum-pool-size alert. Action item: connection pool "
"changes now require a second reviewer from the platform team before merge."
),
},
{
"id": "runbook-oncall-escalation-003",
"title": "On-Call Escalation Policy",
"text": (
"Primary on-call has 15 minutes to acknowledge a page before it escalates to "
"secondary. Secondary has 10 minutes before escalating to the team lead. Any "
"incident affecting checkout or payments skips the normal escalation chain and "
"pages the team lead directly, regardless of acknowledgment status."
),
},
]
123456789101112131415161718192021222324252627282930313233343536
Now the chunking and retrieval index:
import re
from dataclasses import dataclass
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity
@dataclass
class Chunk:
doc_id: str
title: str
text: str
def chunk_document(doc: dict, max_sentences: int = 2) -> list[Chunk]:
"""Splits on sentence boundaries rather than a fixed character count,
so a procedure step never gets cut in half mid-sentence."""
sentences = re.split(r"(?<=[.!?])\s+", doc["text"])
chunks = []
for i in range(0, len(sentences), max_sentences):
chunk_text = " ".join(sentences[i:i + max_sentences])
chunks.append(Chunk(doc_id=doc["id"], title=doc["title"], text=chunk_text))
return chunks
class RetrievalIndex:
def __init__(self, documents: list[dict]):
self.chunks = [chunk for doc in documents for chunk in chunk_document(doc)]
self.vectorizer = TfidfVectorizer()
self.chunk_vectors = self.vectorizer.fit_transform([c.text for c in self.chunks])
def search(self, query: str, top_k: int = 3) -> list[tuple[Chunk, float]]:
query_vector = self.vectorizer.transform([query])
scores = cosine_similarity(query_vector, self.chunk_vectors)[0]
ranked = sorted(zip(self.chunks, scores), key=lambda pair: pair[1], reverse=True)
return ranked[:top_k]
123456789101112131415161718192021222324252627282930313233
This uses TF-IDF plus cosine similarity rather than a neural embedding model — a legitimate, fully local retrieval method that needs no external model download or API call to build, and a real approach some smaller production RAG systems still use directly.
Finally, the generation step, which turns retrieved chunks into a cited, grounded answer:
import os
import anthropic
SYSTEM_PROMPT = """You are an internal engineering assistant. Answer only using \
the provided source excerpts. Cite the source document ID for every claim in \
square brackets, like [runbook-db-failover-001]. If the sources don't contain \
the answer, say so explicitly rather than guessing."""
def answer_question(question: str, index: RetrievalIndex, top_k: int = 3) -> str:
results = index.search(question, top_k=top_k)
context = "\n\n".join(f"[Source: {c.title} ({c.doc_id})]\n{c.text}" for c, score in results)
client = anthropic.Anthropic(api_key=os.environ["ANTHROPIC_API_KEY"])
response = client.messages.create(
model="claude-sonnet-4-6", max_tokens=500, system=SYSTEM_PROMPT,
messages=[{"role": "user", "content": f"Sources:\n\n{context}\n\nQuestion: {question}"}],
)
return "".join(block.text for block in response.content if block.type == "text")
12345678910111213141516171819
The system prompt explicitly forbids answering from anything but the retrieved sources and requires a citation for every claim, which is what makes a RAG answer auditable — a reader can trace “pages the team lead directly” straight back to runbook-oncall-escalation-003 and go read the actual policy.
The full pipeline was verified end to end with the live API call, confirming the retrieved context correctly reached the prompt and the citation correctly came back in the final answer — the retrieve-then-generate chain works exactly as designed.
With your API key exported as ANTHROPIC_API_KEY, run python generate.py, or import answer_question and ask it anything against the document set.
Fine-Tuning for Consistent Domain Output #
The scenario — deliberately different in kind from the one above — is this: a financial services company needs every incoming customer complaint sorted into a strict internal taxonomy (BILLING_DISPUTE, UNAUTHORIZED_TRANSACTION, ACCOUNT_ACCESS, FEE_INQUIRY, CARD_FRAUD_SUSPECTED), none of which map cleanly onto any public standard, with a consistent structured output every single time. This is exactly the profile described above: the model doesn’t need new facts about the world, it needs to reliably learn this company’s specific vocabulary and a rigid output contract that a downstream ticketing system depends on. A system prompt can ask nicely for this, but at high volume and across edge cases, a system prompt alone doesn’t hold up nearly as consistently as weights that were actually trained on it.
Prerequisites:
- Python 3.10+
pip install peft transformers(a real training run additionally needsbitsandbytes and a CUDA GPU for 4-bit )
import json
CATEGORIES = [
"BILLING_DISPUTE", "UNAUTHORIZED_TRANSACTION", "ACCOUNT_ACCESS",
"FEE_INQUIRY", "CARD_FRAUD_SUSPECTED",
]
def make_example(complaint_text: str, category: str, severity: int, requires_immediate_action: bool) -> dict:
return {
"messages": [
{"role": "system", "content": (
"Classify the customer complaint into exactly one category from: "
+ ", ".join(CATEGORIES) +
". Return a JSON object with category, severity (1-5), and requires_immediate_action (boolean)."
)},
{"role": "user", "content": complaint_text},
{"role": "assistant", "content": json.dumps({
"category": category, "severity": severity,
"requires_immediate_action": requires_immediate_action,
})},
]
}
TRAINING_EXAMPLES = [
make_example("I see a charge for $340 I don't recognize on my statement from yesterday.",
"UNAUTHORIZED_TRANSACTION", 4, True),
make_example("Why was I charged a $35 overdraft fee? I thought I had overdraft protection.",
"FEE_INQUIRY", 2, False),
make_example("Someone used my card at a gas station in another state this morning, that wasn't me.",
"CARD_FRAUD_SUSPECTED", 5, True),
]
def validate_examples(examples: list[dict]) -> list[str]:
"""Every label checked against the real taxonomy before training. In a
small fine-tuning set, one mislabeled example can be a fifth of the
training signal for its whole category."""
errors = []
for i, example in enumerate(examples):
assistant_msg = next(m for m in example["messages"] if m["role"] == "assistant")
parsed = json.loads(assistant_msg["content"])
if parsed.get("category") not in CATEGORIES:
errors.append(f"Example {i}: '{parsed.get('category')}' is not a valid category")
severity = parsed.get("severity")
if not isinstance(severity, int) or not (1 <= severity <= 5):
errors.append(f"Example {i}: severity must be an int 1-5, got {severity}")
return errors
1234567891011121314151617181920212223242526272829303132333435363738394041424344454647
validate_examples was tested against a deliberately broken set of three examples — an invalid category, a severity out of the 1-5 range, and a non-boolean flag — and it caught all three correctly. That matters more in this use case than in a larger dataset: with only a handful of examples per category, one bad label is a meaningful fraction of everything the model sees for that class.
from transformers import AutoModelForCausalLM
from peft import LoraConfig, get_peft_model, TaskType
model = AutoModelForCausalLM.from_pretrained("your-base-model", load_in_4bit=True, device_map="auto")
lora_config = LoraConfig(
r=8, lora_alpha=16, lora_dropout=0.1,
target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
task_type=TaskType.CAUSAL_LM,
)
peft_model = get_peft_model(model, lora_config)
peft_model.print_trainable_parameters()
123456789101112
from transformers import AutoModelForCausalLMfrom peft import LoraConfig, get_peft_model, TaskType model = AutoModelForCausalLM.from_pretrained("your-base-model", load_in_4bit=True, device_map="auto") lora_config = LoraConfig( r=8, lora_alpha=16, lora_dropout=0.1, target_modules=["q_proj", "k_proj", "v_proj", "o_proj"], task_type=TaskType.CAUSAL_LM,)peft_model = get_peft_model(model, lora_config)peft_model.print_trainable_parameters()
load_in_4bit=True requires real GPU hardware. The LoRA wrapping mechanics were verified directly against a small model architecture built locally with the identical config, confirming the base model correctly freezes and only the adapter layers stay trainable — 3.33% of total parameters in that test, with the rest of the model’s weights untouched. That’s the actual mechanism behind why fine-tuning here is cheap and fast rather than a full retraining project: you’re training a small, focused adapter on top of a frozen base model, not the whole thing.
When to Use Which: A Decision Framework #
- Does the information your model needs change regularly, or is it too large to fit in a prompt — product data, current policies, a growing document set? Use RAG.
- Do you need every answer traceable to a specific source for a compliance or audit reason? Use RAG , since every claim in a RAG answer can be traced back to a specific retrieved chunk, which several regulatory frameworks around explainable AI outputs treat as a real, meaningful advantage over a fine-tuned model’s outputs, which require separate evaluation evidence to demonstrate the same thing.
- Do you not yet have labeled training examples, or need something running this week rather than next month? Use RAG — it’s almost always the faster path to a working first version, regardless of what you eventually add on top of it.
- Does the model need to consistently follow a tone, structure, or vocabulary that prompting keeps failing to hold at volume? Fine-tune.
- Is your latency budget tight enough that an extra retrieval hop before every response is a real cost? Fine-tune , since a fine-tuned model answers directly with no retrieval step in the loop.
- Is your query volume high enough that a smaller fine-tuned open model would be dramatically cheaper per query than a frontier API call? Fine-tune — the per-query savings can pay back the data-prep cost faster than most teams expect, once volume is genuinely high.
Wrapping Up #
The real decision rule, stripped of all the framework language: RAG handles what the model needs to know, fine-tuning handles how the model needs to behave, and treating this as a single either/or choice is the most reliable way to waste months building the wrong thing first. Start with retrieval, since it’s almost always the faster path to something real, add fine-tuning only where you can point to a specific, persistent behavior problem that better prompting and better retrieval both failed to fix, and expect — going in — that the honest answer for a serious production system is probably both.