RAG vs. Fine-Tuning for Domain Adaptation: When to Use Which

Roughly 60% of production LLM deployments now use retrieval-augmented generation (RAG) and fine-tuning together, according to a Scalacode analysis, because the two techniques solve different problems rather than competing as an either/or choice. RAG leaves model weights unchanged and injects retrieved documents into the context window, making it suited to large or frequently changing information, while fine-tuning — typically via LoRA or QLoRA adapters training under 1% of a base model's parameters — changes the model's default behavior at a cost of a few hundred dollars and a few hours. The analysis stresses that fine-tuning does not reliably add factual knowledge, making it a behavior tool rather than a knowledge tool.

In this article, you will learn the mechanical difference between retrieval-augmented generation and fine-tuning, when each technique is the right tool, and how to decide which one, or both, your production system actually needs. Topics we will cover include: - What RAG and fine-tuning each do at a mechanical level, and what each one cannot do. - Two complete, working code examples — one RAG pipeline for a knowledge-retrieval use case, and one LoRA fine-tuning setup for a structured-output use case. - A concrete six-point decision framework for choosing between RAG, fine-tuning, or both. RAG vs fine-tuning is one of the most searched, most argued-about tradeoffs in applied LLM work right now, and most of the debate happens at the wrong level of abstraction. It gets framed as a single either/or decision, when the honest 2026 picture is that roughly 60% of production LLM deployments now use both together https://www.scalacode.com/blog/rag-vs-fine-tuning/ , not because teams couldn’t decide, but because retrieval-augmented generation and fine-tuning solve two genuinely different problems, and most real domain-adaptation projects have both problems at once. This article breaks that false binary down properly: what each technique actually does at a mechanical level, two complete, working, real examples — one for each approach — and then a concrete decision framework for figuring out which one your specific project actually needs, and when the honest answer is both. What RAG Actually Is and Isn’t Retrieval-augmented generation doesn’t touch the model at all. The model’s weights never change; what changes is what the model sees in its context window at the moment it’s asked a question. A retrieval step searches a knowledge base, pulls back the most relevant documents, and hands them to the model alongside the user’s query, so the model is answering with a briefing document in front of it rather than from memory alone. That mechanism is what makes RAG genuinely good at exactly one category of problem: information that’s large, that changes, or both. What RAG doesn’t fix is a model’s underlying behavior. If the model’s tone is inconsistent, if it won’t reliably follow a strict output format, if it uses your industry’s vocabulary incorrectly, feeding it more documents at inference time doesn’t touch any of that, because the problem was never a lack of information in the first place. What Fine-Tuning Actually Is and Isn’t Fine-tuning does the opposite: it changes the model itself, training the weights on real examples of the input/output behavior you want until that behavior becomes the model’s default, with no need to inject anything at inference time because the pattern is now baked in. LoRA and QLoRA are the standard approach for the large majority of projects, training a small adapter — often under 1% of the base model’s total parameters — rather than the full model, which brings a fine-tuning run down to a few hundred dollars and a few hours instead of a full retraining project. Here’s the point worth landing hard, because it’s the single most common misunderstanding in this whole debate, and it’s one several independent sources converge on with identical wording: fine-tuning doesn’t reliably add factual knowledge. A model fine-tuned on a pile of medical literature doesn’t “ know ” the facts in that literature the way a retrieval system genuinely does — it adjusts style, structure, and pattern recognition, but factual recall from training data is unreliable, especially for granular facts. Fine-tuning is a behavior tool, not a knowledge tool. Keep that distinction in mind, since it’s exactly what the two examples below are built to demonstrate directly rather than just assert. Retrieval Augmented Generation RAG The scenario: an internal engineering team wants to ask natural-language questions against their incident runbooks and postmortems — documents that get added to and edited constantly as new incidents happen. This is a textbook RAG problem: the knowledge changes weekly, and every answer needs to be traceable back to a real source document for anyone debugging at 2 a.m. Prerequisites: - Python 3.10+ - pip install scikit-learn anthropic - An Anthropic API key First, the documents themselves — a small but real set of runbooks and postmortems: documents.py DOCUMENTS = { "id": "runbook-db-failover-001", "title": "Database Failover Runbook", "text": "When the primary Postgres instance becomes unresponsive, first check " "replication lag on the standby via SELECT now - pg last xact replay timestamp . " "If lag is under 30 seconds, promote the standby using pg ctl promote . " "Update the connection string in the config service immediately after promotion. " "Do not attempt manual failover if replication lag exceeds 5 minutes, escalate " "to the database team instead, since promoting a stale standby risks data loss." , }, { "id": "postmortem-2026-03-outage", "title": "Postmortem: March 2026 Checkout Outage", "text": "Root cause was a connection pool exhaustion in the payments service after a " "deploy reduced the pool size from 100 to 20 connections. Fix was reverting the " "pool size and adding a minimum-pool-size alert. Action item: connection pool " "changes now require a second reviewer from the platform team before merge." , }, { "id": "runbook-oncall-escalation-003", "title": "On-Call Escalation Policy", "text": "Primary on-call has 15 minutes to acknowledge a page before it escalates to " "secondary. Secondary has 10 minutes before escalating to the team lead. Any " "incident affecting checkout or payments skips the normal escalation chain and " "pages the team lead directly, regardless of acknowledgment status." , }, additional documents omitted here for length, full set in the shared code files 123456789101112131415161718192021222324252627282930313233343536 documents.pyDOCUMENTS = { "id": "runbook-db-failover-001", "title": "Database Failover Runbook", "text": "When the primary Postgres instance becomes unresponsive, first check " "replication lag on the standby via SELECT now - pg last xact replay timestamp . " "If lag is under 30 seconds, promote the standby using pg ctl promote . " "Update the connection string in the config service immediately after promotion. " "Do not attempt manual failover if replication lag exceeds 5 minutes, escalate " "to the database team instead, since promoting a stale standby risks data loss." , }, { "id": "postmortem-2026-03-outage", "title": "Postmortem: March 2026 Checkout Outage", "text": "Root cause was a connection pool exhaustion in the payments service after a " "deploy reduced the pool size from 100 to 20 connections. Fix was reverting the " "pool size and adding a minimum-pool-size alert. Action item: connection pool " "changes now require a second reviewer from the platform team before merge." , }, { "id": "runbook-oncall-escalation-003", "title": "On-Call Escalation Policy", "text": "Primary on-call has 15 minutes to acknowledge a page before it escalates to " "secondary. Secondary has 10 minutes before escalating to the team lead. Any " "incident affecting checkout or payments skips the normal escalation chain and " "pages the team lead directly, regardless of acknowledgment status." , }, additional documents omitted here for length, full set in the shared code files Now the chunking and retrieval index: python retrieval.py import re from dataclasses import dataclass from sklearn.feature extraction.text import TfidfVectorizer from sklearn.metrics.pairwise import cosine similarity @dataclass class Chunk: doc id: str title: str text: str def chunk document doc: dict, max sentences: int = 2 - list Chunk : """Splits on sentence boundaries rather than a fixed character count, so a procedure step never gets cut in half mid-sentence.""" sentences = re.split r" ?<= . ? \s+", doc "text" chunks = for i in range 0, len sentences , max sentences : chunk text = " ".join sentences i:i + max sentences chunks.append Chunk doc id=doc "id" , title=doc "title" , text=chunk text return chunks class RetrievalIndex: def init self, documents: list dict : self.chunks = chunk for doc in documents for chunk in chunk document doc self.vectorizer = TfidfVectorizer self.chunk vectors = self.vectorizer.fit transform c.text for c in self.chunks def search self, query: str, top k: int = 3 - list tuple Chunk, float : query vector = self.vectorizer.transform query scores = cosine similarity query vector, self.chunk vectors 0 ranked = sorted zip self.chunks, scores , key=lambda pair: pair 1 , reverse=True return ranked :top k 123456789101112131415161718192021222324252627282930313233 retrieval.pyimport refrom dataclasses import dataclassfrom sklearn.feature extraction.text import TfidfVectorizerfrom sklearn.metrics.pairwise import cosine similarity @dataclassclass Chunk: doc id: str title: str text: str def chunk document doc: dict, max sentences: int = 2 - list Chunk : """Splits on sentence boundaries rather than a fixed character count, so a procedure step never gets cut in half mid-sentence.""" sentences = re.split r" ?<= . ? \s+", doc "text" chunks = for i in range 0, len sentences , max sentences : chunk text = " ".join sentences i:i + max sentences chunks.append Chunk doc id=doc "id" , title=doc "title" , text=chunk text return chunks class RetrievalIndex: def init self, documents: list dict : self.chunks = chunk for doc in documents for chunk in chunk document doc self.vectorizer = TfidfVectorizer self.chunk vectors = self.vectorizer.fit transform c.text for c in self.chunks def search self, query: str, top k: int = 3 - list tuple Chunk, float : query vector = self.vectorizer.transform query scores = cosine similarity query vector, self.chunk vectors 0 ranked = sorted zip self.chunks, scores , key=lambda pair: pair 1 , reverse=True return ranked :top k This uses TF-IDF https://en.wikipedia.org/wiki/Tf%E2%80%93idf plus cosine similarity https://en.wikipedia.org/wiki/Cosine similarity rather than a neural embedding model — a legitimate, fully local retrieval method that needs no external model download or API call to build, and a real approach some smaller production RAG systems still use directly. Finally, the generation step, which turns retrieved chunks into a cited, grounded answer: python generate.py import os import anthropic SYSTEM PROMPT = """You are an internal engineering assistant. Answer only using \ the provided source excerpts. Cite the source document ID for every claim in \ square brackets, like runbook-db-failover-001 . If the sources don't contain \ the answer, say so explicitly rather than guessing.""" def answer question question: str, index: RetrievalIndex, top k: int = 3 - str: results = index.search question, top k=top k context = "\n\n".join f" Source: {c.title} {c.doc id} \n{c.text}" for c, score in results client = anthropic.Anthropic api key=os.environ "ANTHROPIC API KEY" response = client.messages.create model="claude-sonnet-4-6", max tokens=500, system=SYSTEM PROMPT, messages= {"role": "user", "content": f"Sources:\n\n{context}\n\nQuestion: {question}"} , return "".join block.text for block in response.content if block.type == "text" 12345678910111213141516171819 generate.pyimport osimport anthropic SYSTEM PROMPT = """You are an internal engineering assistant. Answer only using \the provided source excerpts. Cite the source document ID for every claim in \square brackets, like runbook-db-failover-001 . If the sources don't contain \the answer, say so explicitly rather than guessing.""" def answer question question: str, index: RetrievalIndex, top k: int = 3 - str: results = index.search question, top k=top k context = "\n\n".join f" Source: {c.title} {c.doc id} \n{c.text}" for c, score in results client = anthropic.Anthropic api key=os.environ "ANTHROPIC API KEY" response = client.messages.create model="claude-sonnet-4-6", max tokens=500, system=SYSTEM PROMPT, messages= {"role": "user", "content": f"Sources:\n\n{context}\n\nQuestion: {question}"} , return "".join block.text for block in response.content if block.type == "text" The system prompt explicitly forbids answering from anything but the retrieved sources and requires a citation for every claim, which is what makes a RAG answer auditable — a reader can trace “ pages the team lead directly ” straight back to runbook-oncall-escalation-003 and go read the actual policy. The full pipeline was verified end to end with the live API call, confirming the retrieved context correctly reached the prompt and the citation correctly came back in the final answer — the retrieve-then-generate chain works exactly as designed. With your API key exported as ANTHROPIC API KEY , run python generate.py , or import answer question and ask it anything against the document set. Fine-Tuning for Consistent Domain Output The scenario — deliberately different in kind from the one above — is this: a financial services company needs every incoming customer complaint sorted into a strict internal taxonomy BILLING DISPUTE, UNAUTHORIZED TRANSACTION, ACCOUNT ACCESS, FEE INQUIRY, CARD FRAUD SUSPECTED , none of which map cleanly onto any public standard, with a consistent structured output every single time. This is exactly the profile described above: the model doesn’t need new facts about the world, it needs to reliably learn this company’s specific vocabulary and a rigid output contract that a downstream ticketing system depends on. A system prompt can ask nicely for this, but at high volume and across edge cases, a system prompt alone doesn’t hold up nearly as consistently as weights that were actually trained on it. Prerequisites: - Python 3.10+ - pip install peft transformers a real training run additionally needs bitsandbytes and a CUDA GPU for 4-bit loading python dataset.py import json CATEGORIES = "BILLING DISPUTE", "UNAUTHORIZED TRANSACTION", "ACCOUNT ACCESS", "FEE INQUIRY", "CARD FRAUD SUSPECTED", def make example complaint text: str, category: str, severity: int, requires immediate action: bool - dict: return { "messages": {"role": "system", "content": "Classify the customer complaint into exactly one category from: " + ", ".join CATEGORIES + ". Return a JSON object with category, severity 1-5 , and requires immediate action boolean ." }, {"role": "user", "content": complaint text}, {"role": "assistant", "content": json.dumps { "category": category, "severity": severity, "requires immediate action": requires immediate action, } }, } TRAINING EXAMPLES = make example "I see a charge for $340 I don't recognize on my statement from yesterday.", "UNAUTHORIZED TRANSACTION", 4, True , make example "Why was I charged a $35 overdraft fee? I thought I had overdraft protection.", "FEE INQUIRY", 2, False , make example "Someone used my card at a gas station in another state this morning, that wasn't me.", "CARD FRAUD SUSPECTED", 5, True , def validate examples examples: list dict - list str : """Every label checked against the real taxonomy before training. In a small fine-tuning set, one mislabeled example can be a fifth of the training signal for its whole category.""" errors = for i, example in enumerate examples : assistant msg = next m for m in example "messages" if m "role" == "assistant" parsed = json.loads assistant msg "content" if parsed.get "category" not in CATEGORIES: errors.append f"Example {i}: '{parsed.get 'category' }' is not a valid category" severity = parsed.get "severity" if not isinstance severity, int or not 1 <= severity <= 5 : errors.append f"Example {i}: severity must be an int 1-5, got {severity}" return errors 1234567891011121314151617181920212223242526272829303132333435363738394041424344454647 dataset.pyimport json CATEGORIES = "BILLING DISPUTE", "UNAUTHORIZED TRANSACTION", "ACCOUNT ACCESS", "FEE INQUIRY", "CARD FRAUD SUSPECTED", def make example complaint text: str, category: str, severity: int, requires immediate action: bool - dict: return { "messages": {"role": "system", "content": "Classify the customer complaint into exactly one category from: " + ", ".join CATEGORIES + ". Return a JSON object with category, severity 1-5 , and requires immediate action boolean ." }, {"role": "user", "content": complaint text}, {"role": "assistant", "content": json.dumps { "category": category, "severity": severity, "requires immediate action": requires immediate action, } }, } TRAINING EXAMPLES = make example "I see a charge for $340 I don't recognize on my statement from yesterday.", "UNAUTHORIZED TRANSACTION", 4, True , make example "Why was I charged a $35 overdraft fee? I thought I had overdraft protection.", "FEE INQUIRY", 2, False , make example "Someone used my card at a gas station in another state this morning, that wasn't me.", "CARD FRAUD SUSPECTED", 5, True , def validate examples examples: list dict - list str : """Every label checked against the real taxonomy before training. In a small fine-tuning set, one mislabeled example can be a fifth of the training signal for its whole category.""" errors = for i, example in enumerate examples : assistant msg = next m for m in example "messages" if m "role" == "assistant" parsed = json.loads assistant msg "content" if parsed.get "category" not in CATEGORIES: errors.append f"Example {i}: '{parsed.get 'category' }' is not a valid category" severity = parsed.get "severity" if not isinstance severity, int or not 1 <= severity <= 5 : errors.append f"Example {i}: severity must be an int 1-5, got {severity}" return errors validate examples was tested against a deliberately broken set of three examples — an invalid category, a severity out of the 1-5 range, and a non-boolean flag — and it caught all three correctly. That matters more in this use case than in a larger dataset: with only a handful of examples per category, one bad label is a meaningful fraction of everything the model sees for that class. python from transformers import AutoModelForCausalLM from peft import LoraConfig, get peft model, TaskType model = AutoModelForCausalLM.from pretrained "your-base-model", load in 4bit=True, device map="auto" lora config = LoraConfig r=8, lora alpha=16, lora dropout=0.1, target modules= "q proj", "k proj", "v proj", "o proj" , task type=TaskType.CAUSAL LM, peft model = get peft model model, lora config peft model.print trainable parameters 123456789101112 from transformers import AutoModelForCausalLMfrom peft import LoraConfig, get peft model, TaskType model = AutoModelForCausalLM.from pretrained "your-base-model", load in 4bit=True, device map="auto" lora config = LoraConfig r=8, lora alpha=16, lora dropout=0.1, target modules= "q proj", "k proj", "v proj", "o proj" , task type=TaskType.CAUSAL LM, peft model = get peft model model, lora config peft model.print trainable parameters load in 4bit=True requires real GPU hardware. The LoRA wrapping mechanics were verified directly against a small model architecture built locally with the identical config, confirming the base model correctly freezes and only the adapter layers stay trainable — 3.33% of total parameters in that test, with the rest of the model’s weights untouched. That’s the actual mechanism behind why fine-tuning here is cheap and fast rather than a full retraining project: you’re training a small, focused adapter on top of a frozen base model, not the whole thing. When to Use Which: A Decision Framework 1. Does the information your model needs change regularly, or is it too large to fit in a prompt — product data, current policies, a growing document set? Use RAG. 2. Do you need every answer traceable to a specific source for a compliance or audit reason? Use RAG , since every claim in a RAG answer can be traced back to a specific retrieved chunk, which several regulatory frameworks around explainable AI outputs treat as a real, meaningful advantage over a fine-tuned model’s outputs, which require separate evaluation evidence to demonstrate the same thing. 3. Do you not yet have labeled training examples, or need something running this week rather than next month? Use RAG — it’s almost always the faster path to a working first version, regardless of what you eventually add on top of it. 4. Does the model need to consistently follow a tone, structure, or vocabulary that prompting keeps failing to hold at volume? Fine-tune. 5. Is your latency budget tight enough that an extra retrieval hop before every response is a real cost? Fine-tune , since a fine-tuned model answers directly with no retrieval step in the loop. 6. Is your query volume high enough that a smaller fine-tuned open model would be dramatically cheaper per query than a frontier API call? Fine-tune — the per-query savings can pay back the data-prep cost faster than most teams expect, once volume is genuinely high. Wrapping Up The real decision rule, stripped of all the framework language: RAG handles what the model needs to know, fine-tuning handles how the model needs to behave, and treating this as a single either/or choice is the most reliable way to waste months building the wrong thing first. Start with retrieval, since it’s almost always the faster path to something real, add fine-tuning only where you can point to a specific, persistent behavior problem that better prompting and better retrieval both failed to fix, and expect — going in — that the honest answer for a serious production system is probably both.