I Tested RAG vs Fine-Tuning vs Long-Context on the Same Docs — the 1M-Token Window Collapsed at 24x…

A developer tested RAG, fine-tuning, and long-context approaches on the same 1.1M-token knowledge base, finding that long-context cost 24x more than RAG and failed on mid-document facts, while fine-tuning increased hallucination rates. RAG outperformed both alternatives in accuracy and cost efficiency.

Member-only story I Tested RAG vs Fine-Tuning vs Long-Context on the Same Docs — the 1M-Token Window Collapsed at 24x the Cost For two years the loudest prediction in applied AI was that retrieval-augmented generation was a temporary hack. Once context windows hit a million tokens, the story went, you would just paste your whole knowledge base into the prompt and RAG would die. Well, the windows arrived. Claude Sonnet 5 ships with 1M tokens, Gemini 3.5 Flash with 1M, and Gemini 3.1 Ultra with 2M. So I finally ran the experiment everyone keeps arguing about on Hacker News: same knowledge base, same 18 questions, three architectures — RAG, fine-tuning, and brute-force long-context. The long-context approach lost. Not narrowly. It cost roughly 24x more than the RAG pipeline at scale, it started dropping facts buried in the middle of the document well before it hit its advertised limit, and fine-tuning — the option people reach for when they want to "teach the model their data" — was the single worst way to add new facts. It hallucinated more than the base model I started with. Here is the whole test, the numbers, and copy-paste code so you can reproduce it before lunch. The setup: one knowledge base, three ways to answer I built a knowledge base out of ~1,000 internal-style documents product docs, policies, changelogs, support transcripts totaling about 1.1M tokens — deliberately sized so it just fits inside a 1M-ish window if you're willing to…