RAG vs Fine-Tuning: What Are You Actually Using in Production LLM Apps?

A developer survey on Hacker News asks practitioners where they draw the line between retrieval-augmented generation (RAG) and fine-tuning in production large language model (LLM) applications, highlighting trade-offs in retrieval quality, latency, knowledge freshness, and inference cost. The discussion reflects ongoing uncertainty about the best approach for real business data, with some noting that smaller specialized models can perform well at lower cost.

There is a lot of discussion around whether production LLM applications should rely on RAG, fine-tuning, prompt engineering, or increasingly capable base models. But the answer becomes less obvious when dealing with real business data. For example: RAG can provide current and private information but introduces retrieval quality, chunking, ranking, and latency challenges. Fine-tuning can improve specialized behavior and domain performance, but maintaining updated knowledge can become difficult. Prompt engineering is faster to implement but may become fragile as applications grow. And increasingly, smaller specialized models can sometimes perform surprisingly well at a fraction of the inference cost. For people building production systems: Where do you currently draw the line between RAG and fine-tuning? I’m especially interested in experiences involving: What has actually worked for you beyond the prototype stage?