Why Fine-Tuning Is No Longer Your First Choice for Custom AI?

Harvey, a legal AI company, found in 2025 that seven general-purpose Frontier AI models surpassed its custom fine-tuned model on its own legal benchmark, despite no custom legal fine-tuning. Bloomberg's BloombergGPT, trained from scratch, was outperformed by GPT-4 and ChatGPT on many financial benchmarks. These results suggest that fine-tuning is no longer the first choice for custom AI, as general models have caught up and techniques like RAG offer alternatives.

Is fine-tuning large language models still needed today? Let’s take a well-known legal AI company called Harvey. Back in 2023, they fine-tuned their own model — their own custom AI — in partnership with OpenAI. In blind tests, attorneys preferred this fine-tuned model over the Frontier model at the time, which was GPT-4. They preferred it 97% of the time. A win for fine-tuning: they built a custom AI that lawyers actually preferred over the off-the-shelf leading model. So the lesson is, if a general-purpose model isn’t quite right for some specific use case like legal work, then fine-tune it. Right? First, let’s define what fine-tuning actually is. We start with a base model, a base LLM. This is what comes off the shelf — either working with a Frontier lab directly or picking an open-source model. The base model is trained on a massive amount of data; effectively, we scrape information from the internet and use it to train the base model, so there’s a lot of general knowledge baked into the model's weights. Fine-tuning takes that base model and customizes it by continuing its training, but now on a much more focused dataset — some specific documents really focused in one particular area. It might be legal contracts, or an organization’s internal support tickets: the stuff that isn’t sitting around on the internet waiting to be scraped by base models. The result is a new model, a fine-tuned model, and that fine-tuned model incorporates the information from the focused dataset plus all of the weights from the base model, now adjusted with that additional data. This resulting model should get better at certain narrow tasks. That is the wonder of fine-tuning. But in practice, how well does it work? Let’s go back to that legal AI company Harvey. In 2025, they created their own legal benchmark to measure how effective their models were at performing particular tasks, and they tested their fine-tuned system against the latest crop of Frontier AI models — the custom model against a bunch of general-purpose Frontier AI models available at the time. The result? Seven of the general-purpose models had now surpassed the company’s custom model on the benchmark — models that had never received any custom legal fine-tuning, yet were still better. Bloomberg saw something similar. They famously trained BloombergGPT from scratch, and later evaluations found GPT-4 and ChatGPT outperforming BloombergGPT on many financial benchmarks. So where does that leave fine-tuning today? Is all that custom training worth doing when big frontier general models keep getting smarter on their own? To answer that, it’s worth considering how general models have, in many cases, caught up to custom-trained ones. There are a few reasons. General models have gotten better — but if we’re not adjusting weights, how do we make a general model behave like a specialist, like a legal scholar, for example? It turns out there’s a whole stack of customization techniques that don’t touch the model weights at all. The first is RAG, retrieval-augmented generation. Instead of training the documents into the model, the application retrieves — that’s the R in RAG — the documents at query time and then feeds them into the prompt. Minimal RAG: retrieve relevant chunks at query time, then feed them into the promptfrom openai import OpenAIclient = OpenAI def answer with rag question: str, vector store, k: int = 5 - str: R — retrieve the most relevant documents for this specific query docs = vector store.similarity search question, k=k context = "\n\n".join d.page content for d in docs A + G — augment the prompt with that context, then generate prompt = "Answer the question using only the context below.\n\n" f"Context:\n{context}\n\n" f"Question: {question}" resp = client.chat.completions.create model="gpt-5", messages= {"role": "user", "content": prompt} , return resp.choices 0 .message.content In practice, this means the model never “learns” your documents — it simply reads the most relevant ones fresh on every query, so your knowledge base can change without ever retraining a thing. There’s also the consideration of context, specifically context engineering. The idea is that a good prompt is a carefully assembled bundle of context: the system prompt, the relevant data, maybe some format guidelines and the like, all packaged together. Context engineering: assemble the prompt as a deliberate bundle of contextdef build context system prompt: str, retrieved data: str, format rules: str, user query: str : return {"role": "system", "content": system prompt}, who the model should be {"role": "system", "content": f"Relevant data:\n{retrieved data}"}, grounding {"role": "system", "content": f"Output format:\n{format rules}"}, guardrails {"role": "user", "content": user query}, the actual ask Notice there’s no training here at all — the “specialisation” comes entirely from how deliberately the prompt is assembled, not from the weights. The third thing to consider is agent skills — those MD files you can create. Skills are folders of files that package up procedural knowledge: basically how to do something, and the tools to use to do it. The model loads them on demand when it sees a task that calls for them. So instead of fine-tuning a model to know how to write SQL queries against a very specific schema, a SQL agent skill can tell the model exactly what to do — and any general-purpose model can use that skill. Here’s what a minimal SQL agent skill might look like — a simple Markdown file that hands the model the schema, the rules, and the tool it needs: ---name: sql-reporting-agentdescription: Write and run SQL against the analytics warehouse schema.--- SQL Reporting Skill Schema- orders id, customer id, amount, status, created at - customers id, name, region, signup date Rules- Always filter out status = 'cancelled' for revenue queries.- Use explicit JOINs, never comma joins.- Return at most 1000 rows unless asked otherwise. Tools- run sql query: str - table executes read-only SQL and returns rows Procedure1. Restate the question as a metric + dimensions + time range.2. Draft the SQL using the schema above.3. Validate column names against the schema before running.4. Call run sql, then summarise the result for the user. So essentially, fine-tuning isn’t the only path to customization. There’s a whole stack of options that work without ever touching model weights. Having all these no-weight options is a good thing, because fine-tuning is not free. In addition to the training run itself, there’s a cost in collecting the examples, evaluating results, and avoiding regressions — plus a cost of maintaining the custom models as the frontier models move on. All of this begs the question: does anyone still need to fine-tune at all? Yes — but for a much narrower set of reasons than back in 2023. There’s a modern technique called LoRA , low-rank adaptation, that lets a team fine-tune by training a small adapter that sits on top of an existing base model. We’ve got the base model with its weights, and then this adapter that sits on top of it. Most of the original weights stay locked. In fact, a lot of what gets labeled as fine-tuning in production today is some flavor of LoRA or a related parameter-efficient method. LoRA fine-tuning: train a small adapter, keep the base weights frozenfrom peft import LoraConfig, get peft modelfrom transformers import AutoModelForCausalLMbase = AutoModelForCausalLM.from pretrained "base-llm-7b" lora config = LoraConfig r=8, rank of the adapter small = few extra params lora alpha=16, target modules= "q proj", "v proj" , lora dropout=0.05, task type="CAUSAL LM", model = get peft model base, lora config model.print trainable parameters Only the adapter is trainable; the original weights stay locked. Notice that only the small adapter is trainable while the base model stays frozen — which is exactly why LoRA is so much cheaper and faster than full fine-tuning. Fine-tuning does still make sense in certain situations: Reinforcement fine-tuning RFT : sample answers, grade them, reward the good onesdef rft step model, prompt: str, expected ans: str, grader - None: candidates = model.sample prompt, n=4 sample candidate answers Programmatically grade each candidate scored = ans, grader expected ans, ans for ans in candidates Pseudocode: update the model so higher-scoring answers become more likely update policy model, prompt, scored def exact match grader expected: str, answer: str - float: Works only when there is a definitive right answer return 1.0 if answer.strip == expected.strip else 0.0 This is why RFT only works when correctness can be measured programmatically — no grader, no reward signal, no training. So fine-tuning isn’t dead. From a practical decision framework today, I think of the order going something like this: But what do you think? Does fine-tuning still have its place? Let me know in the comments. If you found this helpful, consider clapping👏 so others can find it too and follow me for more amazing technical AI content Why Fine-Tuning Is No Longer Your First Choice for Custom AI? https://pub.towardsai.net/why-fine-tuning-is-no-longer-your-first-choice-for-custom-ai-ddd69c23298f was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.