A machine manufacturer I worked with has roughly 900 service reports. Somewhere in that pile sits the answer to genuinely valuable questions. Which machine series fails most, and why. Which fix actually holds. What breaks in winter installations that never breaks in summer.
Here is the uncomfortable part: nobody can actually give you those answers. Pieces live in a few experienced heads, but nothing is written out anywhere in full. No human being has read all 900 reports, and no human ever will. The knowledge exists inside the company and is simultaneously out of the company’s reach.
Almost every organization past a certain age has this condition. And almost everything we currently build with AI fails to treat it.
Companies do not actually want to search their data. Search was solved by Ctrl+F. What they want is the learnings:
The learnings are the valuable thing. And the learnings have a cruel property: they exist only in fragments, never in one place. No single document contains them. They live between documents, as patterns across hundreds of reports, as the process that emerges from three hundred filed cases, as the rule your two most senior people carry in their heads. When those two retire, the company does not lose employees. It loses its memory.
Now look at what the industry offers for this: “chat with your documents.” Embed the pile, retrieve the closest chunks, generate. I am not here to dunk on RAG. For lookup, it is fine. But notice what retrieval fundamentally is: it returns things that are written. Ask it a learning question, say “which series fails most and why,” and it grabs three chunks from three reports and improvises a confident synthesis on the fly. Every time. Differently. The learning was never extracted, never checked, never made into a thing. You are not querying knowledge. You are rebuilding it from scratch per question, badly.
Worse, the pile itself is a mess. Contradicting policy versions, outdated specs, mistakes copied forward. Any system that reads the pile raw inherits the pile. Mine once faced two PDFs claiming 12 and 24 months of warranty. Both retrieved. Coin flip answer.
And even when someone does the synthesis once, a consultant’s report or an intern’s summary wiki, two questions kill it. How do you know it is right? How do you know it is complete? There is no ground truth, no test, no definition of done. It starts rotting quietly the day it is finished.
That is the actual problem. Not retrieval. Turning tons of data into a body of learnings you can trust, and keeping it true.
Or, in one line: mining diamonds from a coal mine. The diamonds are in there. Nobody wants to sift the coal by hand.
Fair instinct. And notice why it feels right: finetuning a model is proper conventional machine learning. Data, labels, a loss, weight updates, a test set the model never saw. The full discipline. Exactly the rigor that the “chat with your documents” world is missing.
But it updates the wrong layer. The weights are a black box. When the answer about cold pours is wrong, you cannot open neuron four million and fix it. Your learnings get baked in at training time and go stale with the next batch of reports; every update means another training run, another eval, another deploy. And model finetuning is famously good at teaching skills and style, famously unreliable as a way to store facts.
So you would have the right loop wrapped around the wrong object.
Here is the move: keep the entire conventional ML training loop, and run it one layer up. Not on the weights. On the knowledge itself.
That is what I call Agentic Finetuning.
The mapping is structural, not poetic:
The trainable artifact is a wiki: readable pages of extracted learnings, every claim tied to its sources. The cold pour behavior page. The how filings actually go page. Pages a human can open and argue with.
The agent on top stays deliberately boring. It reads the wiki, nothing else. All the intelligence lives in what the wiki contains. So when an answer is wrong, you do not tweak a prompt and pray. You fix the knowledge, the way backprop fixes weights.
1. Dump plus purpose. Upload everything, unsorted, and answer one underrated question: what should be learned from this? “How our materials behave.” “How filings actually go.” The purpose steers the mining toward learnings instead of summaries.
2. Deep mining, wiki v0. Mine the whole pile properly, document by document, region by region, as deep as the purpose demands. Yes, this is the expensive step, and yes, it runs once. There is a reason for going deep now instead of lazily deepening later: if depth only arrives where the benchmark later complains, the wiki’s shape becomes an echo of the exam, and you have quietly optimized the knowledge toward the benchmark instead of toward the data. Deep first means the wiki’s completeness comes from the pile itself; the benchmark then verifies and repairs instead of steering the extraction. Contradictions found along the way, like my 12 versus 24 months pair, get collected into a queue for a human. They do not block the mining. They get answered at the end, because no model on earth knows which of your PDFs is the outdated one.
3. The benchmark, built from the data. Never from the wiki. This is the trap I fell into first. Generate test questions from the wiki, and whatever the wiki missed has no question about it. The blind spot writes its own exam and passes it. So questions come from the raw documents (“what should someone learn from this report?”), from the purpose (questions across documents: trends, causes, processes), and from the humans (“what do people actually ask you?”). Then split: 75 percent for training, 25 percent locked away as secret questions. Same topic mix, zero paraphrases, nobody sees them. If anyone teaches to the test, the test stops meaning anything.
4. Answer, grade, diagnose, rewrite. Every miss gets a root cause, and each cause routes differently. The learning exists in the data but the wiki missed it? Mine exactly that region again, deeper. Exists nowhere? Ask the human, never invent. Two sources disagree? Surface it. The question itself was bad? Fix the question, not the agent. And one rule above everything: the question’s answer is never written into the wiki. The benchmark says where to dig. The documents decide what gets written: general pages that any future question can draw from. Break this rule and your wiki quietly becomes an answer key.
5. The honesty check. When the training score saturates, the secret questions run, once. Small gap between training and secret scores: the wiki genuinely holds the learnings. Certify it, version it, ship it with its limits printed on the box. Big gap: congratulations, you built a parrot. So you refresh the exam: retire the burned questions, draft new ones from the data pool with new phrasings and new samples, lock away a brand new secret set, and keep tuning. The wiki drifted toward the old questions specifically, so the next rounds aim at targets it has never seen. The first time I watched 95 on training turn into 71 on secret questions, I felt personally betrayed. Then I remembered I would have caught the same overfit in any Kaggle notebook in 2019. The old lessons transfer. That is the entire point of Agentic Finetuning.
After launch, reality becomes the endless test set. Every question the agent fumbles in production becomes a wiki fix and a new benchmark question. New reports trigger the loop again. The company finally has a living, tested memory. One that survives retirements.
One sentence to hold it all:
The benchmark decides where to dig. The data decides what gets written. The secret set decides when you are done.
Organizations whose value is buried in finite, ownable piles. Engineering and materials firms with decades of reports and test protocols. Law and IP practices where “how the process really goes” lives across hundreds of cases. Manufacturers with manuals and service histories. Support teams with years of conversations, where the benchmark is basically lying in the export, waiting.
And any buyer who wants proof. A certification with secret question scores lands very differently than “trust us, it runs on the newest model.”
Two byproducts turn out to be products of their own. The cleaned wiki is onboarding gold: new hires read the learnings directly. And the audit (“your two policies disagree, and your team has been improvising”) is something companies pay consultants real money to discover. The loop produces it on the way.
I want to be straight about the limits, because the method earns trust by knowing them:
Nothing in Agentic Finetuning is a research breakthrough. Test sets are older than I am. LLM graders are standard. Knowledge extraction is a whole field. The edge is that nobody does the boring parts. Everyone is comparing models while their company’s actual learnings sit unread in a folder called FINAL_v2_old_DO_NOT_DELETE.
The models keep getting better on their own. Your data’s lessons do not extract themselves. Train the part that is actually yours. 😌
tl;dr Companies do not need better search over their data. They need the learnings: the knowledge that lives between documents and in retiring heads, which retrieval cannot return because it was never written down. Model finetuning has the right ML discipline but stores knowledge in the wrong place. Agentic Finetuning runs the same loop on a knowledge wiki: mine broad, benchmark from the raw data (never from the wiki, circularity is the classic trap), answer, grade, diagnose, rewrite, with secret questions to catch memorization and a certificate with honest limits at the end. Built for finite, arbitrable company knowledge. Not a fix for model skills, world knowledge, or data nobody can agree on.
I am a Data Scientist, Founder of Dataotter and building Agentic Finetuning into a proper product. If your company’s most valuable knowledge is currently a folder nobody has read end to end, or you are building something similar: comment or reach out. I genuinely love talking about this stuff.
Agentic Finetuning: Your Data Knows Things Nobody in Your Company Knows was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.