You can't fine-tune a model on tribal knowledge that only exists in someone's head. RAG pipelines choke when the source documents are missing because the SME who wrote them retired last quarter. I've watched two projects stall completely because the "subject matter expert" listed in the project charter had already left, and the backup knew the what but not the why.
The knowledge extraction problem #
Current approach at most shops: schedule three hour-long interviews, record them, feed transcripts to an LLM, hope for the best. Result: hallucinated SOPs that miss the critical exceptions.
What actually works (slow, unglamorous, expensive):
-
Shadow shifts — sit with the expert for two full weeks, capture the micro-decisions they don't even realize they're making
-
Decision logs — force the expert to document why they chose option B over A for 50 real cases, not the textbook version
-
Synthetic validation — have the expert review LLM-generated scenarios and flag where the model would fail in production
Cost per expert: roughly 80-120 hours. Most leadership teams balk at this. They'd rather burn $200K on a vendor promising "automated knowledge capture" that delivers a polished PDF nobody trusts.
Where AI actually helps #
Not in replacing the extraction — in scaling it once you have the structured data. A fine-tuned 7B model on 2,000 validated decision logs outperforms GPT-4 on domain-specific routing tasks at 1/50th the inference cost. But you need the logs first.
The companies pulling this off treat knowledge capture as a capital expense, not an IT project. They budget for it like they budget for machine maintenance. The ones treating it as "change management" are the ones retraining models every six months because the distribution shifted when Martha from accounts payable left.
Hard truth #
If your AI strategy assumes a stable workforce to maintain and validate it, you don't have an AI strategy — you have a hiring plan disguised as tech. The demographic data isn't speculative. Workers 55+ are exiting faster than 25-35s are entering in every sector running legacy infrastructure. Either budget for systematic knowledge extraction now, or budget for model decay and production incidents later. There's no third option where the AI magically learns the undocumented exception handling by reading the ERP manual.
Next Starcloud's $250M raise shows orbital compute is hitting launch →
a guide to making money with AI, with plenty of directly applicable cases.