{"slug": "post-training-post-morten-tenet-harvey", "title": "Post-Training Post-Morten: Tenet / Harvey", "summary": "A post-training engineer analyzed Harvey's first flagship legal AI model, Tenet, a post-trained Kimi-K3 built with Fireworks and Mercor, arguing that post-training is exceptionally difficult because control flows through a single objective function. The engineer outlined four practical reasons teams post-train models—expanding capabilities, improving first-try consistency, distillation for efficiency, and user alignment—noting these rank in decreasing order of complexity and value.", "body_md": "## **Some context.**\n\nI’ve spent the bulk of my adult life post-training models - first for social media at TikTok, then AI search at Neeva and for the last few years, at [Emissary](https://www.withemissary.com/) - supporting teams training everything from clinical note extractors to segment anything models, and even old-school BERT-based classifiers. Countless models, many scars - and reflexes built on spending more time staring at loss graphs than touching grass.\n\nSo when a new post-trained model drops, I can’t help but dig in. *Especially* when everyone seems to have big feelings about it. Harvey, probably the pre-eminent legal AI company, [released their first flagship model](https://www.harvey.ai/blog/introducing-harvey-ii) recently, a post-trained Kimi-K3 designed for legal reasoning. And it’s one worth analyzing - not only in itself, but in the context of the spate of recent similar efforts. \n\nBefore we go on, there’s two really important things to flag (courtesy, said scars).\n\nFirst, post-training is actually really hard. Like really really hard. Like throwing darts blindfolded after being spun around, hard. This is because the main way to control how a model is moulded is through its *‘objective function’* - a unified signal comprising all the different dimensions you want to score a model on. \n\nSecond, Tenet was built by a *post-training super-team*. Harvey’s application layer leadership combined with Fireworks’ (the literal leaders of Pytorch - a foundational DL library!) ML chops, all backed Mercor’s limitless and exceptional data acquisition capabilities. This is, in essence:\n\n**optimal data processed through optimal training techniques for the optimal use-case.**\n\nTl;dr: really smart people doing really hard things - worth commending regardless of outcome.\n\n## **Why do teams post-train.**\n\nBefore looking into the numbers, it’s important to consider what scenarios warrant a post-train. This becomes relevant later, I promise. Besides the obvious perception & talent attraction angle (“we’re not just a wrapper”; “we’re investing in frontier problems like long-horizon tool-calling”), there are four practical reasons to start messing around with model weights:\n\n1. **Expanding the possible** : This is a capability-oriented effort. There are clear task sets that existing frontier models fail at. Both foundational labs and vertical AI companies are investing in post-training to expand the set of tasks that AI models can accomplish.\n2. **Making the possible, probable** : Models can achieve a lot when they get many shots at goal - and that’s how frontier labs measure model capability. Pass@K = K shots at goal. But asking a model the same query 5 times and getting a right answer*somewhere* isn’t production-ready So, teams post-train models to increase the likelihood of Pass@1 - consistently getting the right answer on the first try.\n3. **Making the probable, accessible:** This is for efficiency. Making some smaller model  deliver the same quality of outputs as the frontier, cheaper and faster  (lesser parameters = less multiplications) - commonly branded as*distillation.* \n4. **Making the model more** ***user*****-***** aligned:*** This is a calibration-focused effort. Models being more concise, or having a gentler tone, or using formal language, are all great examples of this kind of post-training.\n\nThere are some reasons I refrain from mentioning above (like creating new action spaces) because they don’t apply commonly to the app layer, but these cover the most common ones.\n\nWithout gross generalization, to most MLEs - this list is in decreasing order of complexity and value: the hardest & most valuable goal in ML is expanding the scope of the possible, while consistency / efficiency is relatively straightforward and the preference alignment, *mostly* trivial.\n\nNow, with that in mind, let’s dig in.\n\n## **What the Tenet numbers show.**\n\nIn terms of approach - publicly available sources point to Harvey post-training Kimi-K3 using some form of grouped policy optimization (GRPO/GSPO) using rank-64 LoRA on mock dispute and case files graded by expert attorneys across ~2000 tasks.\n\nWe have 2 pieces of publicly released Tenet Data.\n\nFirst, the one on the public posts and blogs. This graph plotted the all-pass rate of different models on Harvey’s new [LAB benchmark dataset](https://www.harvey.ai/blog/introducing-harveys-legal-agent-benchmark) - 1,200 agent tasks across 24 legal practice areas, and is evaluated by over 75,000 expert-written rubric criteria.\n\n**A few notes:**\n\n- The graph has an  *inverted log-scaled X axis* - which makes it somewhat confusing.\n- A [comment](https://www.linkedin.com/feed/update/urn:li:activity:7496631286158135296?commentUrn=urn%3Ali%3Acomment%3A%28activity%3A7496631286158135296%2C7497027729213669377%29&replyUrn=urn%3Ali%3Acomment%3A%28activity%3A7496631286158135296%2C7497303394756452352%29&dashCommentUrn=urn%3Ali%3Afsd_comment%3A%287497027729213669377%2Curn%3Ali%3Aactivity%3A7496631286158135296%29&dashReplyUrn=urn%3Ali%3Afsd_comment%3A%287497303394756452352%2Curn%3Ali%3Aactivity%3A7496631286158135296%29) from Lin (CEO @ Fireworks, former Pytorch lead) also flags:\n  - The all pass metric was noisy, varying by ~5% per round. The uncertainty on the mean was controlled using 6 rounds of evals on the LAB and contracts.\n  - Most of the gains came from training with a harness having access to an extra tool and letting the model learn to use said tool via RL.\n\nTo make this easier to comprehend, I (yes, AI) redrew the graph without the axis inversion. I also attempted to rescale the graph to linear scale - but most models cluster under $3, making it really noisy - so  the scale is still **logarithmic**.\n\n**Here’s what’s immediately obvious:**\n\n- The Pareto changes: this is a really important flag because it impacts model selection for post-training, not just for Harvey, but all agentic AI companies.\n- Newer, not bigger models seem better at agentic behavior. Even Fable (rumored to be ~5-9T parameters) only barely beats much smaller models, within noise.\n- Agentic Behavior can be post-trained: with just 2000 samples, Tenet has a ~9 point lead on its base. That’s a lot of bang for every post-training buck invested.\n\nBut this is just one benchmark - one that admittedly is HEAVY on agentic tasks by design. So we get to look at the second piece of data:\n\nThis one is a lot more informative because it provides insights across LAB and more conventional legal AI benchmarks, allowing us to clearly see tradeoffs.\n\n- *Tenet’s gains over Kimi K3 in the four benchmarks with multi-step work & tool use.* Of those that test static document comprehension (APEX-v1, PRBench, CUAD and MAUD), Tenet is at parity or slightly below its own base model on datasets you’d conventionally expect to see gains on if legal capability was learnt.\n- Muse-Spark is actually a good overall legal AI model. It didn’t just beat Tenet at LAB, but also comfortably beat it at CUAD, MAUD and PR Bench. That’s both agentic and non-agentic datasets, at a fraction of the size (their size is not publicly released but models priced at their range are usually <1T MoEs).\n\nOff the data, it seems that Harvey & Meta both created a new level of agentic reasoning capability, while arriving at it from different angles.\n\n## **The Analysis.**\n\n**Potential goal / technique mismatch.**\n\nFrom an ML standpoint, there is an immediate core tension in the training. The [stated goal](https://www.harvey.ai/blog/post-training-update-harvey-tenet) of the effort was capability expansion - creating ‘frontier models’ that do tasks they couldn’t before, but the technique deployed was consistency-oriented. While RLVR - LoRA techniques continue to be the standard for app-layer AI companies, there is a mountain of evidence showing their limitations on this goal ([RLVR](https://arxiv.org/abs/2504.13837) | [LoRA](https://arxiv.org/pdf/2410.21228)). RLVR, largely, has been effective at raising consistency (Reason 2), but not capability, and LoRA only further constrains learning capacity.\n\nIt’s important to flag: In the absence of pass@K data, we don’t know whether the increase from K3 → Tenet is one of capability or consistency. It may be consistency branded as capability (in which case RLVR may still be appropriate).\n\n**Need for key ablations.** \n\nGiven that most gains admittedly emerge from the new tool, it would have been valuable to see an ablation of additional tool access without post-training, especially on the frontier Meta model.\n\nAnother fact of note is the commitment to a specific base model. The team chose to post-train (only?) Kimi-K3, interesting for two reasons:\n\n- This model is not on the corrected pareto: For capability expansion, as with most post-trains, the natural choice is to start from the best and take further,\n- A standard post-training effort  almost always involves a base model sweep - especially given the low cost of a LoRA finetune on only 2000 tasks. This provides a clear analysis of the value of the post-train vis a vis a model choice.\n\nPost-training across models would’ve yielded rich datapoints for next steps.\n\n**Generalized goals for a specialized domain-specific alpha.**\n\nThe third thing that stood out is the choice to focus on long-horizon workloads, even in the presence of unsaturated domain-specific benchmarks. This is a *personal* take, but long-horizon capability seems incredibly generalized, putting the team at no relative advantage to frontier labs. This should explain why Meta Muse is at par on LAB, even without ostensibly investing in legal data at the same scale. In comparison, building stronger domain-specific capability, especially where public data is lacking, would’ve been distinctly non-replicable. (NB: [Harvey is doing this with M&A](https://www.harvey.ai/blog/post-training-rlm-agents-for-m-and-a-diligence)).\n\n## **Suggestions**\n\nNotwithstanding any of the above, this is an incredibly valuable post-train project - incredibly valuable, unsaturated benchmark (LAB), meaningful performance gain over the base model, and a rich corpus of incredible takeaways and most importantly, and (hopefully) a repeatable training regime that can be run on new base models. It’s worthwhile to note that a change as small as a delayed Muse launch would’ve made this project land very differently, and that says more about the risk of post-training than the quality of their effort.\n\nIf I were training models at the app-layer, here’s what I’d do:\n\n- *You can’t hillclimb what you can’t define* . Start by defining a clear, trustworthy metric. Right now, the data suggests that LAB defines*agentic* reasoning, not*legal* agentic reasoning. Tenet’s gains sit entirely in the benchmarks with multi-step tool use and vanish on static comprehension, and Muse-Spark matches on LAB without a comparable legal data investment. The metric worth considering may be the delta - score on a legal task minus score on the same task with the legal judgment stripped out. The cheap version doesn’t need a new benchmark: tag the 75k existing rubric criteria as legal-load-bearing or general execution - potentially by running a non-lawyer through the same labelling exercise, with generalist-attorney ties being dead items.\n- *Get deep baselines* : The worst feeling in ML is sinking months into a project and realizing you had the wrong goal, or moved a post with no business value attached. Once the metric exists, I’d spend time before a single training run: base model sweeps, pass@K sweeps, trajectory-level failure and success breakdowns. Pass@K is the most underrated here, it’s the thing that separates*capability* gains from*consistency* gains, and without it the Tenet result stays ambiguous even to the team that produced it. Know your gaps before spending any compute fixing them.\n- *Then select one clear goal***.** Consistency, cost, capability, or style. Each has a different technique and a different definition of done, and stacking several together compounds complexity and risk. For capability specifically, the business question comes before the ML one: do you have a right to win it? That means unique data access, cheaper/better feedback loops (in-house attorneys, a trained LM judge), or trajectory-selection intel. Long-horizon tool-calling probably fails that test - it generalizes, which is why Muse arrived at the same place from a different direction. Domain capability where public data is thin does not. And if the delta from step 1 comes back near zero across the board, that’s a finding, not a failed measurement: it means the work factorizes into retrieval plus execution, and the right investment is consistency and cost.\n- *Start with low-hanging fruit.* Post-training for cost is not sexy in the way solving long-horizon tool-calling is. But it’s the highest-probability, shortest-time-to-value work, and getting a single post-trained model into production that has genuinely earned its place is a confidence builder without comparison. Ship that one first.\n\n## Closing thoughts.\n\nFor most of my career there have been two separate worlds: the people who train models and the people who build on top of them, with a clean handoff in between and not much shared vocabulary. That line is dissolving. Harvey, and other application AI companies, are pioneering a closed loop between model creation and consumptions, and the questions that open up are so much more interesting than either side is used to asking alone.\n\nI’d love to do more of these. If you’re post-training something and want a second opinion on a graph, a metric, or a result you can’t quite explain – send it over.", "url": "https://wpnews.pro/news/post-training-post-morten-tenet-harvey", "canonical_source": "https://newpriors.substack.com/p/post-training-post-mortem-tenet-by", "published_at": "2026-09-23 23:23:34+00:00", "updated_at": "2026-09-23 23:30:26.827515+00:00", "lang": "en", "topics": ["large-language-models", "ai-research", "ai-products", "ai-startups", "mlops"], "entities": ["Harvey", "Tenet", "Kimi-K3", "Fireworks", "Mercor", "Emissary", "TikTok", "Neeva"], "alternates": {"html": "https://wpnews.pro/news/post-training-post-morten-tenet-harvey", "markdown": "https://wpnews.pro/news/post-training-post-morten-tenet-harvey.md", "text": "https://wpnews.pro/news/post-training-post-morten-tenet-harvey.txt", "jsonld": "https://wpnews.pro/news/post-training-post-morten-tenet-harvey.jsonld"}}