{"slug": "is-it-even-possible-for-the-chinese-labs-to-distill-us-models", "title": "Is it even possible for the Chinese Labs to distill US models?", "summary": "Ex-OpenAI AI researcher and co-author of the original RLHF paper argues that Chinese labs cannot effectively distill US frontier reasoning models because these models do not return full reasoning traces, only abridged summaries. The researcher distinguishes three types of distillation—logit distillation, behavior cloning, and behavior parroting—and asserts that behavior parroting, the only feasible method via APIs, is unlikely to succeed for frontier reasoning models due to hidden reasoning steps.", "body_md": "# Is it even possible for the Chinese Labs to distill US models?\n\n### TL;DR: yes, but how it works is unintuitive!\n\nEveryone is arguing whether Kimi K3 distilled Anthropic. This whole conversation involves a horrible abuse of the term “distillation” which has come to mean almost anything the speaker wants it to mean.\n\nI’m an AI researcher who’s ex-OpenAI, I was a co-author on the original RLHF paper that led to ChatGPT. I know what I’m talking about. So here’s how to actually understand the distillation debate, because it’s way more complicated than the government is trying to make it seem.\n\n## What do people mean by distillation?\n\nThere actually are multiple techniques here that often get mixed up for each other. But here are the three types of distillation, in order of how powerful they are.\n\n### 1: Logit Distillation\n\nThis is what was originally meant by distillation. Most people (not Schmidhuber) would consider the original distillation paper as [Hinton et al.](https://arxiv.org/abs/1503.02531) There they show that models can be trained much more efficiently given a teacher’s raw outputs (“logits” - basically probabilities of each next token), which started the trend of using the technique to more efficiently train mostly smaller models given large teachers.\n\nWhen people say that all the labs distill models, this is what they mean. Anthropic for example likely distilled Opus from training on Mythos logits, trying to get Opus’ logits to match Mythos’ logits as closely as possible.\n\nBut when talking about cross-lab distillation, no one refers to logit distillation. This is because logits aren’t returned via frontier APIs anymore (they used to be!), although it would be possible to do logit distillation on an open weight model. Obviously, if we’re accusing the Chinese of doing distillation, it’s not classic distillation, because they don’t have access to logits.\n\n### 2: Behavior Cloning\n\nInstead of learning the raw logits, one can instead learn to replicate output tokens from a teacher. It’s not as informative as distilling from logits, but is similar in expectation and most importantly, you can do it from the outputs of an API.\n\nSupervised Fine-tuning (SFT), the first phase of most post-training algorithms, is equivalent to this, and it’s also commonly referred to as Imitation Learning.\n\nWhen referring to cross-lab distillation, people almost always refer to behavior cloning, but importantly, **this is not possible with today’s frontier reasoning models** because reasoning models don’t return their full outputs! They show abridged reasoning traces which are just summaries, specifically to make behavior cloning less effective. The thinking goes, if you can’t see the *actual *steps the model took to reason and instead just see some “chicken scratch” and the final answer, you won’t be able to re-derive that yourself.\n\nDistillation had been a big issue long before today, and the big labs have known this for a while, which is why when reasoning models were created, they chose not to share the reasoning traces with users and instead showed a summary. This allows for the model to do hidden work before outputting a final response, and this was supposed to prevent distillation - hence all the discussion about how distillation may not be possible (including from myself).\n\n### 3: Behavior Parroting\n\nInstead, I want to talk about what I call “behavior parroting” - mimicking the outputs of privileged work - and whether or not this can be used to train frontier models.\n\nNote: past work ([Chen et al.](https://arxiv.org/abs/2502.18001)) has called the no-chain-of-thought version of this “Only Answer” training, though it was mostly negative results on small models.\n\n## Can Behavior Parroting work?\n\n### The case against behavior parroting: Hallucination\n\nIntuitively, it doesn’t look like it should work. If you make a smart model do a lot of work in secret, then copy its outputs, the student doesn’t learn to mimic that work, it just learns to make plausible looking outputs. Hence, hallucination.\n\n### Case for behavior parroting #1: Style Transfer\n\n#### Point 1.1: most of instruction following is style\n\nThis was a very bitter pill to swallow when first teaching models to follow instructions, but it’s unfortunately the simplest best explanation for a lot of the phenomena we’ve observed. One of the most surprising results here was in [Hewitt et al.](https://arxiv.org/abs/2409.14254) where training on only outputs without instructions closed most of the gap with fully trained RLHFed models. (Sounds familiar?)\n\n#### Point 1.2: most of reasoning is style too\n\nThere is fairly strong evidence that this applies to reasoning models as well.\n\nReasoning models work almost as well when trained on reasoning traces that lead to *incorrect* answers. [[source1](https://arxiv.org/abs/2512.22255)] [[source2](https://arxiv.org/abs/2502.07374)]\n\n#### Point 1.3: there seems to be transfer between reasoning and response\n\nThis one is more of a guess, but the final assumption to pull our argument together is that there is some style transfer between the reasoning and the final response (the text returned from an API). I don’t know of any papers on this, but some points in favor are that model providers have found a need to separate non-reasoning and reasoning models because they interface with each other.\n\nThe [Hewitt et al.](https://arxiv.org/abs/2409.14254) paper seems to also imply that a large amount of post-training is simple token-level output bias, which would facilitate that transfer.\n\n### Case for behavior parroting #2: Pre-training works\n\nFun fact: You can interpret large scale pre-training as behavior parroting on the Internet! Most would likely consider this Behavior Cloning, but when writing the internet, humans do all sorts of intermediate steps + have additional information not present in the tokens.\n\nIt’s hard to argue against the unreasonable effectiveness of pre-training!\n\n## Summary\n\nWhen talking about distillation in the future, we, as an industry, should be more clear about what type we’re talking about.\n\nbehavior parroting - learning to imitate token outputs after reasoning, but without reasoning traces (because they’re hidden)\n\nbehavior cloning - learning to imitate tokens\n\nlogit distillation - learning to imitate probabilities\n\nOverall, there is a pretty compelling case that distillation can occur from commercial APIs, but it is an approximation (behavior parroting) of an approximation (behavior cloning) of (logit) distillation.\n\nThis is obviously an important problem, and I’d love to see scaling laws on behavior parroting of reasoning models!\n\n*Big thanks to Ke Deng, Sasha Sheng, Erik Gafni, and Haseeb Qureshi for helping me write/review this post.*", "url": "https://wpnews.pro/news/is-it-even-possible-for-the-chinese-labs-to-distill-us-models", "canonical_source": "https://www.completeskeptic.com/p/is-it-even-possible-for-the-chinese", "published_at": "2026-07-24 04:55:21+00:00", "updated_at": "2026-07-24 05:23:05.005653+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-research", "ai-policy"], "entities": ["OpenAI", "Anthropic", "Kimi K3", "Hinton et al.", "Chen et al.", "RLHF", "ChatGPT"], "alternates": {"html": "https://wpnews.pro/news/is-it-even-possible-for-the-chinese-labs-to-distill-us-models", "markdown": "https://wpnews.pro/news/is-it-even-possible-for-the-chinese-labs-to-distill-us-models.md", "text": "https://wpnews.pro/news/is-it-even-possible-for-the-chinese-labs-to-distill-us-models.txt", "jsonld": "https://wpnews.pro/news/is-it-even-possible-for-the-chinese-labs-to-distill-us-models.jsonld"}}