{"slug": "distilling-the-moat", "title": "Distilling The Moat", "summary": "AI companies' models, protected only as trade secrets, can be replicated through repeated queries in a standard industry technique called distillation, undermining the 'moat' of training costs that supports their valuations. Elon Musk testified that xAI used distillation on OpenAI models to train Grok, calling it a general practice, while Anthropic accused three Chinese AI labs of setting up over 24,000 fake accounts to generate 16 million exchanges with Claude for distillation.", "body_md": "The original function of a Web server was to respond to queries by revealing the appropriate part of their internal data. This necessarily meant that repeated queries, for example from a search engine's or an internet archive's web crawler, could extract the server's entire internal data. Since the extracted data had been published on the Web, it was protected by copyright. This has led to many lawsuits, for example against the Internet Archive, Google and others. It is\nthe reason search engines only display \"snippets\" of the content they collect.\n\nAI companies' intellectual property is their models. They spend vast sums funding the technical and human resources to \"train\" these models, the racks of GPUs in the data centers, and the hordes of workers labeling images, and having \"genuine human conversations\" with the nascent model. These expenditures are thought to create a \"moat\" around the value thus generated, because it would be equally expensive for a competitor to create an equivalent model. It is thhis moat that supports their extraordinary valuations, despite their lack of earnings.\n\nLike many Web services, the function of an LLM is to respond to queries by revealing the appropriate part of their internal data, i.e. part of their model. Thus, repeated queries could in principle replicate the model. These models are not published, thus not protected by copyright law[[1](https://blog.dshr.org/feeds/posts/default#Footnote1)], and cannot be patented. Their only protection is as trade secrets. Someone using repeated queries would probably be violating the system's terms of service, but this isn't a strong legal protection. The AI companies are not in a strong position to argue that \"crawling\" their Web servers is illegal because that is how they created their models in the first place, which they argue is fair use.\n\nIt turns out that replicating other models by repeated queries is a standard technique in the industry, called \"distillation\". For example, Tim Fernholz reportd that *Elon Musk testifies that xAI trained Grok on OpenAI models*:\n\nOn the stand [in a California federal court](https://techcrunch.com/2026/04/29/on-the-stand-elon-musk-cant-escape-his-own-tweets/) on Thursday, Elon Musk was asked if xAI has used distillation techniques on OpenAI models to train Grok, and he asserted it was a general practice among AI companies. Asked if that meant “yes,” he said, “Partly.”\n\nAnd Rebecca Bellan reported that *Anthropic accuses Chinese AI labs of mining Claude as US debates AI chip exports*:\n\nAnthropic is [accusing](https://www.anthropic.com/news/detecting-and-preventing-distillation-attacks) three Chinese AI companies of setting up more than 24,000 fake accounts with its Claude AI model to improve their own models.\n\nThe labs — [DeepSeek](https://techcrunch.com/2025/09/29/deepseek-everything-you-need-to-know-about-the-ai-chatbot-app/), [Moonshot AI](https://techcrunch.com/2024/02/21/moonshot-ai-funding-china/), and [MiniMax](https://techcrunch.com/2025/01/15/chinese-ai-company-minimax-releases-new-models-it-claims-are-competitive-with-the-industrys-best/) — allegedly generated more than 16 million exchanges with Claude through those accounts using a technique called “distillation.” Anthropic said the labs “targeted Claude’s most differentiated capabilities: agentic reasoning, tool use, and coding.”\n\nDistillation works really well because the victim is massively subsidizing the use of its service. If use of the service was extremely profitable, distillation would be unaffordable.\n\nIn practice, distillation only replicates a part of the victim model. Companies use this, especially on open-weights models, to produce small, specialized models such as those described by David Berreby in *Small AI Models Gain Traction Around the World*. But the Chinese distillers [accused by Anthropic](https://techcrunch.com/2026/02/23/anthropic-accuses-chinese-ai-labs-of-mining-claude-as-us-debates-ai-chip-exports/) aren't starting from scratch, they already have a model. All they are trying to do is to replicate some capabilities that their model lacks. So they don't need to extract the whole model, just the relevant parts.\n\nOne of the features of the modern Internet that I've been writing about for more than a decade is the security disaster that is the *The Internet of Things*. Because there are hundreds of millions of \"smart\" devices exposed to the internet, almost all with known, unpatched vulnerabilities, it is trivial to construct vast botnets to attack Web sites from innocent, unattributable IP addresses. Ian Kelling of the Free Software Foundation reports from the front lines in * Our small team vs millions of bots*:\n\nTo begin with, GNU Savannah, the FSF's collaborative software development system, was hit by a massive botnet controlling about five million IPs starting in January. As of this writing, the attack is still ongoing, but the botnet's current iteration is mitigated. The goal is likely to build an LLM training dataset. We do not know who or what is behind this.\n\nThis is an example of an AI company using a botnet to collect training data. Suppose a company were to use this 5M IP address botnet for distillation. MiniMax''s [\"over 13 million exchanges\"](https://www.anthropic.com/news/detecting-and-preventing-distillation-attacks) would be a little under 3 exchanges per IP address.\n\nTo maintain their moat, the AI platforms have to do one of two things:\n\n- Implement anti-distillation defenses.\n- Raise prices enough to make distillation attacks uneconomic.\n\nThe 5M node botnet would be very difficult to defend against; each IP wouldn't generate enough traffic to characterize, and most would be residential addresses that might well be a customer. The AI platforms would be locked into an arms race with the distillers. Anthropic is [trying](https://www.anthropic.com/news/detecting-and-preventing-distillation-attacks):\n\nWe have built several classifiers and behavioral fingerprinting systems designed to identify distillation attack patterns in API traffic. This includes detection of chain-of-thought elicitation used to construct reasoning training data. We have also built detection tools for identifying coordinated activity across large numbers of accounts.\n\nThere is a cost to applying these defenses to most traffic, and Anthropic [admits](https://www.anthropic.com/news/detecting-and-preventing-distillation-attacks):\n\nno company can solve this alone. As we noted above, distillation attacks at this scale require a coordinated response across the AI industry, cloud providers, and policymakers.\n\nSuppose the closed models all have a capability the distillers want. They can spread distillation not just across millions of IP addresses and tens of thousands of accounts, but across all the closed models with the required capability.\n\nThe \"frontier\" models are already on the flattening part of the S-curve of technology evolution. The closer the open-weight models are to the closed ones the fewer distillation exchanges they need to catch up. Note that MiniMax needed \"over 13M\" but DeepSeek needed only \"over 150K\". Moonshot's recently released Kimi K3 is apparently close on the benchmarks — this might have had something to do with Moonshot's \"over 3.4M exchanges\" with Claude.. @jordanschneider tweeted [this image](https://x.com/jordanschneider/status/207883705959103734).\n\nThe [AI Security Institute tweeted](https://x.com/AISecurityInst/status/2078103148665667648):\n\nOur first public analysis of the open/closed weight gap in frontier cyber capabilities finds it is 4–7 months with GLM-5.2 and DeepSeek V4-Pro, narrowing from 6–10 months through most of 2025. Advanced capabilities are reaching less safeguarded open models faster than before.\n\nThe full details are in *How Far Behind the Frontier are Leading Open Weight Models on Cyber?* on the Institute's blog.\n\nImportantly, as [Max Weinbach tweeted](https://x.com/enzo_gte/status/2078102070482153717), the open weights models have capabilities the closed models lack:\n\nAfter using all three recent releases, Fable, GPT 5.6, and now Kimi, it's clear that the full power of the models has been significantly held back by the safeguard restrictions caused by last months debacle with the USG -- leading to the top models being quite literally lobotomized in some areas, which leads to subpar results as the safeguards pollute its entire thinking and problem solving abilities.\n\nThe funny part? Is that you could have predicted this outcome 2-3 years ago when you started to see the rise of Chinese EVs and smartphones compared to western alternatives.\n\nThey quite literally tried to copy the Tesla Model S and iPhone as hard as possible and then eventually it started to diverge to the point where their EVs and phones are just genuinely better *(which is why we have export controls banning their EVs, because they would literally drive all US manufacturers to ZERO)*\n\nThe AI Security Institute [confirms this](https://x.com/AISecurityInst/status/2078103156630712517):\n\nOur open weight model evaluations were largely unimpeded by safeguards. Of the two recent open models we tested, DeepSeek V4-Pro occasionally refused narrow cyber tasks, but this was easily circumvented by a small number of repeat attempts at refused tasks.\n\nThese findings indicate a narrow window before today’s frontier cyber capabilities may become widely accessible without safeguards.\n\nThe best the closed models' defenses could do would be to slow down the distillers enough to keep the platform's models a decreasing amount ahead; unlikely to justify their massive cost difference.\n\nWorse, raising prices probably wouldn't be an option. Not merely because *AI's Affordability Crisis* means that doing so would lose a lot of the enterprise customers they need in order to pay off the massive debts they are incurring. But more importantly, the existence of a free tier for a limited number of \"exchanges\" is an essential marketing tool. Because each of the botnet's addresses would fit into a free tier, the distiller would not see the increased, or indeed any, price.\n\nIf the AI platforms cannot deter distillation by pricing, and can only hope to stay slightly ahead of the distillers and the open-weight models by an expensive arms race, their moat is extremely shallow. It doesn't come close to justifying trillion-dollar IPOs, covering the country in data centers, or [launching them into space](https://blog.dshr.org/2025/12/data-centers-in-spaaaace.html).\n\n### Footnotes", "url": "https://wpnews.pro/news/distilling-the-moat", "canonical_source": "https://blog.dshr.org/2026/07/distilling-moat.html", "published_at": "2026-07-21 15:00:00+00:00", "updated_at": "2026-07-21 15:32:37.405426+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-ethics", "ai-policy", "ai-research"], "entities": ["Elon Musk", "xAI", "OpenAI", "Grok", "Anthropic", "Claude", "DeepSeek", "Moonshot AI"], "alternates": {"html": "https://wpnews.pro/news/distilling-the-moat", "markdown": "https://wpnews.pro/news/distilling-the-moat.md", "text": "https://wpnews.pro/news/distilling-the-moat.txt", "jsonld": "https://wpnews.pro/news/distilling-the-moat.jsonld"}}