Anthropic’s latest model has been ‘distilled,’ and now there are open-weight models out there that perform almost as well as Anthropic’s latest model at a fraction of the cost.
This is causing all sorts of consternation and political engagement. From a profit-and-loss perspective, we’ve all now seen that there is a major line item for frontier labs that we should have been accounting for but haven’t been.
Previously, we could break down OpenAI’s or Anthropic’s business by thinking about the inputs of oodles of R&D combined with gigawatts of compute to make state-of-the-art frontier models. After the models are made, they need to be served, and there’s more compute that goes with commercialization efforts. There is a cycle of training new models and supporting legacy ones that keeps going as long as a competitor keeps trying to surpass the state of the art.
What was missing in that simple understanding was countermeasures. The Louvre needs to pay for security, Coca-Cola needs steps to protect its recipe, and Disney needs to take steps to protect its intellectual property. What is different for the frontier AI labs is that their products are giving away tiny bits of the secret sauce with every output and response that’s provided to users.
For a lab to capture value from a state-of-the-art model, its model needs to be markedly better than alternatives. The difference over the competition comes just as much from protecting against distillation (or theft) as it does from novel research breakthroughs to make the model in the first place. Anthropic, and likely OpenAI, have no choice but to develop ways to stop distillation from happening to future models. The current valuations for both companies assume that there is significant value capture from their frontier models. Frontier labs might still have months between release and when the model is distilled, but it’s naive to assume distillation won’t accelerate as distillers invest heavily to improve. Lengthening the time from releasing a model until it’s distilled will take considerable effort from the labs.1