Ai2, the nonprofit lab behind the Olmo models, answered Lets Data Science's questions in writing about what releasing training data, code and intermediate checkpoints enables that open weights alone cannot. The lab says the evidence for a whole class of research questions sits upstream of the final checkpoint, points to outside work hunting paraphrased benchmark contamination in its public training data, and argues the remaining open-closed gap is increasingly about surrounding infrastructure rather than the models. It also confirmed mixture-of-experts models in the next Olmo generation.
Most of what gets called an open model is a set of downloadable weights. Ai2, the nonprofit Allen Institute for AI, has spent the Olmo program releasing considerably more than that: the training data, the code, the evaluation suites and the intermediate checkpoints from every stage of training. Lets Data Science asked the lab, in writing, what that extra openness has actually bought, and where the honest limits of the open-model story sit in 2026. The answers below were provided directly to LDS and are attributed to Ai2.
The evidence is upstream of the checkpoint
The lab's core claim is that open weights and full openness support different classes of research. "Open weights let you study, and sometimes build on, the finished model," Ai2 told Lets Data Science. "Full openness lets you inspect every step of the model development process."
The distinction matters because of where the evidence lives. Researchers using Olmo artifacts have studied benchmark contamination in the training data, how capabilities emerge during reinforcement learning, how alignment changes across fine-tuning stages, and which properties of pretraining make models more robust to later fine-tuning or quantization. "You simply cannot answer many of those questions reliably from weights alone," the lab said, "because the evidence you need is upstream of the final checkpoint."
What outsiders actually did with it
Asked for the most interesting outside use of the artifacts, Ai2 pointed to contamination research. Because Olmo's training data is public, researchers could search it not just for copies of benchmark questions but for paraphrased versions that conventional contamination checks miss. "That turns an abstract concern about evaluation reliability into something you can actually measure," the lab said.
The part Ai2 finds most compelling is the shift from evaluation to intervention. Outside teams were able to get inside the model, test a hypothesis about how a behavior was represented, intervene on that representation, and watch the downstream effect. That is the kind of mechanistic research the lab says fully open models exist to make possible, and much of it was work Ai2 never anticipated when it shipped the artifacts.
Where the gap is narrowing, and where the comparison breaks
On its own measurements, Ai2 says the open-closed gap is narrowing quickly on core language and reasoning, and increasingly on specialized capabilities. "Open models are no longer mainly compelling because they are open," the lab said. In its view they are now strong enough to be practical choices for a growing range of research and production use cases, which changes what the openness itself is for: less a consolation prize, more a property you select deliberately.
Where the comparison gets less honest is at the system level. Commercial products bundle the model with proprietary retrieval, tooling and inference optimization, "so you are not always comparing model to model." The lab's reading is that much of the remaining difference reflects infrastructure built around models rather than an inherent capability gap: "Openness is not a technical handicap." It is also the niche Ai2 claims between academia and the giants: scale that university labs cannot sustain, with the data, checkpoints, evaluations and lessons treated as public scientific output rather than proprietary advantage.
The honest decision rule for 2026
Asked for a practitioner's decision rule between a fully open model and a commercial API, Ai2 did not pretend openness always wins. If you want the fastest path to a very capable system and can accept an external provider's pricing, product constraints and limited visibility, "a commercial API can be the rational choice."
The open model becomes compelling when the requirements harden: running on your own infrastructure, deep fine-tuning, sensitive data, reproducible results, or independence from a single provider. The lab called out regulated industries such as healthcare, where teams may be working with sensitive data and face stricter requirements around where models run, how they are evaluated, and what they can inspect about the system. There is also a middle ground it sees often in practice: teams prototype against an API for speed, then migrate workloads to open models once customization, privacy or deployment flexibility becomes the binding constraint. "I would not argue that openness automatically wins every deployment decision. I would argue that it gives you options an API, by definition, cannot."
What comes next, and what Ai2 is afraid of
The lab confirmed the next generation of the Olmo family is in progress, including mixture-of-experts models, alongside continued work on post-training, reinforcement learning and evaluation infrastructure. The goal, it says, is not simply a stronger checkpoint but releases that stay inspectable across the full lifecycle, from data and pretraining through post-training and evaluation.
The closing warning was the sharpest line in the exchange. As models get more capable and training pipelines more complicated, Ai2 said, "it would be easy for 'open' to shrink back toward meaning only downloadable weights. We are trying to move in the opposite direction to preserve open science."
Key Points #
- 1Ai2 told Lets Data Science that weights alone cannot answer questions about benchmark contamination, capability emergence during reinforcement learning, or alignment drift across fine-tuning stages, because the evidence sits upstream of the final checkpoint. - 2Outside researchers used Olmo's public training data to hunt paraphrased benchmark contamination that conventional checks miss, turning an abstract worry about evaluations into something measurable. - 3The lab confirmed the next Olmo generation includes mixture-of-experts models, and warned that as pipelines grow more complex, "open" could shrink back toward meaning only downloadable weights.
Scoring Rationale #
Exclusive written Q&A provided directly to Lets Data Science by Ai2 on the Olmo program: what fully open training data and checkpoints enable beyond open weights, where the open-closed gap is narrowing in their own measurements, and an honest decision rule for practitioners.
Sources #
Original reporting, with the public references used alongside it.
LDS Exclusive
Reporting based on written answers given directly to Let's Data Science by Ai2 (Allen Institute for AI).
Practice interview problems based on real data
1,625 SQL & Python problems across 15 industry datasets — the exact type of data you work with.