{"slug": "marin-535b-a23b", "title": "Marin 535B-A23B", "summary": "Open Athena's Marin team has launched its largest model yet, a 535B-parameter mixture-of-experts (MoE) model with 23B active parameters, scheduled to pretrain for about 3 months on 18 trillion tokens including agentic, coding, and scientific data. The run is documented openly on GitHub and a W&B report, and so far it is progressing smoothly, aligning with predictions. This milestone comes just over a year after Marin joined Open Athena, during which the team grew from one to ten full-time employees and developed tools like the Delphi scaling suite and Iris scheduler.", "body_md": "A couple weeks ago, we kicked off our capstone run of the year, a mixture of experts (MoE) model with 535B total parameters and 23B active. The plan is for it to pretrain for about 3 months on 18 trillion tokens, including agentic, coding, and scientific data. Consistent with our commitment to [open development](https://openathena.ai/blog/open-development-of-frontier-ai/), the entire run—including architecture, data, predicted loss, etc.—is documented on [GitHub](https://github.com/marin-community/marin/issues/8435).\n\nIt's the largest model we've ever trained, and so far, it's going almost boringly well, lining up with our predictions and chugging along with a minimum of infrastructural or numerical fuss. You can follow along in our [W&B report](https://wandb.ai/marin-community/marin_moe/reports/535B-A23B-18T-Token-Hero-Run-Scaling-Ladder--VmlldzoxNzc2MDM5Ng), on the [GitHub run tracker](https://github.com/marin-community/marin/issues/8435) or on our purpose-built [dashboard](https://mtracker.oa.dev/hero-run-535b).\n\nIt also happens to be just over a year since Marin joined Open Athena, and so it's natural to reflect a bit. In short, it's been a big year. Since joining Open Athena, Marin has grown from one FTE to ten. And in that time, we've built a lot of the machinery we used to wish existed.\n\n- We published our\n[Delphi scaling suite](https://openathena.ai/blog/delphi/), releasing models trained up to 1e23 FLOPs, with scaling laws that allow us to predict performance of models 300x larger within 0.2%. - We scaled\n[Marin MoE V1](https://openathena.ai/blog/pretraining-speedup/)beyond 100B parameters and to 1e23 FLOPs. The 129B-parameter, 16B-active run landed within 1% of its preregistered loss prediction and achieved a 6.7× theoretical (3.6× realized on TPU v4) speedup over our dense Delphi recipe. - We developed\n[Iris](https://openathena.ai/blog/cluster-scheduling-with-iris/), our global compute scheduler that lets us take advantage of compute anywhere around the world. We also developed[DataKit](https://storage.googleapis.com/marin-public/held/harrier-k40-cluster-overview/2026.08.18/index.html?revision=uniform-sampling), our end-to-end system for curating and mixing pretraining data. - We're finishing up\n[Snowball](https://storage.googleapis.com/marin-public/benjaminfeuer/standing-up-a-cold-start-sft-pipeline-for-marin-models/2026.08.16/index.html), a 67B-A2B Mixture-of-Experts model on 10T tokens. We extended it to 262K context and built an end-to-end post-training pipeline to post train it. We'll be releasing it soon.\n\nWe also published some papers and helped incubate two separate efforts [building foundation models for biology](https://openathena.ai/blog/marin-dna/).\n\nAny one of those accomplishments would have seemed wildly ambitious for Marin a year or two ago. Unsurprisingly, I keep coming back to the 535B run. What I find most remarkable is just how boring it has felt since launch.\n\nYou see, our hero runs were traditionally *not* boring. When Percy and I started Marin back in 2024, our goals, though objectively more modest, still seemed audacious. The plan was to match Llama 3.1 8B with a fully open source model. Google's TPU Research Cloud generously gave us access to 512 TPU v5e cores, a dramatic jump in our compute resources at the time. [Marin 8B succeeded](https://marin.readthedocs.io/en/latest/reports/marin-8b-retro/): the model matched or exceeded Llama 3.1 8B on 16 of 19 base model evaluations.\n\nWe followed that with Marin 32B, opportunistically launched on a pile of preemptible v5p chips between reservations. We yolo'd the hypers and kicked off the run. No big deal.\n\nBut, we soon discovered that, among other things, one FTE is actually not enough to babysit a large training run on cobbled-together infrastructure on top of preemptible compute, particularly once it starts [throwing loss spikes](https://wandb.ai/marin-community/marin/reports/Marin-32B-Work-In-Progress--VmlldzoxMzM1Mzk1NQ). Who knew? After much surgery (notably QK norm) and an eventual retreat to a stable v4-2048, the model somehow limped across the finish line. It even turned out to be [a pretty good model](https://marin.readthedocs.io/en/latest/reports/marin-32b-retro/).\n\nGood times, but it was painfully clear that we were hitting a wall. We had shown that a tiny team could train surprisingly good models in the open, but we had also more or less exhausted what could be accomplished by one full-time person, borrowed infrastructure, and whatever compute happened to be lying around.\n\nAround this time, Jeff Hammerbacher, CEO of Open Athena, reached out with an offer to bring Marin into Open Athena, to give it the resources to become a real lab, without giving up on our commitment to doing the work in the open. We eagerly took him up on it. With the support of Jeff, David Siegel, and the rest of Open Athena, we went from one person working full-time on Marin to ten.\n\nThe goal over the following year was straightforward, if not exactly easy: take everything that had previously depended on improvisation and heroics and turn it into science and infrastructure. Then scale it up to get as close to the frontier as we could.\n\nWe were also incredibly fortunate this year to receive a [generous grant from the Jen-Hsun and Lori Huang Foundation](https://openathena.ai/blog/huang-foundation-marin-535b-training-run/) to support Marin's compute over the coming year. That support made our next leap in scale possible, starting with this 535B-A23B run. It's a pretty extraordinary bet to place on a ten-person open lab, and we're deeply grateful for it.\n\nI keep coming back to the \"boringness\" of our hero run because that maturity has been the point of everything we've been building. Our focus on smooth scaling laws with stable hyperparameter transfer has meant that we know almost exactly what model we'll be getting before we even kick it off. All the work on Iris and our other infrastructure has meant that we haven't even needed to set up a pager system for our on-call rotation. Hardware failures happen, but the system automatically recovers and the run keeps going.\n\nThat's not to say that there wasn't plenty of work to do in the lead up to the hero run launch. We had to finalize the architecture, prepare the final data mixture, do the scaling law runs to register our predictions, and also port our entire infrastructure and training stack from Google TPUs on GCP to Nvidia GPUs on CoreWeave. No big deal.\n\nObviously, there was a ton to do, and… I missed it. First, I was on a long-scheduled family vacation to Costa Rica. (In my defense, CoreWeave delivered the GPUs several weeks early. When do the GPUs ever come early?) Then I was lost in a chikungunya-induced fever dream for weeks while everyone else was pushing for the launch.\n\nAs painful as it was for me to miss the lead-up to the run, the fact that the project barely missed a beat in my absence is one of the things I'm proudest of. Two years ago, or even just a year ago, when Marin had one FTE, that obviously would have been impossible. I couldn't even take a fully unplugged vacation. So, despite my FOMO, I could not be prouder of this run and of Marin overall.\n\nBut it also means the credit for this launch belongs to the people who weren't shivering under a pile of blankets, briefly convinced they had developed hyperphantasia.\n\nFoundation model work increasingly gets attributed to organizations, or to enormous author lists where it's hard to tell who actually did what. These models are team efforts, but teams are made of people, and people deserve credit for the work they did. So:\n\n**Larry Dial** led the model and training recipe: architecture selection, optimizer and routing choices, scaling-ladder design and execution, and the final run specification. Without him, our loss curves would not be nearly so boring, nor so good.**Will Held** owned much of the production data and evaluation preparation. He found most of our 20 trillion tokens of source data, ran the experiments to determine our final training mixture, and built the evaluations that let us know whether the recipe was working.**Isaac Hodes** held everything together. He took a sprawling set of experiments and objectives and distilled them into the handful of questions we needed to answer before launch. He somehow kept that picture current while the answers—and sometimes the questions—changed underneath us.**Mark Muchane**, despite joining the team only weeks before launch, contributed across both the data pipeline and the training kernels. He built and audited parts of our quality-scoring pipeline, then brought the new dropless \"[Mixture-of-Kittens](https://cursor.com/blog/mixture-of-kittens)\" backend to the full hero shape and figured its remaining performance bottlenecks.**Russell Power** has been key to making the run boring from an infrastructure perspective. In addition to developing Iris and getting it running on CoreWeave, he hunted down reliability problems throughout the stack and cut recovery from hardware faults from roughly an hour to a few minutes.**Matt Wittmann** rebuilt our expert-parallel path around XLA:GPU's ragged all-to-all, fixing problems in the primitive itself along the way. The result nearly eliminated token dropping and raised throughput from 21% to 24% MFU, cutting the projected remaining runtime by roughly one-eighth.**Rafal Wojdyla** made the run possible across both data and systems. He created the DataKit processing pipeline with its embedding-based reclustering, provenance tracking, and global fuzzy deduplication. He also built our initial GPU-friendly expert-parallel implementation.**Romain Yon** brought vLLM support to our custom \"Grug MoE\" architecture. He validated checkpoint exports and inference parity, worked out how models at this scale could be served, and built the path from a finished training checkpoint to evaluation and downstream use.\n\nWe also aren't alone in this. Our partners at Nvidia and CoreWeave have been essential in getting this run off the ground. Nvidia's JAX team helped us with all the weird corner cases we dug up and the CoreWeave team has been incredibly responsive in tracking down hardware issues. Thanks also to Alec Radford and Kaiyue Wen for the sage advice in the leadup to the launch.\n\nFinally, we also want to thank **Zak Stone** and **Google's TPU Research Cloud**, which provided compute for the scaling ladder behind this run and has supported Marin since the very beginning. It is hard to overstate how much that support mattered. Marin simply would not exist without it.\n\nThis is just the pretraining phase of our 2026 hero run. Once it finishes, the model passes to our post-training team, led by **Ben Feuer**. We'll have much more to say about that soon.\n\nStepping back, this past year we've focused on building the machine that can reliably, boringly produce models. We've produced a lot of data, a lot of intermediate models, and a lot of information along the way. Almost all of it is public: on GitHub, on WandB, in experiment reports, and in conversations on Discord.\n\nThe problem is that there is now simply too much of it. Transparency is not the same thing as legibility. So that's one of our big goals for the coming year: to distill that transparency into something legible. We want to do a better job turning all of those experiments, decisions, successes, and failures into knowledge that people outside Marin can actually use.\n\nTwo years ago, Marin was created to see if a tiny open project could train a competitive foundation model at all. This past year, we've been trying to build a lab that can do it reliably. This next year, we need to get much better at making what we learn along the way useful to everyone else.\n\n## Cite this post\n\n```\n@misc{hall2026_marin_535b_launch_note,\n  author = {Hall, David},\n  title = {Marin 535B-A23B launch note},\n  year = {2026},\n  month = {sep},\n  howpublished = {\\url{https://www.openathena.ai/blog/marin-535b-launch-note/}},\n  note = {Open Athena Blog}\n}\n```\n\n", "url": "https://wpnews.pro/news/marin-535b-a23b", "canonical_source": "https://openathena.ai/blog/marin-535b-launch-note/", "published_at": "2026-09-04 06:48:36+00:00", "updated_at": "2026-09-04 07:23:18.659277+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-research", "ai-infrastructure"], "entities": ["Open Athena", "Marin", "Marin 535B-A23B", "Marin MoE V1", "Iris", "DataKit", "Snowball", "Google TPU Research Cloud"], "alternates": {"html": "https://wpnews.pro/news/marin-535b-a23b", "markdown": "https://wpnews.pro/news/marin-535b-a23b.md", "text": "https://wpnews.pro/news/marin-535b-a23b.txt", "jsonld": "https://wpnews.pro/news/marin-535b-a23b.jsonld"}}