{"slug": "pretraining-at-scale-without-a-pretraining-team", "title": "Pretraining at scale without a pretraining team", "summary": "Mirendil pretrained its first coding-specialized foundation model, Mirendil-Base, with a single researcher working alongside AI agents in 3.5 months for under $10M in all-in compute, the company reported. Mirendil said AI agents wrote over 99% of the code, ran experiments and diagnosed failures, while the researcher directed the work; the model was built from an empty repository on PyTorch and JAX and trained only on public GitHub and web data, with no distillation, synthetic data or licensed data. Mirendil said its next model, Mirendil-Base-Large, is about 80% through training with gains in line with its scaling trends, and the largest, Mirendil-Base-XL, is about 19% through.", "body_md": "Mirendil exists to help scientists, engineers and small teams take on their most ambitious problems with AI. We build models and tools to speed up our own research, and plan to make those same tools available to anyone working on hard problems in their own field.\n\nFor most of the past decade, pretraining a competitive foundation model has taken a dedicated team: dozens of specialists at open-model efforts, and hundreds at frontier labs. Some research what the model should look like and how to scale it well. Some gather and clean the trillions of tokens it learns from. Others optimize and keep running the thousands of chips it trains on, and at the largest labs now, hundreds of thousands.\n\n            This year, that started to change. As AI capabilities have advanced, agents have taken\n            on a growing share of the work of building the next generation of AI inside frontier\n            labs. Both\n            [Anthropic](https://www.anthropic.com/institute/measuring-pace-of-ai-development)\n            and\n            [OpenAI](https://openai.com/index/research-acceleration-view-inside-openai/)\n            have published internal measurements showing this shift accelerating in recent months.\n            We wanted to push this to its extreme: could one researcher, working with agents,\n            pretrain a competitive coding model?\n          \n\nMirendil-Base is the result of this experiment. Our first pretrained model was built by one researcher in 3.5 months, with less than $10M in compute. This includes all the research and small-scale experiments it took to get there, as well as the final training run. That figure is all-in: it includes the agents’ token spend and data processing, and there was no distillation and we did not purchase or license any additional data. AI agents wrote over 99% of the code, ran experiments and diagnosed failures, while the researcher decided what to try and what to trust, and intervened when needed.\n\nTo run this as a clean experiment, we intentionally decided to write all the code from scratch on top of standard libraries like PyTorch and JAX, and the model learned only from organic, publicly available data.\n\nThis post covers what the model can do and the broader implications of our result.\n\n## What we built\n\nMirendil-Base is the first in a series of coding-specialized models, designed to perform especially well on the coding work done inside a frontier AI lab. The general skills needed here underpin most of the software work in AI research, from implementing experiments to debugging training code.\n\n**Built from scratch.** The project started from an empty repository. The\n            researcher and the agents built the training infrastructure, the model architecture,\n            the training algorithm and the data pipelines themselves, building only on generic\n            libraries like PyTorch, JAX and a handful of others, and using popular open-source\n            projects as references.\n          \n\n**Trained only on public data.** All of the training data came from GitHub\n            and other publicly available code, web pages and public datasets. We did not purchase\n            or license any additional data and used no distillation or synthetic data for this\n            experiment.\n          \n\n**Efficient by design.** We got significant efficiency gains from an\n            improved architecture, advanced tuning of the data mixture along with other data\n            improvements, and kernel optimization. Each of these is work a pretraining team would\n            normally split across specialists; here the researcher and the agents did all three.\n          \n\n**Scales well.** The recipe holds up as the models get bigger.\n            Mirendil-Base-Large, the next size up in the family, is about 80% of the way through\n            training, and its improvement over Mirendil-Base so far is in line with our scaling\n            trends. Mirendil-Base-XL, the largest, is about 19% of the way through and already\n            close to Large’s current score (Fig. 1).\n          \n\nFigure 1 shows how Mirendil-Base compares with five strong coding models: Kimi-K3, GLM-5.3, Nemotron-550B-Base, DeepSeek-V4-Pro-Base and Inkling on the task of predicting held-out code from our repositories.\n\nCross-entropy in bits per byte on seven external repositories published after the models were trained. Lower is better, and the best score in each row is in bold. Kimi-K3 and GLM-5.3 are post-trained models, which generally score slightly worse on bits per byte than their base versions.\n\n| Repository | Kimi-K3 | Mirendil-BaseMirendil | Nemotron-550B-BaseNemotron | GLM-5.3 | DeepSeek-V4-Pro-BaseDeepSeek | \n|---|---|---|---|---|---|\n| codex-sec | 0.200 | 0.228 | 0.241 | 0.247 | 0.245 | \n| grok-build | 0.183 | 0.210 | 0.220 | 0.220 | 0.225 | \n| openworker | 0.229 | 0.268 | 0.296 | 0.287 | 0.304 | \n| json-render | 0.127 | 0.127 | 0.185 | 0.181 | 0.173 | \n| native | 0.210 | 0.257 | 0.263 | 0.257 | 0.263 | \n| scriptc | 0.203 | 0.219 | 0.274 | 0.252 | 0.228 | \n| anydoc | 0.200 | 0.246 | 0.251 | 0.250 | 0.261 | \n| Average | 0.193 | 0.222 | 0.247 | 0.242 | 0.243 | \n\n## What this could mean\n\nUntil recently, the bottleneck in pretraining a competitive model was people: a team of specialists that took years to build. Mirendil-Base is our first evidence that this is changing: what was once only possible with a large research organization is now possible with a single researcher. When agents do most of the hands-on work, the bottleneck moves to compute, to data, and to the judgment of whoever is directing the agents. Here is what we think follows.\n\n**Empowering more people to build AI.** Mirendil-Base was built by an\n            experienced researcher using today’s general-purpose tools. As those tools\n            improve, and as we build ones made for this work, we expect the same to become possible\n            for teams whose deep expertise is in their own field rather than in pretraining: a\n            chip-design team, a robotics lab or a hospital training a model on its own data, for\n            its own questions. Today most organizations rent a general model and adapt it at the\n            edges. Base models, and the systems built around them, would then stop being something\n            only a few labs can make, and become much more accessible.\n          \n\n**New ways to scale organizations with compute.** Traditionally,\n            organizations have scaled their output by hiring. Compute adds another axis of scaling\n            in terms of amplifying humans much further. When agents do most of the hands-on work, a\n            team can take on more by adding compute. Ambition per person goes up: projects that\n            once needed an institution can start with a small team, and as compute gets cheaper and\n            more powerful, more people will be able to attempt them.\n          \n\n**Increasing leverage.** As agents take on more of the work, a\n            researcher’s day will look less like doing the work and more like leading a small\n            lab: setting the research direction, designing the experiments worth running, and\n            deciding which ideas deserve to be pushed further. Agents can already carry multi-day\n            pieces of work from start to finish, and they are starting to help with research taste\n            too: proposing directions and judging how to spend scarce compute, which was thought\n            out of reach for AI as recently as last year.\n          \n\n**The loop is starting to close.** Mirendil-Base didn’t build itself\n            autonomously; a person with expertise built it with AI tools. But the extent to which\n            these tools help with AI development has increased dramatically over the last year. Our\n            next model will be built partly by Mirendil models, and every model that gets better at\n            building AI makes the next one cheaper and faster to build. We don’t think we have\n            reached fully autonomous recursive self-improvement yet, but we are closer than we have\n            ever been.\n          \n\n## What’s next\n\nWe have started post-training Mirendil-Base and will share more updates soon.\n\nMirendil-Base is the smallest model in this family. Mirendil-Base-Large and Mirendil-Base-XL, a frontier-class model, are both still in training, and we will share updates on both.\n\n            If you’re interested in our work or would like to build with Mirendil, we’d\n            like to hear from you:\n            [contact@mirendil.com](mailto:contact@mirendil.com).", "url": "https://wpnews.pro/news/pretraining-at-scale-without-a-pretraining-team", "canonical_source": "https://mirendil.com/news/pretraining-at-scale-without-a-pretraining-team/", "published_at": "2026-10-21 16:22:49+00:00", "updated_at": "2026-10-21 16:42:47.589153+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-agents", "ai-research", "ai-startups"], "entities": ["Mirendil", "Mirendil-Base", "Mirendil-Base-Large", "Mirendil-Base-XL", "Anthropic", "OpenAI", "PyTorch", "JAX"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/pretraining-at-scale-without-a-pretraining-team", "markdown": "https://wpnews.pro/news/pretraining-at-scale-without-a-pretraining-team.md", "text": "https://wpnews.pro/news/pretraining-at-scale-without-a-pretraining-team.txt", "jsonld": "https://wpnews.pro/news/pretraining-at-scale-without-a-pretraining-team.jsonld"}}