{"slug": "what-does-it-take-to-build-an-ai-model-let-s-look-at-olmo", "title": "What Does It Take to Build an AI Model? Let's Look at OLMo", "summary": "A developer's exploration of the Allen Institute for AI's OLMo repository reveals the components required to build an open large language model, including dataset provenance, tokenizer implementation, and model architecture. The walkthrough highlights that training data, tokenization, and transformer layers are essential, with OLMo's code adapted from MosaicML and minGPT.", "body_md": "*Hello, I'm Rijul, and I'm building LiveReview — a blast-radius aware AI code review built for your business-critical systems. [Star us](https://github.com/HexmosTech/LiveReview/) to help devs discover the project, give it a try, and share your feedback to help improve the product.*\n\nWhen we talk about training AI models, it can sound like you just put some training data in, and some chatbot comes out.\n\nOr for open models, we just download one, run it via Ollama, and boom, magic happens. We get some output from all the hard work done by our own GPU.\n\nWe only see the product, but we never see what is behind it.\n\nTo see what actually goes into building a model, it helps to look at projects that publicly release not just the model, but also their code, data, configurations, evaluations, and other artifacts.\n\nSo, to gain an understanding of how such AI models are being made, we will explore one such model.\n\nIt's called **OLMo**.\n\nYou can clone it from here:\n\n[https://github.com/allenai/olmo](https://github.com/allenai/olmo)\n\nWhen you open this repository, you can see many folders and files. Let's explore each part one by one from the POV of building an AI model.\n\nBefore any of the training happens, you need to have sufficient data.\n\nLoads of data.\n\nMatter of fact, AI models don't continuously learn from new data once they are trained. When new data needs to be incorporated, the model needs to be updated or retrained.\n\nSo, how do you know what data was used to train a model?\n\nOpen the README, and you can see the dataset mixes used by OLMo-2.\n\nNow, the next file you want to look at is inside `configs/official-1124/provenance.csv`.\n\n[https://github.com/allenai/OLMo/blob/main/configs/official-1124/provenance.csv](https://github.com/allenai/OLMo/blob/main/configs/official-1124/provenance.csv)\n\nThis gives more detailed information about the provenance of the datasets and data directories used by the project.\n\nNow that the data is sorted out, let's check the next thing.\n\nGo over to `olmo/tokenizer.py` for the tokenizer implementation,\n\nand `olmo_data/tokenizers` for the tokenizer files.\n\nIf you didn't know earlier, models don't directly process words as words. They receive **token IDs**, which are integers. These IDs are then mapped into vectors that the neural network can work with.\n\nA tokenizer is the tool that breaks text into smaller units called **tokens**.\n\nThese tokens aren't necessarily complete words. Depending on the tokenizer, a token can represent a whole word, part of a word, a character, whitespace, or another text unit.\n\nSo the `olmo/tokenizer.py` file:\n\n[https://github.com/allenai/OLMo/blob/main/olmo/tokenizer.py](https://github.com/allenai/OLMo/blob/main/olmo/tokenizer.py)\n\ncontains the logic for tokenizing text.\n\nThe actual vocabulary and the mapping between tokens and their IDs are stored in the tokenizer artifacts.\n\nIf we take one of the files under `olmo_data/tokenizers`, for example:\n\nYou can see the actual lookup table.\n\nHere, we map token strings to numbers.\n\nYou can see mappings like:\n\n```\n\"let\":1169, \"DE\":1170, \"red\":1171\n```\n\nIn `olmo/model.py`:\n\n[https://github.com/allenai/OLMo/blob/main/olmo/model.py](https://github.com/allenai/OLMo/blob/main/olmo/model.py)\n\nYou can see a large file. This contains things like the Transformer definition, attention layers, feed-forward blocks, layer normalization, and how they all stack together.\n\nThe things you see aren't necessarily built from scratch. They are built upon good examples and are combinations of existing techniques with their own tweaks.\n\nThe OLMo implementation was adapted from [MosaicML](https://github.com/mosaicml/examples.git) and [minGPT](https://github.com/karpathy/minGPT.git).\n\nNow we have the data and the model.\n\nNow we need something to connect both.\n\nThat's where we have the training loop.\n\nHere we have `olmo/train.py` and `olmo/optim.py`.\n\n[https://github.com/allenai/OLMo/blob/main/olmo/train.py](https://github.com/allenai/OLMo/blob/main/olmo/train.py)\n\n[https://github.com/allenai/OLMo/blob/main/olmo/optim.py](https://github.com/allenai/OLMo/blob/main/olmo/optim.py)\n\nIn `train.py`, you can find the training loop.\n\nAnd in `optim.py`, you can find the optimizer.\n\nThe training loop doesn't simply connect the data and model. It orchestrates things like running the model forward, calculating the loss, performing backpropagation, updating the model parameters, logging progress, and saving checkpoints.\n\nThis is not a once-run-and-forget loop. Depending on the training run, it can perform millions or even billions of training steps across distributed hardware.\n\nFor configurations, you can check `configs/official-1124/` for a real released model.\n\nOr, if you want a toy example, check:\n\n`configs/tiny/`\n\nEvery training run needs precise settings: what learning rate to use, how many tokens to train on, which dataset files to use, and dozens of other knobs.\n\nThe distributed training environment also needs to provide the required hardware and resources.\n\nOpen the `configs/` folder, and you'll find these choices stored as YAML files for different experiments and official model configurations.\n\nIn `olmo/checkpoint.py`, you can see the code that saves and loads the training progress.\n\nThe project provides checkpoints in both OLMo and Hugging Face-compatible formats.\n\nOLMo doesn't stop after the large-scale training stage.\n\nFor OLMo-2, there is a second, much smaller training stage using carefully chosen, targeted data, including the **Dolmino-mix-1124** dataset from Step 1.\n\nThe released OLMo-2 models used different Stage 2 setups, including multiple training runs. The resulting model checkpoints could then be **merged**, a process often referred to as model souping.\n\nYou can find the relevant configurations by looking for Stage 2 configurations inside the config subfolder.\n\nSo the basic idea is:\n\nAfter large-scale pretraining, the model continues training on a smaller, more targeted dataset.\n\nThink of it as taking a rough draft and then working on it further with a more focused set of examples.\n\nBut if you are looking at the final **OLMo-2 Instruct** models, there is another important part.\n\nAfter pretraining and the additional training stages, the models go through **post-training**.\n\nThis includes techniques such as:\n\nThe OLMo-2 Instruct models use post-training data and methods associated with the Tülu 3 work.\n\nThis is an important distinction.\n\nPretraining teaches the model broad language capabilities along with statistical patterns and other capabilities learned from the training data.\n\nPost-training is used to make the model more useful for following instructions and producing the kind of responses we expect from an assistant.\n\nSo, when we download an instruction-following model and chat with it, we are seeing the result of more than just the initial pretraining stage.\n\nIn the `hf_olmo` folder, there is a file called `convert_olmo_to_hf.py`.\n\nAn OLMo checkpoint has its own format.\n\nBut Hugging Face Transformers uses its own model, configuration, and tokenizer representations.\n\nSo this script provides the bridge between them. It converts an OLMo checkpoint into Hugging Face-compatible files so that it can be loaded by tooling that supports that format.\n\nThis is more of a **release and distribution step** than part of the actual training process.\n\nIn `olmo/eval` and the related evaluation components, you can find the code used for evaluating the model.\n\nFinally, a model needs to be evaluated using benchmarks, task-specific tests, human evaluation, safety evaluations, and other measurements to understand how well it actually performs.\n\nWhen we ignore the scale and the wide amount of jargon, from this example, building a model can be boiled down to this sequence:\n\nHope you got a good walkthrough experience of how such models are built.\n\nIt was interesting to me when I first stumbled across it, and it felt like suddenly a lot of the vagueness disappeared.\n\nSee you in another article.\n\nYour team's attention is limited, and the deluge of AI-generated code is making it harder to keep production reliable and secure without slowing you down.\n\nI'm building **LiveReview**, a blast-radius aware AI code review built for your business-critical systems.\n\nInstead of presenting every diff with equal emphasis, **LiveReview scores each change by blast radius — how far its impact reaches through your call graph — so you can focus attention where it actually matters.**\n\nSpend code review effort where business risk is highest — not spread evenly across every diff.\n\n⭐ Star it on GitHub: \n\nLiveReview is an AI code reviewer that scores every hunk of a diff by **blast radius**: how far a change reaches through your call graph, how much persistent state it touches, and how well-tested it is. A 3-line change to a shared auth check can outrank a 300-line UI tweak. Your team's attention goes to the highest-risk code first, not spread evenly across every diff.\n\n*LiveReview's Blast Radius & Review Priority scoring, live in the diff viewer.*\n\n| The exact math, not a black box | Visualize blast radius at a glance | Every factor that feeds the score | \n|---|---|---|\n\n**Here's the goal:**\n\n**Click below to try LiveReview with your codebase:**", "url": "https://wpnews.pro/news/what-does-it-take-to-build-an-ai-model-let-s-look-at-olmo", "canonical_source": "https://dev.to/rijultp/what-does-it-take-to-build-an-ai-model-lets-look-at-olmo-1k4m", "published_at": "2026-09-09 20:21:38+00:00", "updated_at": "2026-09-09 20:42:28.781014+00:00", "lang": "en", "topics": ["large-language-models", "ai-research", "developer-tools"], "entities": ["OLMo", "Allen Institute for AI", "MosaicML", "minGPT", "LiveReview", "Rijul"], "alternates": {"html": "https://wpnews.pro/news/what-does-it-take-to-build-an-ai-model-let-s-look-at-olmo", "markdown": "https://wpnews.pro/news/what-does-it-take-to-build-an-ai-model-let-s-look-at-olmo.md", "text": "https://wpnews.pro/news/what-does-it-take-to-build-an-ai-model-let-s-look-at-olmo.txt", "jsonld": "https://wpnews.pro/news/what-does-it-take-to-build-an-ai-model-let-s-look-at-olmo.jsonld"}}