# Pretraining a Mini Kimi K3 for $252

> Source: <https://julin.ai/2026/08/21/mini-k3/>
> Published: 2026-08-20 12:00:00+00:00

Vizuara AI Labs trained a miniature Kimi K3 from scratch: 1.02B parameters, 145M active, 5B tokens, one H200, $252.35.

Not simplifying the architecture like [Karpathy’s microgpt](https://karpathy.github.io/2026/02/12/microgpt/), they kept [Kimi K3’s MoE and attention design](https://books.vizuara.ai/book/pretraining-a-mini-k3) intact.

That’s a surprisingly cheap way to learn pretraining (in real-world). They worked through expert collapse, data-mixing bugs, distributed-training bugs, kernels, and GPU utilization on a modern MoE architecture.

A few things worth noting: