Vizuara AI Labs trained a miniature Kimi K3 from scratch: 1.02B parameters, 145M active, 5B tokens, one H200, $252.35.
Not simplifying the architecture like Karpathy’s microgpt, they kept Kimi K3’s MoE and attention design intact.
That’s a surprisingly cheap way to learn pretraining (in real-world). They worked through expert collapse, data-mixing bugs, distributed-training bugs, kernels, and GPU utilization on a modern MoE architecture.
A few things worth noting:
source & further reading
julin.ai — original article
Vanilla Neural Networks
Git Push is Hard
Agents Should Be Durable, Not Long-Lived