I pretrained the best small model under 100M parameters

BarunLM, a 35-million-parameter language model introduced by developer Barun, outperforms models over 6x larger, including LiquidAI's lfm2.5-230m, while training on a single H200 GPU. The model's architecture, inspired by DeepSeek and Kimi, combines hybrid attention, grouped query attention, partial RoPE, gated attention, bounded SwiGLU, and residual selectors, and is further refined with reinforcement learning using verifiable rewards.

introducing BarunLM 35m 🤗 i pretrained the world's best small model under 100m parameters. a 35m parameter language model that outperforms models over 6x larger, including LiquidAI's lfm2.5-230m, while training on a single H200 gpu. the gains come from a better architecture. we took inspiration from deepseek, kimi, and other frontier open source models. BarunLM combines hybrid local and global attention, grouped query attention to reduce memory, partial rope, gated attention, bounded swiglu, and residual selectors to improve how information flows through the network. after pretraining, we further improved the model with rlvr reinforcement learning with verifiable rewards using deterministic verifiers and rloo, allowing the model to reinforce correct reasoning without relying on human preference labels.