Introducing Kilat: An Open-Source Framework for Building and Training Lightweight Language Models Developer Airukua released Kilat, an open-source lightweight transformer framework for pre-training, fine-tuning and deploying language models on consumer hardware, alongside a 151M-parameter model trained from scratch on 10B FineWeb-Edu tokens at a 2,048-token context length, reaching a perplexity of 21. Kilat combines Global Exponential Decay with Latent Multi-Head Latent Attention and a DeepSeekMoE-inspired mixture-of-experts feed-forward layer, with roughly 59.5M active parameters per token, and ships a training pipeline with checkpointing, model export, Hugging Face Transformers integration and fault-injection tests. The release targets researchers who want to experiment with alternative attention mechanisms and efficient training architectures without large-scale enterprise infrastructure. Hi everyone I’d like to introduce Kilat , an open-source project I’ve been developing to explore efficient language model architectures and make LLM training more accessible. The goal behind Kilat is to experiment with building, pre-training, fine-tuning, and deploying language models without relying entirely on large-scale enterprise infrastructure. Kilat is a lightweight transformer framework with several components: Hybrid Attention: Combines Global Exponential Decay with Latent Multi-Head Latent Attention MLA . Mixture of Experts: Includes a DeepSeekMoE-inspired feed-forward architecture. Training Pipeline: Supports pre-training, supervised fine-tuning, checkpointing, and model export. Hugging Face Integration: Supports loading and running exported models through the Transformers ecosystem. Assurance & Testing: Includes verification and fault-injection tests for identifying potential training and integration failures. To demonstrate the framework, I trained a language model from scratch: Parameters: approximately 151M Active parameters per token: approximately 59.5M Training data: FineWeb-Edu Training tokens: 10B Context length: 2,048 tokens PPL: 21 This is a base language model, not an instruction-tuned chatbot. Its purpose is to explore language modeling and the behavior of the proposed architecture. I wanted to explore a question: Can we make language model experimentation more accessible on consumer hardware while investigating alternative attention mechanisms and efficient training architectures? Kilat is my attempt to explore that direction. Framework: GitHub - Airukua/Kilat: Lightweight transformer toolkit, hybrid attention global decay + latent MLA , MoE FFN, KV-cache inference, and production training loop. Built for LLM research. · GitHub https://github.com/Airukua/Kilat Model: AiRukua/kilat-small-150M · Hugging Face https://huggingface.co/AiRukua/150M-Kilat-SMALL-10B I’m particularly interested in feedback from people working on: Efficient attention mechanisms and hybrid architectures Small language model pre-training Mixture-of-Experts implementations Training efficiency and reproducibility Evaluation methodologies for small language models I’d love to hear your thoughts, especially about the architecture, implementation choices, and what experiments would make this project more useful to the community. Thanks for taking a look