Hi everyone!
I’d like to introduce Kilat, an open-source project I’ve been developing to explore efficient language model architectures and make LLM training more accessible.
The goal behind Kilat is to experiment with building, pre-training, fine-tuning, and deploying language models without relying entirely on large-scale enterprise infrastructure.
Kilat is a lightweight transformer framework with several components:
Hybrid Attention: Combines Global Exponential Decay with Latent Multi-Head Latent Attention (MLA).
Mixture of Experts: Includes a DeepSeekMoE-inspired feed-forward architecture.
Training Pipeline: Supports pre-training, supervised fine-tuning, checkpointing, and model export.
Hugging Face Integration: Supports and running exported models through the Transformers ecosystem.
Assurance & Testing: Includes verification and fault-injection tests for identifying potential training and integration failures.
To demonstrate the framework, I trained a language model from scratch:
Parameters: approximately 151M
Active parameters per token: approximately 59.5M
Training data: FineWeb-Edu Training tokens: 10B
Context length: 2,048 tokens
PPL: 21 This is a base language model, not an instruction-tuned chatbot. Its purpose is to explore language modeling and the behavior of the proposed architecture.
I wanted to explore a question:
Can we make language model experimentation more accessible on consumer hardware while investigating alternative attention mechanisms and efficient training architectures?
Kilat is my attempt to explore that direction.
Model: AiRukua/kilat-small-150M · Hugging Face I’m particularly interested in feedback from people working on:
Efficient attention mechanisms and hybrid architectures
Small language model pre-training
Mixture-of-Experts implementations Training efficiency and reproducibility
Evaluation methodologies for small language models
I’d love to hear your thoughts, especially about the architecture, implementation choices, and what experiments would make this project more useful to the community.
Thanks for taking a look!