# Introducing Kilat: An Open-Source Framework for Building and Training Lightweight Language Models

> Source: <https://discuss.huggingface.co/t/introducing-kilat-an-open-source-framework-for-building-and-training-lightweight-language-models/192365#post_1>
> Published: 2026-11-22 18:34:36+00:00

**Key takeaway:** Kilat's 151M-parameter model, with about 59.5M active parameters per token from its DeepSeekMoE-style feed-forward layer, reached perplexity 21 after 10B FineWeb-Edu tokens at a 2,048-token context.

Hi everyone!

I’d like to introduce **Kilat**, an open-source project I’ve been developing to explore efficient language model architectures and make LLM training more accessible.

The goal behind Kilat is to experiment with building, pre-training, fine-tuning, and deploying language models without relying entirely on large-scale enterprise infrastructure.

Kilat is a lightweight transformer framework with several components:

**Hybrid Attention:** Combines Global Exponential Decay with Latent Multi-Head Latent Attention (MLA).

**Mixture of Experts:** Includes a DeepSeekMoE-inspired feed-forward architecture.

**Training Pipeline:** Supports pre-training, supervised fine-tuning, checkpointing, and model export.

**Hugging Face Integration:** Supports loading and running exported models through the Transformers ecosystem.

**Assurance & Testing:** Includes verification and fault-injection tests for identifying potential training and integration failures.

To demonstrate the framework, I trained a language model from scratch:

Parameters: approximately 151M

Active parameters per token: approximately 59.5M

Training data: FineWeb-Edu

Training tokens: 10B

Context length: 2,048 tokens

PPL: 21

This is a base language model, not an instruction-tuned chatbot. Its purpose is to explore language modeling and the behavior of the proposed architecture.

I wanted to explore a question:

Can we make language model experimentation more accessible on consumer hardware while investigating alternative attention mechanisms and efficient training architectures?

Kilat is my attempt to explore that direction.

**Framework:** [GitHub - Airukua/Kilat: Lightweight transformer toolkit, hybrid attention (global decay + latent MLA), MoE FFN, KV-cache inference, and production training loop. Built for LLM research. · GitHub](https://github.com/Airukua/Kilat)

**Model:** [AiRukua/kilat-small-150M · Hugging Face](https://huggingface.co/AiRukua/150M-Kilat-SMALL-10B)

I’m particularly interested in feedback from people working on:

Efficient attention mechanisms and hybrid architectures

Small language model pre-training

Mixture-of-Experts implementations

Training efficiency and reproducibility

Evaluation methodologies for small language models

I’d love to hear your thoughts, especially about the architecture, implementation choices, and what experiments would make this project more useful to the community.

Thanks for taking a look!
