cd /news/large-language-models/introducing-kilat-an-open-source-fra… · home › topics › large-language-models › article
[ARTICLE · art-174297] src=discuss.huggingface.co ↗ pub= topic=large-language-models verified=true sentiment=↑ positive

Introducing Kilat: An Open-Source Framework for Building and Training Lightweight Language Models

Developer Airukua released Kilat, an open-source lightweight transformer framework for pre-training, fine-tuning and deploying language models on consumer hardware, alongside a 151M-parameter model trained from scratch on 10B FineWeb-Edu tokens at a 2,048-token context length, reaching a perplexity of 21. Kilat combines Global Exponential Decay with Latent Multi-Head Latent Attention and a DeepSeekMoE-inspired mixture-of-experts feed-forward layer, with roughly 59.5M active parameters per token, and ships a training pipeline with checkpointing, model export, Hugging Face Transformers integration and fault-injection tests. The release targets researchers who want to experiment with alternative attention mechanisms and efficient training architectures without large-scale enterprise infrastructure.

key takeaway

Kilat's 151M-parameter model, with about 59.5M active parameters per token from its DeepSeekMoE-style feed-forward layer, reached perplexity 21 after 10B FineWeb-Edu tokens at a 2,048-token context.

read1 min views2 publishedNov 22, 2026

Hi everyone!

I’d like to introduce Kilat, an open-source project I’ve been developing to explore efficient language model architectures and make LLM training more accessible.

The goal behind Kilat is to experiment with building, pre-training, fine-tuning, and deploying language models without relying entirely on large-scale enterprise infrastructure.

Kilat is a lightweight transformer framework with several components:

Hybrid Attention: Combines Global Exponential Decay with Latent Multi-Head Latent Attention (MLA).

Mixture of Experts: Includes a DeepSeekMoE-inspired feed-forward architecture.

Training Pipeline: Supports pre-training, supervised fine-tuning, checkpointing, and model export.

Hugging Face Integration: Supports and running exported models through the Transformers ecosystem.

Assurance & Testing: Includes verification and fault-injection tests for identifying potential training and integration failures.

To demonstrate the framework, I trained a language model from scratch:

Parameters: approximately 151M

Active parameters per token: approximately 59.5M

Training data: FineWeb-Edu Training tokens: 10B

Context length: 2,048 tokens

PPL: 21 This is a base language model, not an instruction-tuned chatbot. Its purpose is to explore language modeling and the behavior of the proposed architecture.

I wanted to explore a question:

Can we make language model experimentation more accessible on consumer hardware while investigating alternative attention mechanisms and efficient training architectures?

Kilat is my attempt to explore that direction.

Framework: GitHub - Airukua/Kilat: Lightweight transformer toolkit, hybrid attention (global decay + latent MLA), MoE FFN, KV-cache inference, and production training loop. Built for LLM research. · GitHub

Model: AiRukua/kilat-small-150M · Hugging Face I’m particularly interested in feedback from people working on:

Efficient attention mechanisms and hybrid architectures

Small language model pre-training

Mixture-of-Experts implementations Training efficiency and reproducibility

Evaluation methodologies for small language models

I’d love to hear your thoughts, especially about the architecture, implementation choices, and what experiments would make this project more useful to the community.

Thanks for taking a look!

── more in #large-language-models 4 stories · sorted by recency
── more on @kilat 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/introducing-kilat-an…] indexed:0 read:1min 2026-11-22 · —