Yandex open-sourced an 80B model trained from scratch: what's inside and where it wins

Yandex released AliceAI-Foundation-80B-A3B-Base, an 80B-parameter mixture-of-experts language model trained entirely from scratch on roughly 18T tokens and published on Hugging Face under Apache 2.0. The base model activates 3B parameters per token across 512 experts and uses a hybrid attention stack with Kimi Delta Attention in three of every four layers, supporting a 262,144-token context. Yandex reports it outperforms its larger 235B predecessor on facts, math, code and long-context tasks, with its biggest leads on Russian-language, legal and educational benchmarks, while larger models such as Nemotron-3 Super and DeepSeek-V4 Flash still lead on English general knowledge.

Yandex just open-sourced a language model it trained entirely from scratch — no borrowed weights, no initialization from Qwen or Llama. It's called AliceAI-Foundation-80B-A3B-Base , it's on Hugging Face under Apache 2.0, and it's a surprisingly interesting release if you care about MoE architecture or non-English models. Here's what's inside, where it actually wins, and where the benchmark table deserves a second look. | Parameters | 80B total, 3B active per token | | Architecture | MoE: 512 experts, top-10 routed + 1 shared | | Layers | 48 — hybrid Kimi Delta Attention + gated attention 3:1 | | Context | 262,144 tokens | | Training | ~18T tokens, from scratch | | Languages | Russian, English | | License | Apache 2.0 commercial use OK | It's a base model — not instruction-tuned, not a chatbot. Yandex calls it experimental: a testbed for the architecture of its upcoming unified reasoning model, which will power agentic features in its Alice AI assistant. Compared to Yandex's previous flagship Alice AI LLM, 235B, October 2025 , it's almost 3x smaller overall and 7x smaller in active parameters — and, per the technical report, beats it on facts, math, code and long context. Every token is routed through 10 of 512 small experts plus one shared expert that's always on. You get the knowledge capacity of an 80B model with roughly the per-token compute of a 3B one. The attention stack is the other notable bit. Three out of every four layers use Kimi Delta Attention — a linear attention variant that folds history into a fixed-size state instead of keeping a KV entry for every past token. Only every fourth layer is classic gated attention. In practice that means 262K context without the usual KV-cache blowup — exactly the bottleneck you hit with "read this whole contract" or "understand this repo" workloads. Yandex compared base versions against open models in the same weight class and above. A selection from the official model card, including rows where it loses: | Benchmark | AliceAI 80B-A3B | Qwen3.5 35B-A3B | Nemotron-3 Super 120B-A12B | DeepSeek-V4 Flash 284B-A13B | |---|---|---|---|---| | WikiWebFacts Yandex's own | 86.5 | 62.4 | 72.8 | 83.2 | | HardMultiQA Yandex's own | 67.9 | 47.2 | 54.5 | 65.4 | | EduBench Russian | 74.2 | 42.9 | 44.0 | 67.7 | | ExpertFactsQA Law | 49.6 | 27.9 | 24.3 | 40.5 | | MATH-500 | 91.1 | 81.9 | 84.8 | 80.7 | | LiveCodeBench v5-6 | 50.5 | 50.4 | 50.4 | 38.1 | | BigCodeBench | 48.3 | 43.5 | 48.8 | 49.1 | | TriviaQA | 79.0 | 71.4 | 89.8 | 89.4 | | MMLU-Pro | 66.8 | 63.2 | 69.9 | 66.5 | | LongMemEval 128k | 64.6 | 55.6 | 64.8 | 68.0 | The pattern is clear. Anything that needs knowledge of the Russian-speaking world — language, law, school curriculum, local facts — it beats models 4x its active size. That's the training corpus talking. Math is strong too: HMMT Feb 2026 at 96.9 pass@32 vs 87.9 for Qwen3.5. On English trivia and general knowledge TriviaQA, MMLU-Pro , the bigger Nemotron and DeepSeek lead. No magic: 80B parameters can't hold as much about the world as 284B. 3B active sounds laptop-friendly, but all 80B have to sit in memory: ~160 GB in bf16, roughly 45 GB at 4-bit by my estimate. So in theory it's 64 GB Mac territory — and Apple Silicon is where MoE models shine, since you only read a small slice of weights per token. In practice, not yet. The official recipes are transformers and vLLM on NVIDIA GPUs the example uses four , and the KDA layers need the flash-linear-attention kernels. I couldn't find GGUF or MLX quants on release day, and non-standard attention is exactly what tends to delay llama.cpp support by weeks. Still, a fully in-house model with a permissive license, open benchmarks and a detailed tech report is a genuinely useful foundation for anyone building products in Russian — especially in legal, education and reference use cases where its lead is largest. Originally published at klukyanov.ru https://klukyanov.ru/notes/alice-ai-foundation-llm/ . Shorter weekly write-ups in Russian — on Telegram https://t.me/+KSAy8mfFOsgwYWMy .