{"slug": "vidraft-releases-aether-7b-5attn-a-fully-open-source-moe-llm-with-five-attention", "title": "VIDRAFT Releases Aether-7B-5Attn: A Fully Open-Source MoE LLM with Five Heterogeneous Attention Mechanisms", "summary": "Korean AI startup VIDRAFT has released Aether-7B-5Attn, a 6.59B-parameter Mixture-of-Experts foundation model on Hugging Face under Apache-2.0. The model features five heterogeneous attention mechanisms arranged via a 7×7 Latin Square layout across 49 layers, and includes full training code, data recipes, hyperparameters, logs, intermediate checkpoints, and evaluation code for reproducibility.", "body_md": "TL;DR:Korean AI startup VIDRAFT has released`Aether-7B-5Attn`\n\n, a 6.59B-parameter Mixture-of-Experts foundation model on Hugging Face under Apache-2.0 — shipping not just weights but training data recipes, full training code, hyperparameters, training logs, intermediate checkpoints, and evaluation code. If you care about reproducibility, sovereign AI, or heterogeneous attention research, this one is worth a close look.\n\n`Aether-7B-5Attn`\n\nis an open-source foundation LLM developed by VIDRAFT (비드래프트), a Seoul-based Pre-AGI AI startup. It was released on Hugging Face under the **Apache-2.0 license** and is positioned as a *truly* open model — not just \"open weights\" — following the philosophy of projects like Allen AI's OLMo, Apertus, and LLM-jp.\n\nKey facts from the release:\n\nBoth a base model and an instruct-tuned model are available, along with a live demo, all through Hugging Face.\n\nThe central architectural idea is **heterogeneous attention integration**: rather than using a single attention mechanism throughout all layers, `Aether-7B-5Attn`\n\ncombines five distinct attention types within one model:\n\nThe key engineering question when mixing attention types is: *where do you put each type in the layer stack?* Random or greedy assignment risks clustering certain mechanisms at certain depths, which can create uneven capability distributions across the network. VIDRAFT addresses this with a **7×7 Latin Square layout** across the 49 layers — a combinatorial design that ensures no single attention type is overrepresented at any particular depth. Think of it as an experiment design technique borrowed from statistics, applied to layer-wise architecture decisions.\n\nOn top of that, the **MoE backbone** means only a subset of parameters (~2.98B) are active per forward pass, keeping inference costs manageable relative to the total parameter count.\n\nThe \"fully open\" commitment means the repository is designed so an external team can trace the complete lineage: data composition and tokenization → training code and configuration → logs → intermediate checkpoints → evaluation procedures. The goal is independent reproducibility, not just deployability.\n\nVIDRAFT has previously disclosed the following public benchmark results for their Darwin model family (the production-grade line sitting above the Aether research architecture):\n\nSpecific benchmark numbers for `Aether-7B-5Attn`\n\nitself were not disclosed in the source article at time of publication. Given the full evaluation code is part of the open release, community-run evals on standard benchmarks (MMLU, HellaSwag, etc.) should emerge quickly.\n\nBoth the base and instruct variants of `Aether-7B-5Attn`\n\n, plus a live interactive demo, are publicly available on Hugging Face. Search for `VIDRAFT`\n\nor `Aether-7B-5Attn`\n\non [huggingface.co](https://huggingface.co) to find the model cards and repository.\n\nYou can pull the model with the Hugging Face CLI:\n\n```\npip install huggingface_hub\nhuggingface-cli download VIDRAFT/Aether-7B-5Attn\n```\n\n⚠️ Verify the exact repository slug on Hugging Face before running — use the official VIDRAFT organization page to confirm the published model ID.\n\nTraining code, hyperparameters, training logs, intermediate checkpoints, and evaluation scripts are all included in the public repository, consistent with the project's full-openness commitment.\n\n**Q: How is this different from \"open-weight\" models like LLaMA or Mistral?**\n\nA: Open-weight models let you download and run weights, but typically don't publish training data composition, training code, or the configuration needed to reproduce training from scratch. `Aether-7B-5Attn`\n\naims to publish all of that — data recipes, tokenization pipeline, training code, hyperparameters, logs, checkpoints, and eval code — so the entire manufacturing process is auditable and reproducible by anyone.\n\n**Q: Why does the Latin Square layout matter for the heterogeneous attention design?**\n\nA: When you mix multiple attention types across layers, naïve assignment can lead to certain mechanisms clustering at specific depths (e.g., all sparse attention in early layers, all full attention late). A Latin Square constraint distributes each type evenly across depth positions, preventing depth-wise bias and making it easier to isolate the contribution of each mechanism experimentally.\n\n**Q: Is this model production-ready, or more of a research release?**\n\nA: Based on the source article, `Aether-7B-5Attn`\n\nis framed as a research-grade foundation model release emphasizing transparency and reproducibility. VIDRAFT's Darwin family remains their production-tier line. That said, the Apache-2.0 license means you can use and build on it commercially.\n\n*Originally reported by 전자신문 (2026-07-20) — source article.*", "url": "https://wpnews.pro/news/vidraft-releases-aether-7b-5attn-a-fully-open-source-moe-llm-with-five-attention", "canonical_source": "https://dev.to/ai_openfree_b23025ef075cf/vidraft-releases-aether-7b-5attn-a-fully-open-source-moe-llm-with-five-heterogeneous-attention-2n11", "published_at": "2026-07-21 23:01:22+00:00", "updated_at": "2026-07-21 23:30:12.292302+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-research", "ai-startups", "ai-products"], "entities": ["VIDRAFT", "Aether-7B-5Attn", "Hugging Face", "Allen AI", "OLMo", "Apertus", "LLM-jp"], "alternates": {"html": "https://wpnews.pro/news/vidraft-releases-aether-7b-5attn-a-fully-open-source-moe-llm-with-five-attention", "markdown": "https://wpnews.pro/news/vidraft-releases-aether-7b-5attn-a-fully-open-source-moe-llm-with-five-attention.md", "text": "https://wpnews.pro/news/vidraft-releases-aether-7b-5attn-a-fully-open-source-moe-llm-with-five-attention.txt", "jsonld": "https://wpnews.pro/news/vidraft-releases-aether-7b-5attn-a-fully-open-source-moe-llm-with-five-attention.jsonld"}}