# Aether-7B-5Attn: VIDRAFT's Fully Open-Source MoE Foundation Model with Five Heterogeneous Attention Mechanisms

> Source: <https://dev.to/ai_openfree_b23025ef075cf/aether-7b-5attn-vidrafts-fully-open-source-moe-foundation-model-with-five-heterogeneous-attention-2ilh>
> Published: 2026-07-22 23:01:12+00:00

TL;DR:Korean AI startup VIDRAFT has released Aether-7B-5Attn on Hugging Face — a fully open-source Mixture-of-Experts foundation model with 6.59B total parameters that integrates five distinct attention mechanisms arranged in a novel Latin-square layer structure. It ships under Apache 2.0 with full training data recipes, code, hyperparameters, logs, intermediate checkpoints, and evaluation code, making it one of the most transparently documented open releases to date.

Aether-7B-5Attn is a foundation language model developed by VIDRAFT (비드래프트), a Seoul-based Pre-AGI AI deep-tech company. The model is published on Hugging Face under the **Apache 2.0 license** and is part of VIDRAFT's broader four-layer integrated AGI system, which includes the Darwin model family, the AETHER architecture, the PROMETHEUS world model, and the HEPHAESTUS embedded AI stack.

Key release artifacts that are all publicly available:

The project's stated philosophy is "true full-disclosure open source" — going significantly further than releases that publish weights alone.

**Architecture: Mixture of Experts (MoE)**

The model has **6.59 billion total parameters** but activates only approximately **2.98 billion parameters per token**, following the sparse MoE pattern where a router selects relevant expert sub-networks for each token. This keeps inference compute substantially lower than a dense model of equivalent total parameter count.

**Five Heterogeneous Attention Mechanisms**

The defining technical characteristic of this architecture is the simultaneous integration of five attention variants across the network:

Rather than assigning all layers to a single attention type, VIDRAFT distributes these five mechanisms across the **49-layer network using a 7×7 Latin square arrangement**. A Latin square ensures that each attention type appears exactly once in each row and column of the grid — preventing any single mechanism from clustering at a particular depth range and promoting balanced representational capacity throughout the network.

**Training Data Composition**

The model was trained on approximately **144.2 billion tokens** with an intentionally multilingual, non-English-centric data mix:

This weighting reflects VIDRAFT's goal of building a foundation model with first-class Korean language capability alongside strong mathematical reasoning, rather than treating both as secondary to English.

The source article does not report benchmark numbers specific to Aether-7B-5Attn itself. However, it cites the following public results for VIDRAFT's broader Darwin model family, which provides organizational context:

Aether-7B-5Attn's own evaluation results are available via the publicly released evaluation code on Hugging Face; engineers are encouraged to run benchmarks directly on their target tasks.

The base model, instruction-tuned model, and a live demo are all publicly accessible via Hugging Face. To get started:

```
# Search for VIDRAFT's models on Hugging Face
huggingface-cli search vidraft
```

You can also browse directly at `https://huggingface.co/vidraft`

or search for `Aether-7B-5Attn`

in the Hugging Face model hub. The training code and evaluation scripts are available in the associated repositories.

VIDRAFT also develops **VKAE** (an inference acceleration engine) and **VKUE** (a lightweight runtime engine) as companion AI infrastructure — check the Hugging Face organization page for the latest availability of these tools.

**Q: Is this safe to use in commercial products?**

A: Yes. The model is released under the **Apache 2.0 license**, which permits commercial use, modification, and redistribution provided you retain the license notice. Always verify the current license on the Hugging Face model card before deploying.

**Q: Why five attention mechanisms instead of picking the best one?**

A: Different attention mechanisms have different strengths at different network depths and context types. The 7×7 Latin square layer arrangement is designed to distribute all five mechanisms evenly across depth, so the model can leverage global context, local efficiency, and noise suppression properties simultaneously rather than making a single trade-off.

**Q: Why is math weighted so heavily in the training data at 37.8%?**

A: Heavy mathematical data is a well-documented strategy for improving general reasoning and instruction-following capability, not just numerical problem-solving. Combined with balanced Korean and English coverage, this mix targets strong reasoning across both languages.

**Q: Is this related to VIDRAFT's quantum computing work?**

A: VIDRAFT does research in quantum-assisted physical and life sciences, but Aether-7B-5Attn is a classical deep learning model. No quantum computing components are described in this release.

*Originally reported by 텐센트 뉴스 (중국) (2026-07-20) — source article.*