Soofi Consortium Releases Soofi S 30B-A3B: An Open Hybrid Mamba-Transformer MoE Foundation Model For German And English

The Soofi Consortium released Soofi S 30B-A3B, an open hybrid Mamba-Transformer MoE foundation model for German and English, achieving the highest aggregate scores among fully open base models tested. Trained on Deutsche Telekom's Industrial AI Cloud using up to 512 NVIDIA B200 GPUs, the model totals ~31.6B parameters and activates ~3.2B per token, with a data recipe emphasizing German language content. The model outperforms peers like Olmo 3 32B and Apertus 70B on English and German benchmarks, demonstrating the effectiveness of its training approach.

A German research consortium has published the pretraining report for Soofi S 30B-A3B . It is an open base model for German and English. Training ran end to end on Deutsche Telekom’s Industrial AI Cloud in Munich. Preview weights are on Hugging Face. It is worth noting that among some of the fully open base models tested, Soofi S records the highest English and German aggregate scores. What is Soofi S 30B-A3B? Soofi S is a Mixture-of-Experts MoE hybrid Mamba Transformer foundation model. It totals ~31.6B parameters and activates ~3.2B per token. As a base model, it has no instruction tuning, alignment, or safety tuning. The KI Bundesverband coordinates the consortium, funded by the German Federal Ministry for Economic Affairs and Energy. Participants include Fraunhofer IAIS, DFKI, TU Darmstadt, ellamind, and Merantix Momentum. How the architecture works ? The efficiency claim starts with the layer stack. The network holds 52 layers. That is 23 Mamba-2 sequence-mixing layers, 23 granular MoE layers, and 6 Grouped-Query Attention GQA layers. Only those 6 GQA layers maintain a KV cache. Each MoE layer holds 128 routed experts, activates 6 per token, and adds 2 shared experts. Other details: model dimension 2688, squared ReLU, RMSNorm, and no positional embeddings. Soofi S adopts the Nemotron 3 Nano reference design without modification. The research team gives three reasons for that choice. Those are deployability on stacks such as vLLM, serving efficiency, and scientific control. Because the backbone is fixed, Nemotron 3 Nano becomes an architecture-identical baseline. The data recipe is the only moving part. The training recipe: ~26.68T consumed tokens in three phases That recipe follows a Warmup–Stable–Decay WSD schedule with a minus sqrt decay segment. Phase 1 consumed ~20T tokens on a diverse, quality-tiered mixture at a 1e-3 plateau. Phase 2 consumed ~6.58T tokens of high-quality annealing data. It decays 1e-3 to 1e-5, then continues at a constant 1e-5. Phase 3 consumed ~0.10T tokens at a 1,048,576-token sequence length. It extends the usable context window up to 1M tokens. German is the deliberate variable. It rises from 7.2% of Phase 1 effective tokens to 15.32% in Phase 2. The reference Nemotron 3 Nano mixture allocates about 5% to all non-English languages combined. German sources include HPLT v3 and v4, German Commons, German FinePDFs, and FineWiki. Genios adds 193M articles from 916 newspaper and trade-press archives, commercially licensed. Infrastructure follows the same sovereignty logic. The run used up to 512 NVIDIA B200 GPUs, from 24 March to 13 May 2026. It consumed ~253,000 B200 GPU-hours. Performance Those choices show up in the evaluation. Soofi S ran against 16 other open base models. All used the same lm-evaluation-harness pipeline, prompts, and few-shot settings. | Benchmark % | Soofi S 30B-A3B | Olmo 3 32B | Apertus 70B | EuroLLM 22B | Alia 40B | |---|---|---|---|---|---| | English aggregate | 70.1 | 67.3 | 62.4 | 61.2 | 59.0 | | German aggregate | 79.1 | 69.2 | 72.8 | 70.6 | 68.4 | | Held-out EN / DE | 41.4 / 41.8 | 33.1 / 36.2 | 27.6 / 33.5 | 30.8 / 33.9 | 28.0 / 29.4 | | HumanEval pass@1 | 73.8 | 63.0 | 30.2 | 39.3 | 23.8 | | MBPP-DE pass@1 | 84.2 | 70.8 | 50.9 | 59.4 | 45.6 | | LBPP pass@1 | 31.0 | 32.1 | 6.4 | 10.7 | 8.6 | | GSM8K | 86.1 | 80.7 | 65.4 | 25.1 | 65.4 | | Minerva MATH-DE | 56.0 | 48.5 | 29.0 | 28.4 | 12.9 | | INCLUDE-DE | 61.2 | 48.2 | 50.4 | 51.1 | 43.9 | | GPQA-Diamond | 43.4 | 33.3 | 27.3 | 30.3 | 29.8 | | GLP-DE | 88.8 | 73.0 | 81.2 | 78.2 | 65.4 | Against its architecture-identical reference, Soofi S gains 1.8 points on the English aggregate. German gains 4.2, and held-out English 6.7. That isolates the data recipe from the backbone. The picture changes against larger open-weight models. Qwen3.5 35B-A3B holds the highest English, German, and held-out means. Soofi S scores 70.1 English against 70.3 for Gemma 3 27B and Ministral 3 14B. On German it leads both, 79.1 to 78.4 and 78.3. Running the base model Reproducing any of this starts with the weights. The base repo is a gated preview, and it ships custom modeling code. pip install -U transformers accelerate torch hf auth login base repo is gated: accept the terms on the model page first from transformers import AutoModelForCausalLM, AutoTokenizer model id = "Soofi-Project/Soofi-S-Base" tok = AutoTokenizer.from pretrained model id, trust remote code=True model = AutoModelForCausalLM.from pretrained model id, trust remote code=True, dtype="auto", device map="auto" Base model: plain text completion. No chat template, no system prompt. prompt = "AI sovereignty is the idea that" inputs = tok prompt, return tensors="pt" .to model.device out = model.generate inputs, max new tokens=128 print tok.decode out 0 inputs "input ids" .shape -1 : , skip special tokens=True The same repo serves through vLLM: vllm serve "Soofi-Project/Soofi-S-Base" Where it fits ? Together, the numbers suggest three deployment shapes. First, German document work: GLP-DE 88.8 and INCLUDE-DE 61.2 suit an insurer fine-tuning on policy PDFs. Second, bilingual code assistance: MBPP-DE 84.2 suits teams prompting in German against Python tasks. Third, high-concurrency long-context serving: a support-ticket RAG system at batch 32 and 40K context matches the measured regime. For that case, test retrieval against the RULER and NaturalQuestions gaps. Key Takeaways - Soofi S activates 3.2B of 31.6B parameters; only 6 of 52 layers hold a KV cache. - It leads fully open base models: 70.1 English aggregate, 79.1 German aggregate. - German hits 15.32% of the Phase 2 mixture, versus ~5% multilingual in Nemotron. - Decode measures 8–9× dense 14–24B models at 40K context, flat from 4K to 256K. - Open gaps: RULER extraction at long inputs, factual recall, gated preview weights, unfinalized license. Check out the Pretraining report , and Project page https://www.soofi.info/soofi-s/ Hugging Face . Also, feel free to follow us on and don’t forget to join our Twitter https://x.com/intent/follow?screen name=marktechpost and Subscribe to 150k+ML SubReddit https://www.reddit.com/r/machinelearningnews/ . Wait are you on telegram? our Newsletter https://www.aidevsignals.com/ now you can join us on telegram as well. https://t.me/machinelearningresearchnews Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us https://forms.gle/wbash1wF6efRj8G58 Asif Razzaq is the CEO of Marktechpost Media Inc.. As a visionary entrepreneur and engineer, Asif is committed to harnessing the potential of Artificial Intelligence for social good. His most recent endeavor is the launch of an Artificial Intelligence Media Platform, Marktechpost, which stands out for its in-depth coverage of machine learning and deep learning news that is both technically sound and easily understandable by a wide audience. The platform boasts of over 2 million monthly views, illustrating its popularity among audiences.