{"slug": "lara-translate", "title": "Lara Translate", "summary": "Translated's Lara Mini, a 7.2B-parameter mixture-of-experts model with 0.6B active parameters per token, ranked #1 of 36 models in its class worldwide after training on 6T tokens, the company reported. Lara Mini was built as the multilingual model underlying Lara 4 Translation AI, covering 44 languages plus English, including all 24 EU official languages, with a 128K-token context window and a 151,643-token vocabulary. Translated said 39% of raw tokens were filtered out of its deduplicated pool and that no source was repeated more than 2 epochs, arguing that multilingual and multicultural data is \"a source of it\" rather than a cost to intelligence.", "body_md": "Every AI lab assumes more languages make a model worse.We bet the opposite.\n\nBorn as the general-purpose LLM underneath Lara 4 Translation AI, Lara Mini was built as the most multilingual and multicultural model from day one. At the end of its 6T‑token training it ranks #1 of 36 models in its class worldwide.\n\nDiversity isn't a cost to intelligence. It's a source of it.\n\nThe belief\n\nThe big labs train mostly on English and treat other languages and cultures as noise that costs quality.\n\nThe bet\n\nTranslated, whose job is translation, built Lara on real text written in many languages and cultures. Seeing one idea through many of them, the model understands the idea better.\n\nThe proof\n\n#1 of 36 models with up to 2B active parameters worldwide, trained on 6T tokens, 128K context.\n\nThe run at a glance\n\nArchitecture\n\n0.6Bactive parameters per token (4 experts)\n\n7.2Btotal parameters, MoE with 128 experts, 1 shared\n\n34layers: memory (GDN), window and global attention\n\n128Ktokens of context window\n\nData\n\n6Ttraining tokens\n\n39%of raw tokens filtered\n\n44 + ENlanguages, all 24 EU official ones\n\n151,643tokens in vocabulary, low resource rebalanced\n\nHow we built it\n\nA hybrid, sparse design for the most multilingual model\n\nA hybrid, sparse model: 34 layers, 128 experts\n\nThree kinds of layer, one repeating unit\n\nwindowmemorywindowmemoryglobal\n\n14 memory · Gated DeltaNet: a linear-attention state per head, updated token by token\n\n13 window · attention over the last 512 tokens, RoPE positions\n\n7 global · full attention, no positional encoding; the last layer is always global\n\nMixture of experts in every layer\n\n4 of 128 routed experts chosen per token by a router\n\n1 shared expert always on\n\n0.6B of 7.2B parameters active per token; hidden size 1,024, tied input and output embeddings\n\nOptimizer\n\nMuon + AdamW\n\nEach momentum update is orthogonalised (Newton–Schulz) before it is applied, so every direction of a weight matrix moves at a similar rate instead of a few dominating.\n\n4,096-token context window in training; 151,643-token vocabulary.\n\nThe most multilingual and multi-cultural model\n\n6T training tokens from a pool deduplicated across all sources: 39% of raw tokens were filtered out. No source is repeated more than 2 epochs (wiki and high-quality math excepted); plain web is never repeated.\n\nEnglish2.49TWeb HQ 1.34T · Synthetic HQ 0.66T · PDFs 0.32T · Web medium 0.15T · Wiki 0.02T\n\nMultilingual2.18TWeb 0.79T · Web HQ 0.66T · PDFs 0.4T · Synthetic 0.28T · Wiki 0.05T\n\nCode0.77TReal code 0.45T · Synthetic 0.31T\n\nMath0.35TMath HQ 0.21T · Math web 0.15T\n\nSFT-like0.21TQA 0.12T · Math 0.07T · Code 0.02T\n\nThree stages, one learning-rate decay\n\nMain · 4.8T0–80% · to step 572.2k\n\nTransition · 0.6T80–90% · to step 643.7k\n\nAnneal · 0.6T90–100% · to step 715.3k\n\nLearning-rate scheduleEvaluated model: end of training, 6T tokens.\n\nEach stage is a separate data blend loaded at its step boundary. Transition: a bridge between the two at the stable learning rate, where the mix already moves halfway toward the anneal's (more high-quality, QA, code and math data) so the model adapts before the decay. Anneal: the highest-quality mix, while the learning rate decays to zero over the last 10%.\n\nEach stage leans further into code, math and SFT-like data\n\nEnglish\n\nMultilingual\n\nCode\n\nMath\n\nSFT-like (English-only)\n\nMain\n\n44%37%12%\n\nTransition\n\n35.5%35%14.5%8%\n\nAnneal\n\n27%33%17%11%12%\n\nWhole run\n\n41%36%13%\n\nShare of each stage's tokens. The data are shuffled, not fed domain by domain: every batch of 2,048 sequences mixes all domains in about these proportions. SFT-like (QA, math and code instruction data): 2% in main, 3.5% over the whole run.\n\nMethodology\n\n26 benchmarks, one harness, every model\n\nOverall = ¼ English + ¼ Multilingual + ¼ Math + ¼ Code. Every model is scored by us with lm-evaluation-harness and the same answer-format corrections. Each multilingual benchmark is averaged over our 44 non-English training languages.\n\nEnglish10\n\nNLI\n\nPAWS\n\nBelebele (en)\n\nCOPA\n\nPIQA\n\nHellaSwag\n\nWinoGrande\n\nARC-C\n\nMMLU\n\nMMLU-Pro\n\nMultilingual9\n\nXNLI\n\nPAWS-X\n\nBelebele\n\nXCOPA\n\nGlobal PIQA NP\n\nGlobal PIQA P\n\nT-HellaSwag\n\nINCLUDE\n\nGlobal MMLU\n\nMath3\n\nGSM8K\n\nMATH\n\nMGSM\n\nCode4\n\nMBPP\n\nMBPP+\n\nCRUX-I\n\nCRUX-O\n\nNo peeking: the final training stage skips every document that holds a benchmark item.\n\nResults\n\n#1 of 36 in tiny models worldwide\n\nFirst on Overall, Text and Multilingual, 3.1 points ahead of granite-4.0-h-tiny on Overall.\n\nOverall\n\nTiny models ranking\n\n#1\n\nout of 36 models\n\nCategory ranking\n\nMath & Code#1\n\nText#1\n\nMultilingual#1\n\nBenchmark definition. Our model is the final checkpoint of the 6T‑token run, with its context extended to 128K; its anneal skips every document holding a benchmark item. Its class: models with up to 2B active parameters (gemma-4-E2B at its 2B effective size). Multilingual scores are averaged over the 44 non-English languages we train on, for every model.\n\nRankings\n\nTop 12 tiny models\n\n36 models with up to 2B active parameters (gemma-4-E2B at its 2B effective size, 5.1B total). Multilingual on our training languages. Bars start from 0.\n\nOverallEuropeTextMath & Code\n\n#1Lara Mini Base (A0.6B/7.2B)57.0\n\n#2IBM - granite-4.0-h-tiny (1B active)53.9\n\n#3Alibaba - Qwen3-1.7B53.8\n\n#4Falcon-H1-1.5B-Deep53.2\n\n#5Hunyuan-1.8B52.2\n\n#6LFM2.5-8B-A1B (1.7B active)50.8\n\n#7Trinity-Nano (1B active)50.7\n\n#8LFM3-1.7B49.2\n\n#9Qwen3.5-2B48.9\n\n#10Falcon-H1-1.5B48.8\n\n#11Qwen2.5-1.5B47.6\n\n#12granite-4.0-1b47.2\n\n⋯\n\n#17Google - gemma-4-E2B41.9\n\nOverall = ¼ English + ¼ Multilingual + ¼ Math + ¼ Code.\n\n#1Lara Mini Base (A0.6B/7.2B)57.0\n\n#2Bielik-1.5B-v3 (Qwen-based)40.9\n\n#3Hugging Face - SmolLM2-1.7B36.8\n\n#4Luciole-1B31.3\n\n#5Apertus-v1.1-1.5B31.2\n\n#6Kyutai - helium-1-2b30.7\n\n#7EuroMoE-2.6B-A0.6B-2512 (0.5B active)29.4\n\n#8EuroLLM-1.7B27.6\n\n#9Apertus-v1.1-0.5B24.2\n\n+16 points (+39%) over the next European model.\n\nEuropean sovereign models of our class (9), from European labs: up to 2B active parameters. Overall score, Multilingual on our training languages.\n\n#1Lara Mini Base (A0.6B/7.2B)61.5\n\n#2IBM - granite-4.0-h-tiny (1B active)59.0\n\n#3Google - gemma-4-E2B58.3\n\n#4Trinity-Nano (1B active)57.9\n\n#5LFM2.5-8B-A1B (1.7B active)57.1\n\n#6Qwen3-1.7B57.0\n\n#7Qwen3.5-2B56.7\n\n#8Falcon-H1-1.5B-Deep55.2\n\n#9Qwen2.5-1.5B54.7\n\n#10Hunyuan-1.8B53.5\n\n#11LFM3-1.7B53.3\n\n#12Falcon-H1-1.5B53.3\n\nText = mean of English (10 benchmarks) and Multilingual (9 benchmarks, our 44 non-English training languages). Separately: English #2 (66.4), Multilingual #1 (56.7).\n\n#1Lara Mini Base (A0.6B/7.2B)52.5\n\n#2Falcon-H1-1.5B-Deep51.2\n\n#3Hunyuan-1.8B50.9\n\n#4Alibaba - Qwen3-1.7B50.6\n\n#5IBM - granite-4.0-h-tiny (1B active)48.9\n\n#6LFM3-1.7B45.1\n\n#7LFM2.5-8B-A1B (1.7B active)44.4\n\n#8Falcon-H1-1.5B44.4\n\n#9Trinity-Nano (1B active)43.5\n\n#10granite-4.0-1b43.1\n\n#11Qwen3.5-2B41.1\n\n#12Qwen2.5-1.5B40.5\n\n⋯\n\n#26Google - gemma-4-E2B (5.1B total)25.4\n\nMath & Code = mean of Math (GSM8K, MATH, MGSM) and Code (MBPP, MBPP+, CRUX-I, CRUX-O). Math #1 (59.9), Code #5 (45.1).\n\nEfficiency\n\n#1 under 2B, with just 0.6B active\n\nLara Mini Base outscores all 35 other models of its class, including models that use up to 3× more active parameters, such as Qwen3-1.7B and gemma-4-E2B.\n\nOverall score against active parameters, log scale. The step line is the frontier: no model with fewer active parameters scores higher. gemma-4-E2B is drawn at its 2B effective size (5.1B total).\n\nLimited release · who gets it\n\nWhat Lara Mini is for\n\nLara Translate app\n\nOffline Translation\n\nLara Mini is the base model behind Lara 4 offline translation feature for the next Lara Translate app. Your text never leaves the phone, and each translation still takes under a second.\n\nTranslated customers\n\nTiny Base LLM\n\nSelected Translated enterprise customers will have access. Lara Mini is designed to be fine-tuned for specialized sovereign applications, runs on mobile devices and on our cloud delivers an end-to-end latency under 20 ms delivering millions of tokens per sec.\n\nResearch\n\nAI researchers\n\nAccess for AI researchers to reproduce our results and build on them, pushing multilingual research further.\n\nWhat's next\n\nThree competitions, one rule\n\nWe train the next stage only if we win a medal in the current one.\n\nStage 1\n\nLocal\n\nTiny · <2B active\n\nNov 2026Lara Mini Base\n\nWin the tiny category with Lara Base (A0.6B/7.2B).\n\nDec 2026Lara Mini\n\nLara 4 Mini and Lara Mini LLM, for ultra-low latency and mobile devices.\n\nMedal won → we train the National stage. No medal, we stop.\n\nStage 2\n\nNational\n\nSmall · <10B active\n\nMar 2027Lara Piccola Base (A7B/138B)\n\nEnter the small category and win a medal there too.\n\nApr 2027Lara 4 & Lara Piccola LLM\n\nLara 4 translation model and Lara Piccola LLM, an EU-sovereign A7B/138B model.\n\nMedal won → we train the Olympics stage. No medal, we stop.\n\nStage 3\n\nOlympics\n\nFrontier · 40B+ active\n\nDec 2027Lara Grande Base (A60B/1.2T)\n\nEnter the frontier category, against the world's best models.\n\nMar 2028Lara 5 & Lara Grande LLM\n\nLara 5 surpasses P95 pro translators. Lara Grande: an EU-sovereign frontier LLM, ahead of Chinese models and approaching Anthropic and OpenAI.\n\nReference models:tiny Gemma 4 E2B · small Gemma 4B/26B · medium Qwen 17B/397B · frontier GPT-6 Astra, Claude 5.6, Kimi K3.\n\nTraining budget\n\nTraining single-run budgets, next releases\n\nModel\n\nTokens\n\nFLOPs\n\nHardware\n\nDays\n\nTraining cost\n\nLara 4 PiccolaMoE 6.7B / 138B\n\n20T\n\n8.0 × 10²³\n\n512 × H100\n\n31\n\n$1.0M\n\nLara 4 GrandeMoE 60B / 1.2T\n\n24T\n\n8.6 × 10²⁴\n\n512 × Vera Rubin\n\n25\n\n$2.4M\n\nLara 4 SaggiaMoE 600B / 12T\n\n36T\n\n1.3 × 10²⁶\n\n2,304 × Vera Rubin\n\n83\n\n$36.6M\n\nFor reference\n\nTypical frontierMoE 150B / 3T\n\n36T\n\n3.2 × 10²⁵\n\n4,096 × B200\n\n51\n\n$18.8M\n\nSource: Lara 4 LLM Training Budget Planning, 3 Nov 2026 · FP8 · compute cost only · Saggia days estimated with the plan's method\n\nNext training\n\nLara Piccola Candidate\n\n60 layers and 138B parameters: the model to beat Gemma 4 31B.\n\n138Btotal parameters\n\n6.7Bactive per token\n\nTokenizernew, rebalanced\n\n60layers\n\n2816hidden size\n\n512 × 512experts, top-8 + 2 shared\n\n400kH100 GPU-hours for 20T tokens\n\n30 dayson 512 H100 GPUs\n\n$1MTraining run cost\n\nGoal: win the small model class (<10B active)\n\nCurrent leaders: Qwen3.5-122B-A10B, GLM-4.5-Air\n\nGood chance\n\nGoal: beat Gemma 4 31B on multilingual\n\nBeat highly multilingual dense models in the medium model class.\n\nGood chance\n\n30.7 s per step at 32 nodes · same hybrid attention as Lara Mini Base (GDN, window, global)\n\nTechnical report\n\nGet the technical report\n\nArchitecture, training data, benchmarks and methodology behind Lara Mini Base, in one document. Leave your details and we'll email it to you.\n\nFirst name*\n\nLast name*\n\nBusiness email*\n\nCompany name\n\nCompany website\n\nI consent to be contacted by Lara for further information and demo. I have read and agree to the Privacy Policy*\n\nSubmit\n\nWork with Lara\n\nTwo ways to put Lara to work\n\nInterested in the generative model?\n\nSelected Translated enterprise customers will have access. Lara Mini is designed to be fine-tuned for specialized sovereign applications, runs on mobile devices and on our cloud delivers an end-to-end latency under 20 ms delivering millions of tokens per sec.", "url": "https://wpnews.pro/news/lara-translate", "canonical_source": "https://laratranslate.com/research", "published_at": "2026-11-12 15:16:59+00:00", "updated_at": "2026-11-12 15:21:29.806695+00:00", "lang": "en", "topics": ["large-language-models", "machine-learning", "natural-language-processing", "artificial-intelligence"], "entities": ["Translated", "Lara Mini", "Lara 4 Translation AI", "Gated DeltaNet", "Muon", "AdamW"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/lara-translate", "markdown": "https://wpnews.pro/news/lara-translate.md", "text": "https://wpnews.pro/news/lara-translate.txt", "jsonld": "https://wpnews.pro/news/lara-translate.jsonld"}}