Korean Pre-AGI startup VIDRAFT released Darwin-180B-RSI, a 180-billion-parameter mixture-of-experts reasoning model built on Qwen3.8-Flash-Next that combines selective model merging with recursive sel…
A developer launched ZixcAI, an agentic AI chat interface built on a custom 770-billion-parameter mixture-of-experts model with 80 billion active parameters per token. The system pairs a native PHP an…
Developer Priyanshu-5257 released SplitMoE, an open-source Mixture of Experts routing architecture that splits wide feed-forward layers into smaller, fine-grained expert partitions, published on GitHu…
NVIDIA published LatentMoE in January 2026, a mixture-of-experts architecture that performs expert arithmetic in a space four times narrower than the model's width and spends the saving on a larger ex…
NVIDIA Transformer Engine is now integrated with JAX to accelerate dropless Mixture of Experts (MoE) training, targeting the conditional computation bottleneck in architectures used by DeepSeek, Qwen,…
NVIDIA reported that combining its Transformer Engine library with the JAX Python library raised DeepSeek-V3 MoE training throughput on NVIDIA GB200 GPUs from an unoptimized baseline of 103 TFLOPS/GPU…
A Proof-of-Concept repository published on GitHub under the project jax-softmax-bypass claims to bypass the Transformer Softmax's transcendental exponential function through a static algebraic contrac…
A developer's final part of a CUDA/ROCm optimization series details implementing Mixture of Experts (MoE) inference, including a custom top-k gating kernel, expert parallelism with All-to-All communic…
A developer's guide to self-hosting LLMs on budget VPS plans in 2026 finds that quantized 3B and 7B models are now viable on servers under $6 per month, with RAM bandwidth as the key bottleneck. The a…
Mixture of Experts (MoE) architectures enable 1-trillion-parameter large language models by activating only a subset of expert networks per token, achieving 8× parameter count with only ~2× compute. M…
A new benchmark suite called JAXBench, comprising 50 JAX workloads for AI-generated kernel optimization on Google Cloud TPUs, shows that target-specific documentation context matters more than model s…
Mixture of Experts (MoE) is a machine learning architecture that divides a model into specialized subnetworks called experts, with a gating network routing each input token to only the most relevant s…
Petals, a project from the BigScience research workshop, lets users run large language models like Llama 3.1 (up to 405B parameters) at home by distributing model parts across a peer-to-peer network, …
A new explainer on QAInsights breaks down the Mixture of Experts (MoE) architecture, which powers models like Mixtral, DeepSeek, and GPT-4o, enabling them to scale to trillions of parameters while mai…
Moonshot AI's Kimi K3, a 671-billion-parameter mixture-of-experts model, is the first Chinese open-weight model to match Western frontier AI, achieving benchmark scores competitive with GPT-4o-class m…
Thinking Machines Labs, founded by former OpenAI CTO Mira Murati, released its first open-weight multimodal AI model, Inkling, a 952-billion-parameter Mixture of Experts system that processes both tex…
A developer published a visual and practical guide to large language models, explaining that they are computer programs trained on massive text datasets to predict the next word. The guide covers how …
Tencent released Hunyuan-3, a 295-billion-parameter mixture-of-experts language model that activates only 21 billion parameters per token, designed specifically for agentic tasks such as tool calling …
Open-weight LLMs like Meta's Llama 3, Mistral's Mixtral, and Google's Gemma are democratizing AI access. A developer shows how to integrate these models via API using the OpenAI-compatible spec, with …
Mixture-of-Experts (MoE) models like Qwen3-30B-A3B and DeepSeek-V3 separate total parameters (memory) from active parameters (compute), allowing a 30B-parameter model to run at the speed of a 3B model…