A new open-source project, Expert Sniper, pools ordinary Macs into a network that runs Mixture-of-Experts (MoE) models none of them could run alone, achieving 4.0 tok/s on a 16 GB M4 mini running Qwen…
A developer fine-tuning open vision-language models with supervised fine-tuning and GRPO-style reinforcement learning reports three failure modes where training runs looked healthy but accomplished no…
A software engineer's analysis argues that MoE routing in transformer inference is fundamentally the same problem as CPU branch prediction, and that KV cache management techniques such as prefix cachi…
Token dropping in mixture-of-experts (MoE) layers occurs when a capacity factor limits each expert's buffer, causing excess tokens to bypass the expert MLP and degrade model quality under production l…
Researchers propose EAACD, an expert-aware adaptive contrast decoding method for mixture-of-experts (MoE) models, to mitigate hallucinations in large language models (LLMs). The method leverages disti…
A new weight reordering technique for Mixture-of-Experts (MoE) models optimizes physical file layout to minimize NVMe seek overhead, achieving massive gains in explicit-read inference engines like MLX…
South Korean AI company Upstage released Solar Open 2, an open-weight foundation model optimized for agentic use, featuring a 250B-parameter MoE architecture with 15B active parameters per token and a…
The MiMo-V2.5 model family introduces Hybrid Sliding Window Attention (Hybrid SWA) to reduce KVCache storage to roughly 1/7 of Full Attention, sparse MoE activation to cut per-token compute, and multi…
Lazarus V5 Active Steering protocol, using Grounded Entropy framework, achieves a 146% increase in reasoning scores and 57% reduction in inference latency on quantized MoE models, eliminating the need…
NVIDIA released Nemotron-Labs-TwoTower, an open-weight diffusion language model that separates token representation and denoising into two towers, achieving 2.42× higher generation throughput while re…
Researchers introduced a three-phase deep reinforcement learning system for personalized portfolio management that overcomes ticker lock-in, monolithic objectives, and static user models. Phase 1 uses…
LongCat-2.0, a 1.6 trillion parameter Mixture of Experts (MoE) architecture, introduces a hierarchical routing mechanism and hybrid parallelism to scale model capacity while maintaining deployment fea…
A developer patched llama.cpp to improve prompt processing throughput by 20% when using Multi-Token Prediction (MTP) on GPU, recovering performance lost to MTP overhead. The proof-of-concept processes…
A developer traced a hybrid Mamba-Transformer MoE inference run and found that MoE all-to-all collective stalls dominate the tail latency, with a 69x tail ratio, despite dashboards showing 96% GPU uti…
A new dataset, fineset-io/efficient-llm-papers, compiles 1,734 records of arXiv and Semantic Scholar papers on efficient LLM techniques like quantization, LoRA, MoE, and FlashAttention, each quality-s…
Mixture-of-Experts (MoE) architecture allows large language models to use only a fraction of their total parameters for each token, enabling models with 671 billion total parameters to run at speeds c…
A high-performance Expert Parallelism (EP) kernel is essential for running large Mixture-of-Experts (MoE) language models across multiple GPUs, as it handles the dynamic routing of tokens to experts l…
The Framework Desktop is a 4.5-liter mini tower built around AMD's Ryzen AI Max+ 395 "Strix Halo" chip with up to 128 GB of soldered unified memory, designed specifically for running large local LLMs …
Mixture of Experts (MoE) models are reshaping the economics of open-model deployments by reducing GPU inference costs and altering serving stack requirements. The shift toward MoE architectures in 202…