A new method called domain-conditional position offsets reduces the cold-start penalty in autoregressive language models by adding a single learned vector to the embedding activation at the first sequ…
A hardware analysis reveals that deploying 70-billion parameter models in FP16 requires moving 140 gigabytes of weights across the memory bus per token, creating a memory-bound bottleneck that limits …
A developer published a visual and practical guide to large language models, explaining that they are computer programs trained on massive text datasets to predict the next word. The guide covers how …
Researchers propose a hybrid block diffusion language model (BDLM) combining Mamba and attention mechanisms that restricts reverse Mamba scans to the active denoising block, enabling exact caching acr…
Researchers introduced QuantFlow, a federated Mamba-based foundation model for time-series forecasting that combines inverted sequence embedding, bidirectional state-space decoders, quantile regressio…
A developer built a novel triple-hybrid LLM combining Mamba, Attention, and a 32-expert Mixture of Experts architecture from scratch for approximately $50, completing Titan v1 and the first training c…
NVIDIA released Nemotron, a free open-source AI model designed for reasoning and agentic tasks, which can run locally on personal computers without cloud dependency. The model uses a hybrid Mamba-tran…
Researchers introduced Gaussian Mixture Attention (GMA), a probabilistic attention mechanism that replaces pairwise query-key comparisons with routing through K learned Gaussian mixture components, ac…
A developer traced a hybrid Mamba-Transformer MoE inference run and found that MoE all-to-all collective stalls dominate the tail latency, with a 69x tail ratio, despite dashboards showing 96% GPU uti…
A new dataset, fineset-io/efficient-llm-papers, compiles 1,734 records of arXiv and Semantic Scholar papers on efficient LLM techniques like quantization, LoRA, MoE, and FlashAttention, each quality-s…
Rudrite Research launched a free, open platform offering interactive, animated visual explainers of landmark AI and ML papers, including Attention Is All You Need, GPT-3, and FlashAttention, to make f…
Rudrite Research published a comparison of three post-Transformer sequence models — S4, Mamba, and RWKV — highlighting their approaches to achieving linear computational cost while maintaining quality…
Xiaomi's MiMo team open-sourced MiMo Code, a terminal-based coding agent designed for long-horizon automated programming tasks. The agent addresses challenges in decision quality and state continuity …
Rudrite Research published a comprehensive comparison of AI and ML approaches, covering 14 side-by-side analyses of techniques such as Transformers vs Mamba, FlashAttention vs PagedAttention, and PPO …
NVIDIA released Nemotron 3 Ultra, a 550-billion-parameter open Mixture-of-Experts model designed for long-running AI agents that plan, use tools, and reason across many turns. The model uses a hybrid …
Researchers have developed TopoMamSurv, a Graph Mamba survival analysis framework that uses topology-aware ordering to address computational bottlenecks in Whole Slide Image analysis. The framework in…