Researchers from UC Berkeley and other institutions released CommBench, a benchmark of over 100 GPU communication problems, to evaluate whether large language models can generate correct and efficient…
A developer found that deploying a 70B Llama model with RAG features caused time-to-first-token (TTFT) to jump from 180 ms to 1.4 seconds, as the model recomputed identical attention states for repeat…
The open-source LLM inference landscape in 2025–2026 has split into three specialized tiers: throughput-oriented serving engines for production GPU clusters, portability-focused runtimes for consumer …
NVIDIA's DGX Spark single-node AI appliance can run OpenAI's open-weight GPT-OSS-120B sparse MoE model at ~50 tokens/s using optimized engines like SGLang or llama.cpp, making it viable as a local cod…
A developer released Thaw, an open-source tool that snapshots a live large language model inference session to create forked branches without re-running prefill, addressing the computational waste of …
Liquid AI released LFM2.5-8B-A1B, an edge model designed for fast tool calling on consumer hardware, with a 128K context window and pretraining scaled to 38 trillion tokens. The model, available on Hu…
StepFun released Step 3.7 Flash, a 198B-parameter Mixture-of-Experts vision-language model with 11B active parameters, for enterprise deployment on NVIDIA-accelerated infrastructure. The multimodal AI…
Researchers have developed a stateful inference architecture for multi-agent tool calling that reduces per-turn computational cost from full reprocessing to delta-only updates, achieving 2.1x faster p…
Researchers have developed PAT, an adaptive tensor parallelism method that dynamically reconfigures GPU resource allocation during the generation stage of synchronous RLHF training to address the bott…
A developer has released a drop-in replacement chat template for Qwen/Qwen3.6-27B that fixes six critical bugs affecting open-source agentic coding harnesses. The fork addresses issues including multi…
NVIDIA has open-sourced the Nemotron-Labs Diffusion family of language models (3B, 8B, and 14B parameters), which replace traditional left-to-right autoregressive generation with a parallel denoising …
NVIDIA has released the Nemotron-Labs Diffusion family of open-weight language models (3B, 8B, 14B, and an 8B VLM) that can operate in three generation modes—autoregressive, diffusion, or self-specula…
NVIDIA released Nemotron-Labs-Diffusion, a tri-mode language model that unifies autoregressive, diffusion, and self-speculation decoding within a single architecture. The model, trained with a joint A…
Hippocratic AI partnered with Modular to integrate the MAX framework into its inference pipelines, achieving sub-500ms mean time to first token and approximately 30% faster P99 end-to-end latency for …
Shepherd Model Gateway (SMG) has disaggregated all CPU-bound workloads from GPU inference in large language model serving, moving tokenization, detokenization, and parsing into a dedicated Rust gatewa…