A buyer who spent roughly $10,000 on both a Mac Studio M3 Ultra with 512 GB of unified memory and a dual NVIDIA DGX Spark setup ran the same 397-billion-parameter model on each machine and measured th…
A high-performance Expert Parallelism (EP) kernel is essential for running large Mixture-of-Experts (MoE) language models across multiple GPUs, as it handles the dynamic routing of tokens to experts l…
Researchers from UC Berkeley and other institutions released CommBench, a benchmark of over 100 GPU communication problems, to evaluate whether large language models can generate correct and efficient…
A developer reports that running gpt-oss-120B on an NVIDIA DGX Spark (GB10) failed due to MXFP4 quantization lacking native hardware support on SM121, causing kernel hangs and memory spikes. Switching…
A developer found that deploying a 70B Llama model with RAG features caused time-to-first-token (TTFT) to jump from 180 ms to 1.4 seconds, as the model recomputed identical attention states for repeat…
Nvidia's upcoming RTX Spark desktop, priced between $3,000 and $5,000, may not deliver the performance leap over the existing DGX Spark that marketing suggests, according to a skeptical analysis from …
A team of developers built Thousand Token Wood, a multi-agent economic simulation where five AI-powered woodland creatures trade goods using a 3-billion-parameter Qwen2.5-3B model. The simulation, cre…
Researchers have released vLLM, a new inference engine designed to efficiently serve large language models by optimizing memory management and batching. The system achieves up to 24x higher throughput…
NVIDIA AI released Dynamo Snapshot, a checkpoint/restore system using CRIU and cuda-checkpoint to eliminate cold-start delays for AI inference workloads on Kubernetes. The system freezes a running inf…
Microsoft released a tutorial demonstrating how to run a browser-use agent workflow using Microsoft Fara in Google Colab. The tutorial guides users through setting up a mock OpenAI-compatible endpoint…
A developer has released LLMhop, a stateless HTTP router that directs OpenAI-compatible API requests to the appropriate LLM inference backend based on the model name in the request body. The single-bi…
NousResearch has released hermes-agent, an open-source adaptive AI agent designed for local execution on consumer GPUs with personalized learning and long-term memory capabilities. PaddlePaddle has ma…
Huawei released KVarN, a native KV-cache quantization back end for vLLM that delivers up to 5x more cache capacity and 1.3x the throughput of FP16 while maintaining FP16-level accuracy. The calibratio…
Google DeepMind released Gemma 4 12B, an encoder-free multimodal model that processes text, images, audio, and video natively without separate vision or audio encoders. The 12-billion-parameter model …
DeepLearning.AI and Red Hat have released a free, intermediate-level course titled "Fast & Efficient LLM Inference with vLLM," taught by Red Hat Senior Developer Advocate Cedric Clyburn. The 1-hour 38…
A Reddit user reported that llama.cpp build b9455 achieved 67-81 tokens per second on a dual RTX 3090 setup running Unsloth's Qwen3.6-27B-UD-Q8_K_XL model, matching the speed of vLLM for multi-GPU inf…
The open-source LLM inference landscape in 2025–2026 has split into three specialized tiers: throughput-oriented serving engines for production GPU clusters, portability-focused runtimes for consumer …
Speculative decoding optimizes LLM inference by using a cheap draft model to predict multiple tokens, which are then verified in a single forward pass of the target model, reducing memory-bandwidth bo…
A developer released Thaw, an open-source tool that snapshots a live large language model inference session to create forked branches without re-running prefill, addressing the computational waste of …
Researchers have developed Stream2LLM, a system that extends the vLLM inference engine to support concurrent streaming of context to large language models, achieving up to 11x faster time-to-first-tok…