Helion kernels were integrated into vLLM for FP8 inference using Qwen3 models and evaluated across NVIDIA H100 and B200 GPUs. The experiments demonstrated that Helion provides a productive PyTorch-nat…
AMD released Lemonade SDK version 10.7, adding NVIDIA CUDA support to its local AI server solution that previously only supported AMD hardware, Apple Metal GPUs, and AArch64 CPUs. The update integrate…
A buyer who spent roughly $10,000 on both a Mac Studio M3 Ultra with 512 GB of unified memory and a dual NVIDIA DGX Spark setup ran the same 397-billion-parameter model on each machine and measured th…
A high-performance Expert Parallelism (EP) kernel is essential for running large Mixture-of-Experts (MoE) language models across multiple GPUs, as it handles the dynamic routing of tokens to experts l…
Researchers from UC Berkeley and other institutions released CommBench, a benchmark of over 100 GPU communication problems, to evaluate whether large language models can generate correct and efficient…
A developer reports that running gpt-oss-120B on an NVIDIA DGX Spark (GB10) failed due to MXFP4 quantization lacking native hardware support on SM121, causing kernel hangs and memory spikes. Switching…
A developer found that deploying a 70B Llama model with RAG features caused time-to-first-token (TTFT) to jump from 180 ms to 1.4 seconds, as the model recomputed identical attention states for repeat…
Nvidia's upcoming RTX Spark desktop, priced between $3,000 and $5,000, may not deliver the performance leap over the existing DGX Spark that marketing suggests, according to a skeptical analysis from …
A team of developers built Thousand Token Wood, a multi-agent economic simulation where five AI-powered woodland creatures trade goods using a 3-billion-parameter Qwen2.5-3B model. The simulation, cre…
Researchers have released vLLM, a new inference engine designed to efficiently serve large language models by optimizing memory management and batching. The system achieves up to 24x higher throughput…
NVIDIA AI released Dynamo Snapshot, a checkpoint/restore system using CRIU and cuda-checkpoint to eliminate cold-start delays for AI inference workloads on Kubernetes. The system freezes a running inf…
Microsoft released a tutorial demonstrating how to run a browser-use agent workflow using Microsoft Fara in Google Colab. The tutorial guides users through setting up a mock OpenAI-compatible endpoint…
A developer has released LLMhop, a stateless HTTP router that directs OpenAI-compatible API requests to the appropriate LLM inference backend based on the model name in the request body. The single-bi…
NousResearch has released hermes-agent, an open-source adaptive AI agent designed for local execution on consumer GPUs with personalized learning and long-term memory capabilities. PaddlePaddle has ma…
Huawei released KVarN, a native KV-cache quantization back end for vLLM that delivers up to 5x more cache capacity and 1.3x the throughput of FP16 while maintaining FP16-level accuracy. The calibratio…
Google DeepMind released Gemma 4 12B, an encoder-free multimodal model that processes text, images, audio, and video natively without separate vision or audio encoders. The 12-billion-parameter model …
DeepLearning.AI and Red Hat have released a free, intermediate-level course titled "Fast & Efficient LLM Inference with vLLM," taught by Red Hat Senior Developer Advocate Cedric Clyburn. The 1-hour 38…
A Reddit user reported that llama.cpp build b9455 achieved 67-81 tokens per second on a dual RTX 3090 setup running Unsloth's Qwen3.6-27B-UD-Q8_K_XL model, matching the speed of vLLM for multi-GPU inf…
The open-source LLM inference landscape in 2025–2026 has split into three specialized tiers: throughput-oriented serving engines for production GPU clusters, portability-focused runtimes for consumer …
Speculative decoding optimizes LLM inference by using a cheap draft model to predict multiple tokens, which are then verified in a single forward pass of the target model, reducing memory-bandwidth bo…