A developer achieved 3x better batch throughput with Qwen3.5 by fixing vLLM on AMD's Strix Halo using the Lemonade Server, enabling more efficient AI inference on AMD hardware.…
UC San Diego researchers developed DFlash, a speculative decoding method that uses a lightweight block diffusion model to draft entire token blocks in parallel, achieving up to 6.08x speedup on Qwen3-…
A developer built an 11-chapter interactive guide explaining how LLM inference works, centered around nano-vLLM, a 1,200-line Python reimplementation of the vLLM serving engine. The guide covers algor…
A developer built a production-ready LangGraph ReAct agent that exposes an OpenAI-compatible API, supports multi-model switching via a gateway, and includes one-line tracing with Langfuse. The deploym…
IBM Research, Red Hat, and NxtGen Cloud Technologies demonstrated that using llm-d to serve AI models on mixed GPU hardware can boost inference speeds by 3 to 5 times and double throughput, enabling e…
Developers can deploy Llama 3.1 8B behind vLLM's OpenAI-compatible API on a rented GPU instance, achieving thousands of output tokens per second through continuous batching. The tutorial covers instal…
SkyPilot launched SkyPilot Endpoints, a production-ready LLM inference system that deploys and manages inference across multiple Kubernetes clusters from a single YAML specification. The system handle…
NVIDIA's confidential computing (CC) on Blackwell GPUs introduces performance overheads ranging from 1% to over 100% for inference and 18–32% for training, according to benchmarks from Tinfoil. The pr…
Modelplane, an open-source control plane for AI inference built on Crossplane, is being released to manage GPU clusters as a single inference fleet, handling provisioning, model placement, autoscaling…
VLLM, a large-model inference serving framework, uses Python for control flow but pushes arithmetic into compiled C++ and CUDA kernels to avoid interpreter overhead. The Python/C++ boundary crossing i…
Neuralwatt launched the first AI inference API with energy-based pricing, charging per kilowatt-hour instead of per token to provide transparency into power consumption and cost. The platform offers r…
Local AI serving engines like Ollama and vLLM offer different trade-offs between ease of use and throughput, with Ollama ideal for single users and vLLM for high-concurrency production workloads. The …
A new Local LLM Hardware Calculator helps users estimate memory requirements for running large language models on their own hardware, factoring in weights, KV cache, and overhead. The tool also compar…
A developer confirmed that GLM-5.2, a 753B-parameter DeepSeek-Sparse-Attention MoE model, runs on 8x A100 80GB GPUs using vLLM PR #38476, which adds a Triton sparse-MLA backend for Ampere architecture…
An AI infrastructure engineer at a major industrial company seeks advice on transitioning from SRE-focused work to a proper software engineering role in AI infrastructure, asking for skills, resources…
A developer explains that the KV cache is the biggest operational bottleneck in production LLM serving on GPUs, consuming more memory than model weights for workloads with high concurrency or long con…
Modal Labs released state-of-the-art DFlash speculators for Qwen 3.5 and Qwen 3.6 models on Hugging Face, achieving 5-20% additional speedups and enabling Qwen 3.5 122B-A10B to run at over 1000 tok/s …
A developer building vLLM from source on an AWS g5 instance with Ubuntu 26.04 and Python 3.14 encountered multiple version-skew, driver, and toolchain issues, including a pitfall where missing nvidia-…
A critical bug in Google's Gemma 4 causes it to malform tool calls under real load, affecting vLLM, llama.cpp, Ollama, and oobabooga. A developer open-sourced a diagnosis, repair, and experimental LoR…
OpenCode, an open-source terminal-first coding agent, now supports self-hosted large language models via OpenAI-compatible APIs, enabling users to connect it to a vLLM server running NVIDIA's Nemotron…