Netflix's AI Platform team published a detailed account of its LLM serving platform, revealing that version pinning between NVIDIA Triton Inference Server and vLLM, a Python GIL bottleneck, and KV-cac…
Prefill-decode disaggregation, now supported by vLLM, SGLang, LLM-d, NVIDIA Dynamo, and TensorRT-LLM, separates LLM inference into compute-bound prefill and memory-bound decode phases on different nod…
LightSeek Org released Shepherd Model Gateway (SMG), an engine-agnostic, high-performance model-routing gateway for large-scale LLM deployments that centralizes worker lifecycle management and balance…
A senior engineer with 11 years of distributed systems experience explains the full LLM inference pipeline, from request arrival to text output, detailing the GGUF file structure and the distinction b…
Open-weights models like Llama and Mistral give developers control over the inference stack, enabling custom quantization, KV cache optimization, and hardware-specific tuning that closed APIs cannot m…
A developer has open-sourced UGR, an AI runtime operating system designed to orchestrate inference across commodity hardware by treating models as collections of resources that can move between memory…
Modelplane, a new open-source control plane for fleet-level AI inference, launched v0.1 to solve the problem of managing inference across hundreds of clusters on any infrastructure. The API separates …
Netflix's AI Platform team built an in-house LLM serving stack, running the full pipeline from model deployment through inference inside its existing production environment. The team selected vLLM as …
A developer released AI-CLI, a tiny C terminal assistant that connects user requests to a local LLM and executes returned shell actions directly, supporting over 20 platforms and most LLM engines. The…
XGrammar, a grammar-constrained decoding library used by vLLM, SGLang, TensorRT-LLM, and MLC-LLM, masks invalid tokens before the model samples, guaranteeing valid JSON output and eliminating retry lo…
BareMetalRT launches TensorRT-LLM natively on Windows without WSL, enabling heterogeneous tensor parallelism across consumer GPUs over standard networking. Users can run HuggingFace models locally via…
Microsoft Foundry now offers a curated catalog of Hugging Face open-weight models deployable on Foundry Managed Compute, with pre-staged weights in Azure and built-in enterprise security, governance, …
Speculative decoding accelerates large language model inference by 20-50% without quality loss, using a draft-verify mechanism that generates multiple tokens per forward pass. The technique amortizes …
NVIDIA's full-stack inference software, codesigned with its hardware, has reduced token costs by up to 5x on the DeepSeek V4 model in one month on the Blackwell platform. Companies like Baseten, Cogni…
NVIDIA's DGX Spark, a $4,699 desktop AI supercomputer with 128GB unified memory, launched in 2025, offering one petaflop of FP4 performance. Benchmarks show it excels at prompt processing but lags in …
LLM inference optimization can reduce serving costs by 5-10x and latency by 3-5x, often determining whether an AI feature ships. The bottleneck is memory bandwidth during autoregressive decoding, and …
An engineer provides a formula to estimate KV cache memory consumption for large language models, showing that the KV cache often becomes the bottleneck before model weights. For Llama 3.1 70B at 128K…
The launch of Z.ai's GLM 5.2 and DeepSeek V4 Flash has created a 50x price gap between open-weight APIs and closed frontier models, reshaping the build-versus-buy calculus for developers. While open-w…
UC San Diego researchers developed DFlash, a speculative decoding method that uses a lightweight block diffusion model to draft entire token blocks in parallel, achieving up to 6.08x speedup on Qwen3-…
NVIDIA announced that its full-stack inference and training optimizations, including the GB200 NVL72 rack-scale system and NVIDIA Dynamo software, can maximize AI factory energy efficiency, reducing t…