A developer has open-sourced UGR, an AI runtime operating system designed to orchestrate inference across commodity hardware by treating models as collections of resources that can move between memory…
Modelplane, a new open-source control plane for fleet-level AI inference, launched v0.1 to solve the problem of managing inference across hundreds of clusters on any infrastructure. The API separates …
Netflix's AI Platform team built an in-house LLM serving stack, running the full pipeline from model deployment through inference inside its existing production environment. The team selected vLLM as …
A developer released AI-CLI, a tiny C terminal assistant that connects user requests to a local LLM and executes returned shell actions directly, supporting over 20 platforms and most LLM engines. The…
XGrammar, a grammar-constrained decoding library used by vLLM, SGLang, TensorRT-LLM, and MLC-LLM, masks invalid tokens before the model samples, guaranteeing valid JSON output and eliminating retry lo…
BareMetalRT launches TensorRT-LLM natively on Windows without WSL, enabling heterogeneous tensor parallelism across consumer GPUs over standard networking. Users can run HuggingFace models locally via…
Microsoft Foundry now offers a curated catalog of Hugging Face open-weight models deployable on Foundry Managed Compute, with pre-staged weights in Azure and built-in enterprise security, governance, …
Speculative decoding accelerates large language model inference by 20-50% without quality loss, using a draft-verify mechanism that generates multiple tokens per forward pass. The technique amortizes …
NVIDIA's full-stack inference software, codesigned with its hardware, has reduced token costs by up to 5x on the DeepSeek V4 model in one month on the Blackwell platform. Companies like Baseten, Cogni…
NVIDIA's DGX Spark, a $4,699 desktop AI supercomputer with 128GB unified memory, launched in 2025, offering one petaflop of FP4 performance. Benchmarks show it excels at prompt processing but lags in …
LLM inference optimization can reduce serving costs by 5-10x and latency by 3-5x, often determining whether an AI feature ships. The bottleneck is memory bandwidth during autoregressive decoding, and …
An engineer provides a formula to estimate KV cache memory consumption for large language models, showing that the KV cache often becomes the bottleneck before model weights. For Llama 3.1 70B at 128K…
The launch of Z.ai's GLM 5.2 and DeepSeek V4 Flash has created a 50x price gap between open-weight APIs and closed frontier models, reshaping the build-versus-buy calculus for developers. While open-w…
UC San Diego researchers developed DFlash, a speculative decoding method that uses a lightweight block diffusion model to draft entire token blocks in parallel, achieving up to 6.08x speedup on Qwen3-…
NVIDIA announced that its full-stack inference and training optimizations, including the GB200 NVL72 rack-scale system and NVIDIA Dynamo software, can maximize AI factory energy efficiency, reducing t…
SkyPilot launched SkyPilot Endpoints, a production-ready LLM inference system that deploys and manages inference across multiple Kubernetes clusters from a single YAML specification. The system handle…
NVIDIA announced that DFlash speculative decoding boosts inference performance on Blackwell GPUs by up to 15x for gpt-oss-120b and nearly doubles interactivity for Llama 3.1 8B compared to EAGLE-3. Th…
Modelplane, an open-source control plane for AI inference built on Crossplane, is being released to manage GPU clusters as a single inference fleet, handling provisioning, model placement, autoscaling…
NVIDIA released a workflow for converting FP8-quantized CLIP model checkpoints into TensorRT engines, enabling faster inference and higher GPU throughput for production deployment. The process involve…
The open-source LLM inference landscape in 2025–2026 has split into three specialized tiers: throughput-oriented serving engines for production GPU clusters, portability-focused runtimes for consumer …