Prime Intellect released Prime Inference, a serving platform for frontier open-source models that offers serverless endpoints and reserved capacity across multiple datacenters on NVIDIA Blackwell GPUs…
Prime Intellect launched Prime Inference, a serverless and reserved-capacity serving platform for frontier open-source models that processed nearly a trillion tokens per day internally before public r…
UniServe, a serving engine for FastH3 8-Step text-to-video-with-audio generation from the hao-ai-lab, delivers lower median end-to-end latency and 20–46% higher throughput than FastVideo, vLLM-Omni an…
NVIDIA DSX MaxLPS, a policy-governed dynamic power-sharing system, lets AI data centers deploy up to 40% more GPUs within the same approved power budget, according to a joint NVIDIA and Nscale evaluat…
Mixture of Experts has changed the structure of AI inference serving by altering which tensors are active per token, what must remain close together, which transfers need strong local bandwidth, and h…
A September 2026 survey of self-hosted inference orchestrators compared LocalAI, exo, GPUStack, vLLM, Ollama, llama.cpp, LiteLLM, Xinference, NVIDIA Dynamo/llm-d, SkyPilot and dstack across modalities…
Prefill, not decode, has become the dominant bottleneck in RAG and multi-agent workloads, according to an analysis citing NVIDIA's published figures of roughly 30x higher served-request counts for lar…
Cognitora released v0.9.1 in September 2026, adding per-model inflight caps on the gateway, fleet tokens_per_watt gauges, and an energy benchmark harness to its open-source LLM inference orchestration…
NVIDIA Dynamo, an open-source inference framework, supports encode-prefill-decode (EPD) disaggregation to accelerate multimodal model serving, achieving up to 5x faster time to first token (TTFT) and …
NVIDIA announced CUDA Rust, a native GPU programming toolchain for Rust, with two tracks: SIMT (cuda-oxide) and Tile, to mature through 2027. The SIMT track, cuda-oxide, is a custom rustc codegen back…
NVIDIA Dynamo, an open-source inference framework, has been deployed with the vLLM backend to serve chat completion requests in both aggregated and disaggregated configurations. The deployment involve…
Modelplane released version 0.4, adding an NVIDIA Dynamo serving stack that brings gang scheduling and peer-to-peer weight transfer to its inference clusters. The release, built with NVIDIA's Dynamo t…
NVIDIA has released the NVIDIA Inference Xfer Library (NIXL), an open-source library designed to accelerate point-to-point communications in AI inference frameworks such as NVIDIA Dynamo, with support…
NVIDIA introduced shadow engine recovery, a preview feature in NVIDIA Dynamo, that restores LLM inference capacity in seconds by keeping a fully initialized shadow engine idle on the same GPUs, using …
Alibaba released the open weights for Qwen3.8-2.4T-A95B (Qwen3.8-Max), a 2.4T-parameter mixture-of-experts model with 95B activated parameters per token, designed for long-context reasoning and agenti…
Prefill-decode disaggregation, now supported by vLLM, SGLang, LLM-d, NVIDIA Dynamo, and TensorRT-LLM, separates LLM inference into compute-bound prefill and memory-bound decode phases on different nod…
Baseten leads the public GLM-5.2 inference speed table with 602.8 output tokens per second, roughly 31% faster than Blackbox AI at 460.4 tokens per second, according to Artificial Analysis median meas…
CoreWeave announced that its NVIDIA Vera Rubin NVL72 system delivers 10x more tokens per second per megawatt than the NVIDIA GB200 NVL72 on the DeepSeek R1 reasoning model at matched interactivity tar…
Baseten has built the world's fastest API for GLM-5.2, achieving over 280 tokens per second as measured by Artificial Analysis. The performance is driven by optimizations including an updated inferenc…
NVIDIA announced that its full-stack inference and training optimizations, including the GB200 NVL72 rack-scale system and NVIDIA Dynamo software, can maximize AI factory energy efficiency, reducing t…