Netflix's AI Platform team published a detailed account of its LLM serving platform, revealing that version pinning between NVIDIA Triton Inference Server and vLLM, a Python GIL bottleneck, and KV-cac…
Prefill-decode disaggregation, now supported by vLLM, SGLang, LLM-d, NVIDIA Dynamo, and TensorRT-LLM, separates LLM inference into compute-bound prefill and memory-bound decode phases on different nod…
LightSeek Org released Shepherd Model Gateway (SMG), an engine-agnostic, high-performance model-routing gateway for large-scale LLM deployments that centralizes worker lifecycle management and balance…
OpenLake, an open-source storage engine for offloading LLM KV caches from GPU memory to RAM and NVMe, cuts GPU time by 48.2% for long-context inference, reducing a 1,169-second workload to 606 seconds…
Baidu released Unlimited-OCR, an MIT-licensed model that can parse dozens of pages of a PDF in a single forward pass without the GPU memory growth typical of transformer-based OCR. The model uses a no…
KTransformers, a Tsinghua-born framework now with 19k GitHub stars, has restructured from a standalone inference system into CPU kernels and placement strategies embedded within SGLang and LLaMA-Facto…
KTransformers, a flexible LLM inference framework, enables deployment of 100B+ parameter models locally on a single RTX 5090 (32GB VRAM) using CPU/GPU heterogeneous computing without quantization. The…
A technical guide to running large language models locally in 2026 provides hardware math, quantization tradeoffs, and benchmarks of five inference engines, with case studies from the author's 16GB Ap…
Hugging Face retired its own Text Generation Inference (TGI) server in December 2025, directing users to SGLang, a tool that began as a UC Berkeley research paper in 2023 and by 2026 was running on ov…
AMD introduced ROCm Infera, an open-source distributed inference reference solution for large-scale deployments, claiming it can improve goodput per GPU for agentic AI workloads by up to 2.6× in inter…
Modelplane, a new open-source control plane for fleet-level AI inference, launched v0.1 to solve the problem of managing inference across hundreds of clusters on any infrastructure. The API separates …
A developer known as workofart has created a PyTorch-like framework for training AI agent harnesses, achieving a 45-minute experiment cycle on terminal bench tasks by enforcing deterministic inference…
Tinfoil announces cached prompt pricing in its Inference API, a feature that reduces compute for eligible requests by caching recently processed inputs, but the company warns that caching introduces t…
Two open-weight coding models, Kimi K2.7 Code from Moonshot AI and GLM-5.2 from Z.ai (Zhipu AI), launched days apart in June 2026, both designed for agentic coding workflows and supporting vLLM and SG…
Thinking Machines Lab, Inc. released Inkling, a general-purpose multimodal model with 975 billion total parameters and 41 billion active parameters, on July 15, 2026 under an Apache 2.0 license. The m…
A new arXiv paper, Ring-Zero: Scaling Zero RL to a Trillion Parameters for Emergent Reasoning, reports a 1T-parameter mixture-of-experts reasoning model trained with reinforcement learning from verifi…
A developer released AI-CLI, a tiny C terminal assistant that connects user requests to a local LLM and executes returned shell actions directly, supporting over 20 platforms and most LLM engines. The…
XGrammar, a grammar-constrained decoding library used by vLLM, SGLang, TensorRT-LLM, and MLC-LLM, masks invalid tokens before the model samples, guaranteeing valid JSON output and eliminating retry lo…
SGLang, an open-source LLM inference engine, introduces RadixAttention for prefix caching and grammar-constrained decoding, achieving 25x throughput improvement over vLLM for structured output tasks. …
Thinking Machines released Inkling, a ~1T-parameter multimodal Mixture-of-Experts LLM with 1M context window and native support for image, audio, and text inputs, now available on Hugging Face. The mo…