Speculative decoding optimizes LLM inference by using a cheap draft model to predict multiple tokens, which are then verified in a single forward pass of the target model, reducing memory-bandwidth bo…
A developer released Thaw, an open-source tool that snapshots a live large language model inference session to create forked branches without re-running prefill, addressing the computational waste of …
Researchers have developed Stream2LLM, a system that extends the vLLM inference engine to support concurrent streaming of context to large language models, achieving up to 11x faster time-to-first-tok…
A developer has released tiny-vllm, a high-performance LLM inference engine written in C++ and CUDA that serves as a smaller sibling to the vLLM project. The open-source repository includes both the f…
Arm's Product Security Team has released Metis, an open-source AI security framework that uses large language models to perform deep reasoning and context-aware code reviews for detecting subtle vulne…
Liquid AI released LFM2.5-8B-A1B, an edge model designed for fast tool calling on consumer hardware, with a 128K context window and pretraining scaled to 38 trillion tokens. The model, available on Hu…
NVIDIA's CUTLASS library, a header-only C++ template framework for writing custom CUDA kernels, powers much of the AI infrastructure behind FlashAttention, vLLM, and PyTorch's internal kernels. The li…
NVIDIA has introduced Dynamo Snapshot, a checkpoint/restore approach for AI inference workloads on Kubernetes that reduces cold-start latency from minutes to near-instantaneous startup times. The solu…
A critical vulnerability tracked as CVE-2026-48710 and nicknamed "BadHost" was disclosed in the Starlette ASGI framework, affecting all versions prior to 1.0.1. The flaw allows attackers to manipulate…
Researchers have developed a stateful inference architecture for multi-agent tool calling that reduces per-turn computational cost from full reprocessing to delta-only updates, achieving 2.1x faster p…
A developer who self-hosted open-source AI models on rented A100 GPUs shut down the setup after two weeks and returned to using an API, finding that self-hosting cost at least $500 per month for GPU a…
Hugging Face researchers released a new method called delta weight sync that reduces the per-step weight transfer in asynchronous reinforcement learning from 1.2 GB to as little as 20 MB for a 0.6B pa…
A developer compared the costs of Claude Sonnet 4.6 API at $3.00 per million input tokens against a self-hosted Llama 3.2 90B instance on a $20/month DigitalOcean GPU Droplet. The analysis found that …
A critical vulnerability tracked as CVE-2026-48710 (BadHost) in Starlette versions prior to 1.0.1 allows attackers to bypass path-based authentication middleware by injecting a crafted Host header tha…
Nexus Labs reduced time-to-first-token (TTFT) from 480ms to 110ms on one tenant by enabling vLLM's prefix cache for agent workloads, while another tenant saw no improvement. The discrepancy was caused…
A hidden bottleneck in LLM proxy architectures is causing 5-13 millisecond blocking delays per request during tokenization, a CPU-bound operation that most systems treat as instantaneous. In event-loo…
A developer has released a drop-in replacement chat template for Qwen/Qwen3.6-27B that fixes six critical bugs affecting open-source agentic coding harnesses. The fork addresses issues including multi…
MaxText has introduced new post-training capabilities, specifically Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL), now available on single-host TPU configurations like v5p-8 and v6e-8. …
Running large language model inference servers like vLLM and TGI in production requires specialized observability because they behave differently from standard web services, with key metrics like late…
Amazon SageMaker AI now supports bidirectional streaming for real-time inference, enabling developers to build voice applications that process audio and return transcriptions simultaneously over a sin…