DeepLearning.AI and Red Hat have released a free, intermediate-level course titled "Fast & Efficient LLM Inference with vLLM," taught by Red Hat Senior Developer Advocate Cedric Clyburn. The 1-hour 38…
A Reddit user reported that llama.cpp build b9455 achieved 67-81 tokens per second on a dual RTX 3090 setup running Unsloth's Qwen3.6-27B-UD-Q8_K_XL model, matching the speed of vLLM for multi-GPU inf…
The open-source LLM inference landscape in 2025–2026 has split into three specialized tiers: throughput-oriented serving engines for production GPU clusters, portability-focused runtimes for consumer …
Speculative decoding optimizes LLM inference by using a cheap draft model to predict multiple tokens, which are then verified in a single forward pass of the target model, reducing memory-bandwidth bo…
A developer released Thaw, an open-source tool that snapshots a live large language model inference session to create forked branches without re-running prefill, addressing the computational waste of …
Researchers have developed Stream2LLM, a system that extends the vLLM inference engine to support concurrent streaming of context to large language models, achieving up to 11x faster time-to-first-tok…
A developer has released tiny-vllm, a high-performance LLM inference engine written in C++ and CUDA that serves as a smaller sibling to the vLLM project. The open-source repository includes both the f…
Arm's Product Security Team has released Metis, an open-source AI security framework that uses large language models to perform deep reasoning and context-aware code reviews for detecting subtle vulne…
Liquid AI released LFM2.5-8B-A1B, an edge model designed for fast tool calling on consumer hardware, with a 128K context window and pretraining scaled to 38 trillion tokens. The model, available on Hu…
NVIDIA's CUTLASS library, a header-only C++ template framework for writing custom CUDA kernels, powers much of the AI infrastructure behind FlashAttention, vLLM, and PyTorch's internal kernels. The li…
NVIDIA has introduced Dynamo Snapshot, a checkpoint/restore approach for AI inference workloads on Kubernetes that reduces cold-start latency from minutes to near-instantaneous startup times. The solu…
A critical vulnerability tracked as CVE-2026-48710 and nicknamed "BadHost" was disclosed in the Starlette ASGI framework, affecting all versions prior to 1.0.1. The flaw allows attackers to manipulate…
Researchers have developed a stateful inference architecture for multi-agent tool calling that reduces per-turn computational cost from full reprocessing to delta-only updates, achieving 2.1x faster p…
A developer who self-hosted open-source AI models on rented A100 GPUs shut down the setup after two weeks and returned to using an API, finding that self-hosting cost at least $500 per month for GPU a…
Hugging Face researchers released a new method called delta weight sync that reduces the per-step weight transfer in asynchronous reinforcement learning from 1.2 GB to as little as 20 MB for a 0.6B pa…
A developer compared the costs of Claude Sonnet 4.6 API at $3.00 per million input tokens against a self-hosted Llama 3.2 90B instance on a $20/month DigitalOcean GPU Droplet. The analysis found that …
A critical vulnerability tracked as CVE-2026-48710 (BadHost) in Starlette versions prior to 1.0.1 allows attackers to bypass path-based authentication middleware by injecting a crafted Host header tha…
Nexus Labs reduced time-to-first-token (TTFT) from 480ms to 110ms on one tenant by enabling vLLM's prefix cache for agent workloads, while another tenant saw no improvement. The discrepancy was caused…
A hidden bottleneck in LLM proxy architectures is causing 5-13 millisecond blocking delays per request during tokenization, a CPU-bound operation that most systems treat as instantaneous. In event-loo…
A developer has released a drop-in replacement chat template for Qwen/Qwen3.6-27B that fixes six critical bugs affecting open-source agentic coding harnesses. The fork addresses issues including multi…