A vLLM RFC from the llm-d team warns that disaggregated inference deployments, where prefill and decode run on separate schedulers, can trigger recomputation-based preemption inside the decode instanc…
A Kubernetes engineer's guide compares LLM serving engines, arguing that vLLM is not the only option and that choosing an engine is a critical production decision. The post highlights vLLM's strengths…
A practitioner with 18 months of experience building production agentic systems reports that deterministic guardrails, structured tool interfaces, and trajectory evals matter more than the LLM itself,…
An engineer's benchmarking guide introduces a cache hit rate metric for measuring prefix caching effectiveness in LLM serving, implemented in the open-source tool llmperf-rs. The metric calculates the…
Flash Attention is an algorithm that speeds up training and inference of transformer models by using smart memory management on GPUs. The original version was released in 2022, followed by Flash Atten…
A senior engineer with 11 years of distributed systems experience explains the full LLM inference pipeline, from request arrival to text output, detailing the GGUF file structure and the distinction b…
Google DeepMind's Gemma team partnered with Hugging Face in the Fast Gemma Challenge, achieving a 5x inference speed boost for the Gemma 4 model on a single NVIDIA A10G GPU, with peak performance of 4…
A developer outlines inference optimization strategies for small-to-mid-scale deployments, focusing on KV cache management, quantization, and latency tradeoffs. The post provides practical guidance fo…
Amazon Web Services announced new capabilities for Amazon SageMaker HyperPod, including inference data capture, direct model deployment from community hubs, NVMe storage for reduced latency, and Route…
A new article compares continuous batching and static batching in LLM inference, explaining how techniques in vLLM and TGI improve throughput and reduce latency. The choice of batching strategy affect…
Local AI serving engines like Ollama and vLLM offer different trade-offs between ease of use and throughput, with Ollama ideal for single users and vLLM for high-concurrency production workloads. The …
A developer published token-sec-calc, an open-source Python CLI tool that benchmarks LLM inference throughput, latency, time-to-first-token, and queue wait against any OpenAI-compatible endpoint. The …
Running large language model inference servers like vLLM and TGI in production requires specialized observability because they behave differently from standard web services, with key metrics like late…