EngineCore

A developer's source-code walkthrough of vLLM 0.22's V1 execution path traces a single offline inference request from the LLM.generate() API through inter-process communication, scheduling, GPU execut…

A developer built a small scheduler in Go to understand vLLM's scheduler for LLM inference, tracing each design decision back to its vLLM equivalent. The scheduler operates in a tick loop with three p…

A new paper, FlowPrefill by Hsieh et al., proposes preempting long LLM inference prefills mid-forward-pass to rescue urgent requests that would otherwise miss their time-to-first-token (TTFT) service-…