vLLM Dashboard: Monitor Token Throughput and Latency

SigNoz published a vLLM monitoring dashboard that tracks token throughput, latency, and KV cache usage for self-hosted vLLM inference servers, requiring SigNoz v0.135.0 or newer and the V2 dashboard schema. The dashboard reads vLLM's native `vllm:*` Prometheus metrics, which the server exposes on the same port as its OpenAI-compatible API without a flag, scraped by the OpenTelemetry Collector and forwarded to SigNoz with no application instrumentation. Panels cover output token throughput derived from `vllm:generation_tokens_total`, time to first token, inter-token latency, end-to-end request latency, request queue time, running versus waiting requests, `vllm:kv_cache_usage_perc`, prefix cache hit rate, and preemption rate, with a model_name variable to filter panels to a single model.

Use this dashboard to watch a self-hosted vLLM inference server. It answers four questions. How many tokens per second does the GPU produce? How long do requests wait for their first token? How deep is the queue? How much of the key-value cache is in use? vLLM exposes its own vllm: metrics in Prometheus format on the same port as its OpenAI-compatible API. It needs no flag to turn them on. The OpenTelemetry Collector scrapes that endpoint and forwards the metrics to SigNoz, so this dashboard needs no instrumentation inside your application. It reads the server, not the code that calls it. Recommended. Uses the V2 dashboard schema https://signoz.io/docs/dashboards/dashboards-v2-api/ and needs SigNoz v0.135.0 or newer. Import it in SigNoz with Dashboards → + New dashboard → Import JSON . Import guide https://signoz.io/docs/dashboards/import-dashboard/ Dashboard Coverage Use these panels to: - Answer "how many tokens per second" : Read the decode rate as a single number, and compare it against the prefill rate next to it. - Separate prefill cost from decode cost : Prompt and generation tokens scale differently, and prompts are the cheaper half to shrink. - Tell a slow first token from a slow stream : Time to first token and inter-token latency fail for different reasons and need different fixes. - Find where a slow request spent its time : Queue time sits next to end-to-end latency. Together they separate a busy server from a long generation. - See saturation before users do : Waiting requests above zero mean the server is at capacity. Preemptions mean it ran out of key-value cache and re-ran work. Metrics Included Token Throughput - Output Token Throughput tokens/s : A rate over vllm:generation tokens total . vLLM publishes no throughput gauge, so this is derived from the counter. - Prompt vs Generation Token Rate : vllm:prompt tokens total and vllm:generation tokens total as per-second rates. Latency All four panels read the .bucket series, because the Collector splits each Prometheus histogram into .bucket , .count , and .sum . - Time to First Token p50 / p95 / p99 : vllm:time to first token seconds.bucket . What users feel as responsiveness. - Inter-Token Latency p50 / p95 / p99 : vllm:inter token latency seconds.bucket . The gap between consecutive output tokens during decode. - End-to-End Request Latency p50 / p95 / p99 : vllm:e2e request latency seconds.bucket . Queueing plus prefill plus decode. - Request Queue Time p50 / p95 / p99 : vllm:request queue time seconds.bucket . Time spent waiting before the request ran. Scheduler and KV Cache - Running vs Waiting Requests : vllm:num requests running against vllm:num requests waiting . - KV Cache Usage : vllm:kv cache usage perc , the fraction of the key-value cache in use. - Prefix Cache Hit Rate : A formula dividing vllm:prefix cache hits total by vllm:prefix cache queries total . vLLM publishes the two counters but no ready-made rate. - Preemption Rate : A rate over vllm:num preemptions total . Preemptions mean the scheduler ran out of key-value cache and re-ran requests, which shows up as latency spikes the token rate does not explain. Dashboard Variables Use this filter variable: - model name : Filter every panel to one model. A single vLLM server can host several models, and each metric carries the model name label.