{"slug": "end-to-end-observability-for-vllm-and-tgi-from-dcgm-to-tokens", "title": "End-to-End Observability for vLLM and TGI: from DCGM to Tokens", "summary": "Running large language model inference servers like vLLM and TGI in production requires specialized observability because they behave differently from standard web services, with key metrics like latency being multi-dimensional (TTFT, ITL, and end-to-end) and the KV cache in VRAM acting as the primary bottleneck. The article recommends a layered monitoring pipeline that correlates signals from the GPU silicon (via DCGM exporter) up through the inference engine to business metrics, emphasizing that hardware issues like degraded NVLink or NCCL stalls directly impact application performance.", "body_md": "Running large language model inference servers in production exposes gaps that neither stock Prometheus dashboards nor the official documentation of vLLM or TGI cover completely. This article maps the layers that matter, names the exact signals to scrape and flags the traps most teams only hit after real traffic arrives.\n\nAudience: SREs, ML platform engineers and observability engineers who operate or are about to operate vLLM or TGI on GPUs.\n\n## Why LLM serving breaks standard observability\n\nA model server is not a regular web service. Four properties invalidate the usual playbook.\n\n**Latency is not scalar.** Time to first token (TTFT), inter-token latency (ITL) and end-to-end latency tell three different stories. Optimizing one usually degrades another. Prefill-bound workloads (long prompts, short outputs) and decode-bound workloads (chat, agents, RAG) have inverse profiles. A single p99 number is meaningless without saying which latency it refers to and what input distribution produced it.\n\n**Batching is dynamic and preemptive.** Continuous batching schedules in-flight requests into the same forward pass. Throughput rises with batch size up to a point where KV cache pressure forces evictions or swaps. Standard \"queue depth\" metrics still apply, but the relationship between queue depth and tail latency is non-linear and bursty. A queue that looks shallow for ninety seconds and explodes for ten is more useful to detect than a steady moderate queue.\n\n**The KV cache is the real bottleneck.** It lives in VRAM, grows with sequence length and dominates memory pressure. When it fills, vLLM preempts or swaps requests. TGI rejects new arrivals. Neither outcome is visible from CPU or network metrics. The KV cache is the single most informative signal on the engine layer, and it has no equivalent in a stateless web service.\n\n**Hardware reaches into the application.** A degraded NVLink, a thermal throttle or an NCCL all-reduce stall propagates directly to the request queue. The observability stack has to reach down to the silicon or it will produce dashboards that look fine while users wait.\n\nThe right answer is a layered pipeline that correlates a token rendered to a user with what happened on the silicon a few milliseconds earlier.\n\n## Layer map\n\n```\n┌────────────────────────────────────────────────┐\n│ Business and cost (€/token, €/tenant, €/h GPU) │\n├────────────────────────────────────────────────┤\n│ API and distributed tracing (OTel GenAI)       │\n├────────────────────────────────────────────────┤\n│ Inference engine (vLLM, TGI: Prometheus)       │\n├────────────────────────────────────────────────┤\n│ Container and OS (cAdvisor, kubelet, eBPF)     │\n├────────────────────────────────────────────────┤\n│ CUDA runtime and collectives (NCCL, cuPTI)     │\n├────────────────────────────────────────────────┤\n│ GPU silicon (DCGM exporter, NVLink, PCIe)      │\n└────────────────────────────────────────────────┘\n```\n\nEach layer has its own native signals. The value of an end-to-end pile comes from the ability to cross-reference them.\n\n## Layer by layer\n\n### GPU silicon\n\nDCGM exporter is the right entry point. The signals worth wiring up from day one:\n\n| DCGM metric | What it actually says |\n|---|---|\n`DCGM_FI_DEV_GPU_UTIL` |\nCoarse indicator. Reaches 100 % for badly vectorized kernels. Do not use alone. |\n`DCGM_FI_PROF_SM_ACTIVE` |\nFraction of cycles where at least one warp is active on an SM. |\n`DCGM_FI_PROF_SM_OCCUPANCY` |\nAverage warps active per SM normalized to the maximum. |\n`DCGM_FI_PROF_PIPE_TENSOR_ACTIVE` |\nFraction of cycles the tensor cores are working. The real utilization signal for LLM inference. |\n`DCGM_FI_PROF_PIPE_FP16_ACTIVE` , `_FP32_ACTIVE`\n|\nPipeline activity by precision. Useful to spot fallbacks. |\n`DCGM_FI_PROF_DRAM_ACTIVE` |\nHBM traffic. Identifies memory-bound workloads. |\n`DCGM_FI_DEV_FB_USED` , `_FB_FREE`\n|\nVRAM in use and free. Cross with `vllm:gpu_cache_usage_perc` . |\n`DCGM_FI_PROF_NVLINK_RX_BYTES` , `_TX_BYTES`\n|\nInter-GPU traffic. Essential under tensor parallelism. |\n`DCGM_FI_PROF_PCIE_RX_BYTES` , `_TX_BYTES`\n|\nGPU to host traffic. Surfaces pressure during model loading and CPU paging. |\n`DCGM_FI_DEV_POWER_USAGE` , `_GPU_TEMP` , `_MEMORY_TEMP`\n|\nPower and thermal. Throttling shows up here before it shows up in user latency. |\n`DCGM_FI_DEV_SM_CLOCK` , `_MEM_CLOCK`\n|\nEffective clocks. A persistent drop is the first sign of thermal throttling. |\n\nDCGM exporter ships as a Helm chart and runs as a DaemonSet on GPU nodes. Default scrape interval is one second, fine for steady-state dashboards but coarse enough to miss sub-second incidents like an eviction storm. Two profiles in production:\n\n-\n`steady`\n\n: 5 seconds, full field set. -\n`incident`\n\n: 250 ms, reduced field set, enabled on alert.\n\nA few hardware notes that change what you should monitor:\n\n-\n**MIG**(Multi-Instance GPU). When MIG slices are active, DCGM exposes per-slice metrics under the same field IDs with a different device label. Pin labels in your relabel config or you will see metrics merge or vanish across reschedules. -\n**NVSwitch**(DGX, HGX). Add the NVSwitch exporter alongside DCGM. NVLink saturation at the switch is invisible from the per-GPU NVLink counters alone. -\n**InfiniBand**. Use the Mellanox`ibutils`\n\nexporter or`ucx`\n\ncounters. RDMA traffic for distributed inference does not appear in the GPU metrics path.\n\n### CUDA runtime and collectives\n\nTensor parallelism and pipeline parallelism rely on NCCL. When one GPU waits for its peers, application latency shows anomalies with no CPU or network cause visible.\n\nSources worth wiring:\n\n-\n`NCCL_DEBUG=WARN`\n\nin production with parseable output, ingested as structured logs.`INFO`\n\nis too verbose and has a non-trivial overhead. -\n`nvidia-nccl-exporter`\n\nwhere the version supports your CUDA stack. - cuPTI for kernel-level and collective-level tracing. Enable on demand only, the overhead is measurable and biases what you are trying to observe.\n- On InfiniBand fabric, export UCX counters and SHARP statistics. NCCL alone does not surface fabric congestion.\n\nCollective patterns to remember when reading dashboards:\n\n-\n**All-reduce** dominates tensor-parallel matmul splits. Saturated NVLink with idle SMs means you are bandwidth-bound on the collective. -\n**All-gather** appears in some attention implementations and in pipeline-parallel weight gathering. -\n**Send/recv** dominates pipeline parallelism. Imbalance between stages shows up as one GPU with low SM activity and a long send wait.\n\nThese traces are not meant to be on all the time. Continuous lightweight counters with on-demand deep tracing is the pattern that scales.\n\n### Container and OS\n\nPlatform layer:\n\n- cAdvisor and kubelet for pod CPU, RAM and IO.\n- kube-state-metrics for Pod state, OOM events and restarts.\n-\n`kube_pod_info`\n\njoined to GPU identity (`nvidia.com/gpu`\n\ndevice id) to map pod to physical GPU.\n\nKernel layer:\n\n- eBPF via Tetragon, bpftrace or Pixie for syscalls, unexpected network egress and model file reads.\n- On-CPU profiling via parca or pyroscope without instrumenting the binary.\n\neBPF is also where the security observability lives. A minimal Tetragon policy that watches model file reads and unexpected egress on the inference pod:\n\n```\napiVersion: cilium.io/v1alpha1\nkind: TracingPolicy\nmetadata:\n  name: vllm-runtime-watch\nspec:\n  podSelector:\n    matchLabels:\n      app: vllm\n  kprobes:\n    - call: \"security_file_open\"\n      syscall: false\n      args:\n        - index: 0\n          type: \"file\"\n      selectors:\n        - matchArgs:\n            - index: 0\n              operator: \"Prefix\"\n              values:\n                - \"/models/\"\n                - \"/root/.cache/huggingface/\"\n          matchActions:\n            - action: Post\n    - call: \"tcp_connect\"\n      syscall: false\n      args:\n        - index: 0\n          type: \"sock\"\n      selectors:\n        - matchArgs:\n            - index: 0\n              operator: \"NotDAddr\"\n              values:\n                - \"10.0.0.0/8\"\n                - \"127.0.0.0/8\"\n          matchActions:\n            - action: Post\n```\n\nThis is a starter: it logs every model file read and every non-RFC1918 outbound connection from vLLM pods. Convert to alerts only after a quiet-period baseline.\n\n### Inference engine\n\nThe layer most teams neglect the longest, while being the densest in business signal.\n\n**vLLM** exposes `/metrics`\n\nby default. The base set:\n\n| Metric | Type | Reading |\n|---|---|---|\n`vllm:time_to_first_token_seconds` |\nhistogram | Server-side TTFT. Compare to gateway TTFT. |\n`vllm:time_per_output_token_seconds` |\nhistogram | ITL. What the user feels in streaming. |\n`vllm:e2e_request_latency_seconds` |\nhistogram | Server-side end-to-end latency. |\n`vllm:num_requests_running` |\ngauge | Requests in the active batch. |\n`vllm:num_requests_waiting` |\ngauge | Queue depth. First saturation indicator. |\n`vllm:num_requests_swapped` |\ngauge | Requests paged to CPU. VRAM pressure. |\n`vllm:gpu_cache_usage_perc` |\ngauge | KV cache occupation. At 1.0 with `swapped > 0` , you are in eviction territory. |\n`vllm:num_preemptions_total` |\ncounter | Cumulative preemptions. Take the per-second rate. |\n`vllm:prompt_tokens_total` |\ncounter | Input tokens processed. |\n`vllm:generation_tokens_total` |\ncounter | Generated tokens. Cost calculation base. |\n\nRecent vLLM versions also expose prefix caching and speculative decoding metrics. The exact names depend on the version, but the families to look for:\n\n-\n`vllm:gpu_prefix_cache_hits_total`\n\n,`vllm:gpu_prefix_cache_queries_total`\n\n. Hit rate dominates the gain from prefix caching in agent and RAG workloads. - Speculative decoding counters that let you derive the acceptance rate of the draft model. If acceptance falls below the break-even point against the draft model overhead, spec decode is costing you throughput.\n\n**TGI** exposes `/metrics`\n\nwith a different naming convention:\n\n| Metric | Reading |\n|---|---|\n`tgi_batch_current_size` |\nActive batch size. |\n`tgi_batch_next_size` |\nNext batch being formed. |\n`tgi_queue_size` |\nQueue depth. |\n`tgi_request_queue_duration` |\nTime in queue. |\n`tgi_request_inference_duration` |\nEngine time. |\n`tgi_batch_inference_duration` |\nPer-batch latency, decomposable into forward and decode. |\n`tgi_request_input_length` , `tgi_request_generated_tokens`\n|\nToken counters per request. |\n\nBoth engines emit histograms with standard Prometheus buckets. Quantiles are computed at query time (`histogram_quantile`\n\nin PromQL or VMQL equivalents).\n\nA practical reading habit: never look at a single engine metric in isolation. The useful patterns are paired.\n\n-\n`vllm:num_requests_waiting`\n\nrising with`vllm:gpu_cache_usage_perc`\n\nat 1.0 and`vllm:num_preemptions_total`\n\nrate > 0: you are in cache thrash. Reduce`max_num_seqs`\n\nor raise`max_num_batched_tokens`\n\n. -\n`vllm:num_requests_waiting`\n\nrising with healthy cache: you are compute-bound. Add capacity or reduce`max_num_batched_tokens`\n\n. -\n`tgi_queue_size`\n\nhigh with`tgi_batch_current_size`\n\nplateauing below maximum: scheduler is starving on token budget. Inspect`max_batch_total_tokens`\n\n.\n\n### API and distributed tracing\n\nTracing answers \"where did my request spend its time\" independently of aggregate metrics.\n\nAdopt OpenTelemetry with the GenAI semantic conventions:\n\n-\n`gen_ai.system`\n\n(for example`vllm`\n\n,`tgi`\n\n), -\n`gen_ai.operation.name`\n\n(`chat`\n\n,`completion`\n\n), -\n`gen_ai.request.model`\n\n, -\n`gen_ai.request.max_tokens`\n\n,`temperature`\n\n,`top_p`\n\n, -\n`gen_ai.usage.input_tokens`\n\n,`gen_ai.usage.output_tokens`\n\n, -\n`gen_ai.response.finish_reasons`\n\n.\n\nA useful span breakdown:\n\n```\nhttp.server.request\n└── gen_ai.completion\n    ├── tokenize\n    ├── schedule\n    ├── prefill\n    ├── decode  (loop, span per batch step)\n    ├── detokenize\n    └── stream_out\n```\n\nAvailable instrumentation libraries: OpenLIT, openllmetry (Traceloop) and OpenInference (Arize). Pick one and stick to it. Mixing them produces inconsistent attribute names that break dashboard queries.\n\nThe `request_id`\n\npropagated from the ingress through to the engine is the key that makes downstream correlation possible. Declare it at the ingress (header `x-request-id`\n\n), propagate it through OTel baggage, log it on the engine side and attach it as a trace attribute.\n\n**Prometheus exemplars** are worth the configuration cost. They link a histogram bucket to one or more traces, so a click on a TTFT p99 spike in Grafana jumps directly to the slowest traces. vLLM does not expose exemplars natively today, but the OTel collector can attach trace IDs to scraped histograms via the `spanmetrics`\n\nconnector. Sample collector snippet:\n\n```\nconnectors:\n  spanmetrics:\n    histogram:\n      explicit:\n        buckets: [10ms, 50ms, 100ms, 250ms, 500ms, 1s, 2s, 5s]\n    dimensions:\n      - name: gen_ai.system\n      - name: gen_ai.request.model\n      - name: tenant\n    exemplars:\n      enabled: true\n```\n\nThis gives you metric-to-trace navigation without changing the engine code.\n\n### Logs\n\nStructured, JSON. VictoriaLogs handles the volume without forcing a complex query syntax.\n\nMinimum fields for the inference layer:\n\n-\n`request_id`\n\n, -\n`tenant`\n\n, -\n`model`\n\n, -\n`prompt_tokens`\n\n,`generation_tokens`\n\n, -\n`ttft_ms`\n\n,`e2e_ms`\n\n, -\n`finish_reason`\n\n, -\n`gpu_id`\n\n(resolved at pod level), -\n`trace_id`\n\n,`span_id`\n\n(for cross-reference with traces).\n\nDo not log prompts and outputs by default. If you need to, allocate a separate channel with short retention and active PII filtering. The legal exposure of an unfiltered prompt log dwarfs any operational benefit.\n\n### Business and cost\n\nThe only layer that talks to leadership. From the native counters you derive three indicators.\n\n**Cost per request, per tenant, per model.** The denominator changes the answer, surface all three.\n\n**Hourly cost of a GPU normalized by tokens produced in the same window.** This is the closest thing to a useful efficiency metric.\n\n**Useful tokens over billed tokens.** A measure of batching efficiency: how many tokens you produce per token of GPU compute time.\n\nCost per tenant, in PromQL:\n\n```\nsum by (tenant) (\n  rate(vllm:generation_tokens_total{tenant=~\".+\"}[5m])\n)\n* on(model) group_left\n  cost_per_generation_token_eur\n```\n\nWhere `cost_per_generation_token_eur`\n\nis a reference series pushed by a configuration job. Maintain prompt vs generation rates separately, they price differently in most providers and they have different production costs (prefill is single forward pass, decode is autoregressive).\n\nA useful refinement is to include idle cost. A GPU running at 30 % utilization still costs the full hourly rate. The \"effective cost per token\" should distribute the full GPU hour over the tokens actually produced:\n\n```\n(gpu_hourly_cost_eur)\n/\n(sum by (gpu) (rate(vllm:generation_tokens_total[1h])) * 3600)\n```\n\nThis is the number that drives capacity decisions, not the marginal cost per token.\n\n## The hard problems\n\n### Cross-layer correlation\n\nLinking a rendered token to a physical GPU is trivial in theory and hard in practice. The concrete plumbing:\n\n-\n`request_id`\n\npropagated from ingress through engine spans. - Engine-side spans carry\n`gpu_id`\n\nas an attribute. - Metric series carry\n`pod`\n\nand`gpu_uuid`\n\nlabels, joined via`kube_pod_info`\n\nto a`pod`\n\nto`gpu_uuid`\n\nmapping (DCGM exposes`UUID`\n\nand`device`\n\nlabels). - Dashboards join temporally on time windows and spatially on\n`gpu_uuid`\n\n.\n\nDCGM samples per GPU, not per request. Fine-grained correlation is always done by time window, never by exact identifier. The illusion of per-request hardware metrics is exactly that, an illusion.\n\n### Cardinality\n\nLabeling by `tenant`\n\nand `model`\n\nis healthy. Labeling by `user_id`\n\n, `session_id`\n\nor `request_id`\n\non metrics is forbidden. Those dimensions belong to traces and logs.\n\nVictoriaMetrics absorbs moderate cardinality well, especially with `vmagent`\n\nstream aggregation pre-rolling histograms. But multi-tenant inference explodes fast. Run the math at design time:\n\n```\ntenants × models × quantiles × histogram_buckets × instances\n```\n\nTen tenants, five models, six quantiles, ten buckets, fifty instances gives 150 000 series for one histogram metric alone. Add three histograms (TTFT, ITL, e2e) and you are at half a million series before counters and gauges. Plan accordingly or use stream aggregation to drop unused dimensions before storage.\n\n### Sampling\n\nThree rhythms coexist: DCGM at 1 s, vLLM at 10 s, traces sometimes at 1 in 100. For brief incidents (preemption bursts, KV eviction storms), prepare:\n\n- OTel collector with tail-based sampling, rule \"if error or slow then keep\",\n- DCGM in incident mode at 250 ms, switched on by an alert webhook,\n- eBPF in continuous collection on critical syscalls (no sampling, the overhead is minimal),\n- vLLM kept at 10 s, no faster path exists without patching.\n\nA tail-based sampling policy that works in practice:\n\n```\ntail_sampling:\n  decision_wait: 10s\n  policies:\n    - name: errors\n      type: status_code\n      status_code: { status_codes: [ERROR] }\n    - name: slow_ttft\n      type: latency\n      latency: { threshold_ms: 2000 }\n    - name: high_value_tenant\n      type: string_attribute\n      string_attribute:\n        key: tenant\n        values: [enterprise_a, enterprise_b]\n    - name: baseline\n      type: probabilistic\n      probabilistic: { sampling_percentage: 5 }\n```\n\nThis keeps every error, every slow TTFT, every trace from high-value tenants and a 5 % baseline of normal traffic.\n\n### Time origin\n\nServer-side TTFT is not what the user feels. Streaming, proxy buffering, HTTP buffer flushes and WAN traversal all change the perceived value. Measure also:\n\n- gateway-side TTFT (Envoy\n`upstream_rq_time`\n\nor equivalent), - client-side TTFT where possible (SDK instrumentation).\n\nWithout these, you optimize a number that does not reflect the experience. The gap between engine TTFT and gateway TTFT is also a useful health signal in itself, a sudden divergence usually means a proxy buffering regression.\n\n### SLO design for LLM serving\n\nStandard SRE SLO patterns need adjustment for LLM serving. A defensible starting set:\n\n| SLO | Definition | Why |\n|---|---|---|\n| TTFT availability | p95 TTFT below threshold over rolling window | Streaming UX collapses without it. |\n| ITL stability | p95 ITL below threshold | Decode stalls feel worse than a long initial wait. |\n| Completion success | success rate of requests that produce at least one token | Hard failure metric. |\n| Streaming completeness | percentage of streams that emit `finish_reason=stop` (not `length` , not `error` ) |\nQuality proxy. |\n| Capacity headroom | p95 queue depth below a threshold | Forward-looking, drives autoscaling. |\n\nThe thresholds depend on the model and workload. Chat: TTFT p95 under 1 s, ITL p95 under 80 ms. RAG: TTFT p95 under 3 s, ITL p95 under 50 ms (long outputs amplify ITL). Code completion: TTFT p95 under 500 ms, ITL p95 under 30 ms.\n\nExpress them as multi-window multi-burn-rate alerts on the underlying SLI series, not as single-threshold alerts. The Google SRE workbook formulas apply unchanged.\n\n## Reference pile\n\nComponents:\n\n| Role | Recommended | Alternative |\n|---|---|---|\n| Metrics | VictoriaMetrics cluster with vmagent | Prometheus with Thanos or Mimir |\n| Logs | VictoriaLogs | Loki, OpenSearch |\n| Traces | Tempo, Jaeger | SaaS (Honeycomb, Datadog) |\n| Application collection | OTel collector (agent and gateway) | Vector, Fluent Bit |\n| GPU collection | DCGM exporter (DaemonSet) | nvidia_gpu_exporter (legacy) |\n| eBPF | Tetragon, Pixie | Falco |\n| Visualization | Grafana | Perses |\n\n### OTel collector pipeline (agent)\n\n```\nreceivers:\n  prometheus:\n    config:\n      scrape_configs:\n        - job_name: vllm\n          scrape_interval: 10s\n          static_configs:\n            - targets: ['localhost:8000']\n        - job_name: dcgm\n          scrape_interval: 5s\n          static_configs:\n            - targets: ['localhost:9400']\n  otlp:\n    protocols:\n      grpc:\n        endpoint: 0.0.0.0:4317\n\nprocessors:\n  batch:\n    timeout: 5s\n  k8sattributes:\n    auth_type: serviceAccount\n    extract:\n      metadata:\n        - k8s.pod.name\n        - k8s.namespace.name\n        - k8s.node.name\n      labels:\n        - tag_name: app\n          key: app\n          from: pod\n        - tag_name: tenant\n          key: tenant\n          from: pod\n  resource:\n    attributes:\n      - key: deployment.environment\n        value: prod\n        action: upsert\n  tail_sampling:\n    decision_wait: 10s\n    policies:\n      - name: errors\n        type: status_code\n        status_code: { status_codes: [ERROR] }\n      - name: slow\n        type: latency\n        latency: { threshold_ms: 2000 }\n      - name: baseline\n        type: probabilistic\n        probabilistic: { sampling_percentage: 5 }\n\nconnectors:\n  spanmetrics:\n    histogram:\n      explicit:\n        buckets: [10ms, 50ms, 100ms, 250ms, 500ms, 1s, 2s, 5s, 10s]\n    dimensions:\n      - name: gen_ai.system\n      - name: gen_ai.request.model\n      - name: tenant\n    exemplars:\n      enabled: true\n\nexporters:\n  prometheusremotewrite:\n    endpoint: http://vmagent.observability.svc:8429/api/v1/write\n    external_labels:\n      cluster: prod-eu-west\n  otlp/tempo:\n    endpoint: tempo.observability.svc:4317\n    tls:\n      insecure: true\n\nservice:\n  pipelines:\n    metrics:\n      receivers: [prometheus, spanmetrics]\n      processors: [batch, k8sattributes, resource]\n      exporters: [prometheusremotewrite]\n    traces:\n      receivers: [otlp]\n      processors: [batch, k8sattributes, resource, tail_sampling]\n      exporters: [otlp/tempo, spanmetrics]\n```\n\nThe `spanmetrics`\n\nconnector turns traces into low-cardinality histograms with exemplars, giving you click-through from metrics to traces without changing engine code.\n\n### Useful starter queries\n\nTTFT p99 by model:\n\n```\nhistogram_quantile(0.99,\n  sum by (model, le) (\n    rate(vllm:time_to_first_token_seconds_bucket[5m])\n  )\n)\n```\n\nPreemptions per second overlaid with cache occupation:\n\n```\nrate(vllm:num_preemptions_total[1m])\n```\n\nEffective tensor core utilization per GPU:\n\n```\navg by (gpu) (DCGM_FI_PROF_PIPE_TENSOR_ACTIVE)\n```\n\nTokens per GPU-second (efficiency):\n\n```\nsum by (gpu) (rate(vllm:generation_tokens_total[5m]))\n/\ncount by (gpu) (DCGM_FI_DEV_GPU_UTIL)\n```\n\nNormalized TGI queue pressure:\n\n```\ntgi_queue_size / on(instance) tgi_batch_current_size\n```\n\nCost per hour per tenant:\n\n```\nsum by (tenant) (\n  rate(vllm:generation_tokens_total[1h]) * 3600\n) * on(model) group_left cost_per_generation_token_eur\n```\n\n## Alerting that does not lie\n\nAlerts on inference servers should fire on user-visible degradation, not on resource thresholds. A working starter set:\n\n**TTFT burn-rate (multi-window).**\n\n```\n- alert: VLLMTTFTBudgetFastBurn\n  expr: |\n    (\n      sum by (model) (rate(vllm:time_to_first_token_seconds_bucket{le=\"1.0\"}[5m]))\n      /\n      sum by (model) (rate(vllm:time_to_first_token_seconds_count[5m]))\n    ) < 0.95\n    and\n    (\n      sum by (model) (rate(vllm:time_to_first_token_seconds_bucket{le=\"1.0\"}[1h]))\n      /\n      sum by (model) (rate(vllm:time_to_first_token_seconds_count[1h]))\n    ) < 0.95\n  for: 2m\n  labels:\n    severity: page\n```\n\n**Cache thrash detector.**\n\n```\n- alert: VLLMCacheThrash\n  expr: |\n    vllm:gpu_cache_usage_perc > 0.95\n    and\n    rate(vllm:num_preemptions_total[2m]) > 0.5\n  for: 5m\n  labels:\n    severity: ticket\n```\n\n**Tensor core idle under load.**\n\n```\n- alert: GPUTensorIdleUnderLoad\n  expr: |\n    avg_over_time(DCGM_FI_PROF_PIPE_TENSOR_ACTIVE[10m]) < 0.2\n    and\n    vllm:num_requests_running > 4\n  for: 10m\n  labels:\n    severity: ticket\n```\n\nThis last alert catches the case where the engine reports work in flight but the tensor cores are idle. The usual cause is a stalled NCCL collective or a CPU-bound bottleneck before the GPU.\n\n**Streaming completion regression.**\n\n```\n- alert: VLLMStreamingTruncations\n  expr: |\n    (\n      sum by (model) (rate(vllm:request_success_total{finish_reason=\"length\"}[10m]))\n      /\n      sum by (model) (rate(vllm:request_success_total[10m]))\n    ) > 0.1\n  for: 15m\n  labels:\n    severity: ticket\n```\n\nWhen more than 10 % of requests stop on `length`\n\n, either `max_tokens`\n\nis too low for the use case or quality has regressed.\n\nAvoid alerting directly on queue depth or GPU utilization. Both vary widely under healthy load. They are diagnostic, not actionable.\n\n## Anti-patterns\n\nTo review every quarter:\n\n- Treating\n`DCGM_FI_DEV_GPU_UTIL`\n\nas utilization. The right read is`DCGM_FI_PROF_PIPE_TENSOR_ACTIVE`\n\n. - Tuning batching against mean latency. Tail latency and queue depth tell the truth.\n- Labeling metrics by\n`request_id`\n\n. That belongs to traces. - Measuring latency only at the engine. Add the gateway, add the client where possible.\n- Capturing prompts and outputs in traces without an active PII filter.\n- Counting \"tokens\" without separating prompt and generation. Pricing is asymmetric, batching capacity is asymmetric.\n- Leaving cuPTI and\n`NCCL_DEBUG=INFO`\n\non in production. Measurable overhead, biased measurements. - Sampling traces uniformly. Tail-based sampling with rules for errors, slow requests and high-value tenants catches more value at lower volume.\n- Storing everything at maximum resolution. Cardinality cost explodes before retention cost.\n- Building alerts on resource thresholds. Alert on user-visible SLOs, treat resource metrics as diagnostic.\n\n## Maturity ladder\n\nWhere teams typically stand and where to move next.\n\n**Level 0: nothing specific.** Generic node and pod metrics. No idea how the engine is doing. Move to level 1 by scraping the engine's `/metrics`\n\n.\n\n**Level 1: engine metrics only.** vLLM or TGI metrics scraped, basic dashboard. Sufficient for an initial deployment, blind to hardware-rooted issues. Move to level 2 by adding DCGM and pod-to-GPU mapping.\n\n**Level 2: engine plus GPU correlated.** Most pragmatic teams stop here. Resolves 70 % of incidents in practice. Move to level 3 when multi-tenant pressure starts and when latency complaints exceed throughput complaints.\n\n**Level 3: distributed tracing with GenAI semconv.** Per-request visibility, exemplar-driven debugging, tenant-aware SLOs. Required at scale. Move to level 4 for regulated workloads and HPC fabrics.\n\n**Level 4: kernel and fabric depth.** eBPF policies in alerting paths, NCCL and InfiniBand observability, audit-grade logging with retention policies, confidential computing where applicable. Required for regulated industries, sovereign deployments and large-scale training-adjacent serving.\n\nMove one level at a time. Skipping levels produces dashboards no one trusts.\n\n## Where to go next\n\nThree topics deserve their own articles:\n\n- KV cache observability: eviction, fragmentation, swap. Native metrics, stress experiments, mitigations.\n- NCCL and tensor parallelism: observing inter-GPU flows and finding the collective that stalls the batch.\n- Securing an inference server: attack surface, eBPF detection, sandboxing, AI Act audit trail.\n\nThe right implementation order in production:\n\n- Inference engine metrics (vLLM, TGI native scrape).\n- GPU metrics (DCGM exporter).\n- Distributed tracing with OTel GenAI semconv.\n- Structured logs with\n`trace_id`\n\nand`request_id`\n\n. - Business and cost layer.\n- eBPF policies for security and runtime observability.\n- NCCL and cuPTI on demand for hard-to-reproduce issues.\n\nStarting with layers 1 and 2 alone resolves most of the incidents observed in production. Everything above that compounds value once the base is solid.\n\n*Corrections and operational war stories welcome.*", "url": "https://wpnews.pro/news/end-to-end-observability-for-vllm-and-tgi-from-dcgm-to-tokens", "canonical_source": "https://dev.to/samuel_desseaux_815f9c463/end-to-end-observability-for-vllm-and-tgi-from-dcgm-to-tokens-4fbj", "published_at": "2026-05-21 11:37:13+00:00", "updated_at": "2026-05-21 12:04:16.067053+00:00", "lang": "en", "topics": ["large-language-models", "machine-learning", "artificial-intelligence", "developer-tools", "data"], "entities": ["vLLM", "TGI", "DCGM", "Prometheus", "GPU", "KV cache", "SRE", "ML platform engineers"], "alternates": {"html": "https://wpnews.pro/news/end-to-end-observability-for-vllm-and-tgi-from-dcgm-to-tokens", "markdown": "https://wpnews.pro/news/end-to-end-observability-for-vllm-and-tgi-from-dcgm-to-tokens.md", "text": "https://wpnews.pro/news/end-to-end-observability-for-vllm-and-tgi-from-dcgm-to-tokens.txt", "jsonld": "https://wpnews.pro/news/end-to-end-observability-for-vllm-and-tgi-from-dcgm-to-tokens.jsonld"}}