SGLang Monitoring and Observability with OpenTelemetry

SigNoz published a guide for monitoring self-hosted SGLang inference servers by sending Prometheus metrics and OpenTelemetry request traces to its observability platform. The setup requires launching SGLang with the --enable-metrics flag, which exposes metrics at /metrics on the API port (default 30000), and adding a Prometheus scrape job with a 30s scrape_interval to an OpenTelemetry Collector's otel-collector-config.yaml. Optional request tracing is enabled with the --enable-trace and --otlp flags plus the OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf and OTEL_EXPORTER_OTLP_TRACES_HEADERS environment variables, sending traces directly from SGLang to SigNoz without passing through the Collector.

Overview SGLang is an inference server that you run yourself. It serves models over an OpenAI-compatible API. It reports its own health in two ways: Prometheus metrics on an HTTP endpoint, and OpenTelemetry traces for each request. This guide sends both to SigNoz. The metrics answer questions that only the server can answer. How many tokens per second is the GPU producing? How full is the key-value cache, the memory pool that holds attention state for active requests? How many requests wait in the queue? Prerequisites - An SGLang server that you can restart - An OpenTelemetry Collector that can reach the SGLang host. See Install the OpenTelemetry Collector https://signoz.io/docs/opentelemetry-collection-agents/get-started/ - An instance of SigNoz Cloud https://signoz.io/teams/ or Self-Hosted https://signoz.io/docs/install/self-host/ Send SGLang metrics to SigNoz Step 1: Start SGLang with metrics enabled SGLang does not expose metrics by default. Add --enable-metrics to the launch command: python -m sglang.launch server \ --model-path <your-model \ --host 0.0.0.0 \ --port 30000 \ --enable-metrics Verify these values: - <your-model : The model you serve, for example Qwen/Qwen2.5-7B-Instruct . SGLang serves metrics at /metrics on the same port as the API. Confirm the endpoint responds before you move on: curl -s http://localhost:30000/metrics | head Step 2: Add a scrape job to the Collector Append this scrape job to your existing otel-collector-config.yaml . Do not replace the whole file. otel-collector-config.yaml receivers: prometheus: config: scrape configs: - job name: sglang scrape interval: 30s static configs: - targets: '<sglang-host :30000' Verify these values: - <sglang-host : The hostname or IP address of the machine that runs SGLang. Then add the receiver to your metrics pipeline: otel-collector-config.yaml service: pipelines: metrics: receivers: prometheus append prometheus to your existing receivers list processors: batch exporters: otlphttp Step 3: Restart the Collector Restart the Collector so that it loads the new scrape job, then watch its logs for scrape errors. sudo systemctl restart otelcol-contrib sudo journalctl -u otelcol-contrib -f docker compose up -d docker compose logs -f kubectl rollout restart deployment/<release -k8s-infra-otel-deployment -n <collector-namespace kubectl logs -f deployment/<release -k8s-infra-otel-deployment -n <collector-namespace The SigNoz K8s Infra chart installs two Collectors. Add the scrape job to the Deployment, which handles cluster-wide scraping, rather than to the per-node <release -k8s-infra-otel-agent DaemonSet. Replace <release with your Helm release name and <collector-namespace with the namespace that runs it. If you run your own Collector under a different name, use that name instead. The MSI installs the Collector as a Windows service named otelcol-contrib , with OpenTelemetry Collector as its display name. Run this in PowerShell: Restart-Service -Name otelcol-contrib Get-EventLog -LogName Application -Source otelcol-contrib -Newest 20 Send a request to SGLang so that the server has something to report. Export Request Traces Optional Metrics tell you how the server behaves as a whole. Traces show where time went inside a single request. Traces go straight from SGLang to SigNoz and do not pass through the Collector, so the scrape job above is unaffected. Set two environment variables, then add two flags to the launch command from Step 1: export OTEL EXPORTER OTLP TRACES PROTOCOL=http/protobuf export OTEL EXPORTER OTLP TRACES HEADERS="signoz-ingestion-key=<your-ingestion-key " python -m sglang.launch server \ --model-path <your-model \ --host 0.0.0.0 \ --port 30000 \ --enable-metrics \ --enable-trace \ --otlp-traces-endpoint https://ingest.<region .signoz.cloud:443/v1/traces Verify these values: - <region : Your SigNoz Cloud region https://signoz.io/docs/ingestion/signoz-cloud/overview/ endpoint . - <your-ingestion-key : Your SigNoz ingestion key https://signoz.io/docs/ingestion/signoz-cloud/keys/ . You must set both environment variables. SGLang sends no authentication header of its own, and it defaults to gRPC. Restart SGLang after you change the launch command. Validate Open Metrics Metrics Explorer and filter with service.name = 'sglang' . The Prometheus receiver takes that value from the job name in your scrape job. Metrics appear within one scrape interval. If you enabled traces, open Traces and filter with the same service.name = 'sglang' . SGLang sets that name itself and ignores OTEL SERVICE NAME , so every SGLang server reports as sglang . Metrics Reference | Metric | Type | What it tells you | |---|---|---| | sglang:gen throughput | Gauge | Output tokens generated per second. This is the tokens-per-second figure. | | sglang:prompt tokens total | Counter | Prefill tokens processed. | | sglang:generation tokens total | Counter | Output tokens produced. | | sglang:time to first token seconds | Histogram | Delay before the first output token. Drives how fast a streaming response feels. | | sglang:inter token latency seconds | Histogram | Delay between output tokens after the first one. | | sglang:e2e request latency seconds | Histogram | Total request duration. | | sglang:num running reqs | Gauge | Requests the server is decoding right now. | | sglang:num queue reqs | Gauge | Requests waiting to start. A number above zero means the server is saturated. | | sglang:token usage | Gauge | Fraction of the key-value cache pool in use, from 0 to 1. | | sglang:cache hit rate | Gauge | Fraction of prefill tokens served from the prefix cache. | Every metric carries model name , engine type , tp rank , pp rank , and moe ep rank labels. The token counters add is streaming . Group by model name when one server hosts several models. On a server that uses tensor parallelism, only rank 0 records request metrics by default, so a sum across ranks counts each request once. The --enable-metrics-for-all-schedulers flag makes every rank record separately, which doubles those sums unless you filter on tp rank . Add tp rank = '0' to your own queries against the scheduler gauges. The dashboard template already does this. SGLang also splits the queue gauges when you run it with --enable-priority-scheduling . It reports the total under priority = '' and a breakdown under priority = '<int ' , so summing every series counts each request twice. Filter on priority = '' for totals. This filter also works on servers that run without priority scheduling, where the label is absent. Metric names vary between SGLang releases. The names above come from v0.5.19. For the full list, see the SGLang production metrics reference https://docs.sglang.io/references/production metrics . Troubleshooting The metrics endpoint returns 404 Likely cause: the server started without --enable-metrics . Fix: add the flag and restart SGLang. Verify: curl -s http://localhost:30000/metrics | head prints lines that start with HELP . The Collector logs a connection refused error Likely cause: SGLang is bound to 127.0.0.1 , so nothing outside the host can reach it. Fix: start the server with --host 0.0.0.0 , and open port 30000 to the Collector host in your firewall. Verify: run curl -s http://<sglang-host :30000/metrics | head from the machine that runs the Collector. Metrics arrive but every value is zero Likely cause: the server has served no requests since it started. Counters and gauges stay at zero until traffic arrives. Fix: send a request to the server. Verify: sglang:prompt tokens total rises in Metrics Explorer. Traces do not appear Likely cause: the server is still exporting over gRPC, which cannot reach SigNoz Cloud. Fix: set OTEL EXPORTER OTLP TRACES PROTOCOL=http/protobuf and OTEL EXPORTER OTLP TRACES HEADERS in the environment of the SGLang process, then restart it. Verify: spans with service.name = sglang appear in the Traces explorer. Next steps - SGLang dashboard template https://signoz.io/docs/dashboards/dashboard-templates/sglang-dashboard/ - Set up alerts https://signoz.io/docs/setup-alerts-notification/ on queue depth and time to first token - LLM observability https://signoz.io/docs/llm-observability/ for tracing the application that calls this server Get Help If you need help with the steps in this topic, please reach out to us on SigNoz Community Slack https://signoz.io/slack/ . If you are a SigNoz Cloud user, please use in product chat support located at the bottom right corner of your SigNoz instance or contact us at cloud-support@signoz.io mailto:cloud-support@signoz.io .