LLM Observability Core Concepts - Learn Spans and Alerts SigNoz is documenting its LLM observability approach, which relies on standard OpenTelemetry libraries emitting `gen_ai.*` spans and metrics over OTLP rather than a SigNoz agent, so LLM call data is stored alongside other application telemetry. The guide distinguishes client-side telemetry from server-side telemetry from self-hosted inference servers such as vLLM and SGLang, which do not emit `gen_ai.*` and instead expose Prometheus-format metrics scraped by an OpenTelemetry Collector, covering tokens per second on the GPU, key-value cache usage, and queue depth. SigNoz argues LLM calls should be instrumented separately from ordinary HTTP dependencies because they are nondeterministic, metered, and slow, and recommends tracking token usage, latency percentiles with time to first token split out for streaming, error classes such as HTTP 429 rate limits, and cost and prompt attribution. LLM observability means that you collect traces, metrics, and logs from applications that call large language models LLMs . With this data, you see what each model call did, how long it took, how many tokens it used, and where it failed. One user request in an LLM application can start retrieval steps, several model calls, tool calls, and retries. Without instrumentation, you see only the total latency and the total bill. SigNoz uses OpenTelemetry for this, and you do not install a SigNoz agent. Your application emits gen ai. spans and metrics through standard OpenTelemetry libraries and exports them over OTLP. SigNoz stores this data next to the rest of your application telemetry. To set up a model provider, agent framework, or gateway, pick its guide in Send Data https://signoz.io/docs/llm-observability/ . Client-Side and Server-Side Telemetry Client-side telemetry comes from the application that calls a model. This page covers client-side telemetry: gen ai. spans and metrics from an instrumentation library, exported over OTLP. Server-side telemetry comes from an inference server that you run yourself, such as vLLM https://signoz.io/docs/vllm-observability/ or SGLang https://signoz.io/docs/sglang-observability/ . These servers do not emit gen ai. . They expose their own metrics in Prometheus format, and an OpenTelemetry Collector scrapes them. These metrics show data that the calling application cannot see: tokens per second on the GPU, key-value KV cache usage, and queue depth. A client-side span shows that a call was slow. Server-side metrics show the cause: a deep queue, a full cache, or a long output. If you host your own models, collect both. What LLM Observability Covers An LLM call is a network request to a service that is nondeterministic, metered, and slow compared to most dependencies. For these reasons, instrument it separately from an ordinary HTTP dependency. Traces and spans across the call chain The unit of analysis is the full request. A retrieval-augmented generation RAG request usually produces an embedding span, a vector search span, and a chat completion span. It often adds a second completion span for reranking or summarization. One trace for all these spans shows which stage used the most time. In SigNoz, these spans appear in the Traces explorer https://signoz.io/docs/userguide/traces/ next to spans from your web framework and database. The LLM call and the SQL query that fed it show in the same waterfall. Token usage Instrumentation records the input and output tokens of each call. Providers bill by token, so token counts by model and by code path give you the closest live cost signal. Providers that support prompt caching report cached input tokens separately. They usually bill cached tokens at a lower rate. Latency LLM latency depends on output length and often has two peaks, so an average hides the real behavior. Track percentiles. For streaming responses, track time to first token TTFT separately from total duration. For example, one response starts to stream after 300 ms and finishes after 8 seconds. Another response shows nothing for 8 seconds and then returns the full answer. Both have the same total duration, but the user feels that the first one is fast. Errors Rate limits HTTP 429 , context-length overflows, content filter rejections, upstream 5xx responses, and client timeouts are different failures. Each one needs a different fix. Record an error class on each span. You can then separate throttling from a prompt that the model rejected. Cost attribution Token counts do not show who spent them. Attribute spend to a model, a tenant, a feature, or a prompt version. A single monthly figure then becomes a cost that you can act on. Model and prompt attribution Capture the requested model and the model that served the response, because providers alias versions behind names like gpt-4 . Record prompt template names and versions to compare quality and cost across prompt revisions. Retrieval steps In RAG, the retrieval stage often causes bad answers. Query latency, the number of returned documents, and empty result sets separate a retrieval failure from a generation failure. Agent and tool calls Agent frameworks run in a loop. The model picks a tool, the tool runs, the framework sends the result back, and the model decides again. Record each tool execution as its own span. You then see runaway loops, tools that fail without an error, and the number of iterations per request. Instrumenting LLM Applications with OpenTelemetry LLM instrumentation works the same way as for any other service that uses OpenTelemetry. A library patches your LLM client at runtime, wraps each call in a span, and adds attributes to the span. An exporter then sends the finished spans over OTLP to SigNoz. Automatic instrumentation Automatic instrumentation is the default and covers most applications. You install an instrumentation package, and it patches the provider SDK. Each call then produces a span, and you do not change your application code. Most LLM calls go through a small number of client libraries, so a few packages cover most traffic. Automatic instrumentation records the model, the token counts, the duration, and the error status. It does not know your application data, such as the tenant that made the request or the experiment group of the user. Manual instrumentation Manual instrumentation adds that data. You create spans with the OpenTelemetry API for stages that only your application has. Examples are a document chunking step or an "answer a support ticket" operation. You can also add your own attributes to existing spans. Most teams use both: automatic instrumentation for provider calls, and manual spans for the application logic around them. Instrumentation libraries Three community projects supply most LLM instrumentation today. All three emit standard OpenTelemetry spans over OTLP, so all three work with SigNoz. - OpenLLMetry https://github.com/traceloop/openllmetry from Traceloop: instrumentation for provider SDKs and frameworks. See the Traceloop integration guide https://signoz.io/docs/traceloop/ . - OpenLIT https://github.com/openlit/openlit : instrumentation for LLM SDKs, vector databases, and GPU metrics. See the OpenLIT integration guide https://signoz.io/docs/openlit/ . - OpenInference https://github.com/Arize-ai/openinference from Arize: broad coverage of agent and orchestration frameworks, such as LangChain, LlamaIndex, and CrewAI. The OpenTelemetry contrib repositories https://github.com/open-telemetry/opentelemetry-python-contrib also maintain instrumentation for some providers, such as opentelemetry-instrumentation-openai-v2 . When an official instrumentation exists, the integration guides use it. No vendor SDK SigNoz ingests OTLP. It has no LLM-specific agent and no SigNoz SDK for your application. Any tool that sends OTLP over HTTP or gRPC can send data to SigNoz. You can pick the instrumentation library that covers your stack best, or mix libraries. Your data stays standard OpenTelemetry. To move it to another backend later, you change the endpoint and keep your instrumentation. Set the exporter the same way as for any other OpenTelemetry service: OTEL EXPORTER OTLP ENDPOINT="https://ingest.