# What is LLM observability? Tracing where the payload is the signal > Source: > Published: 2026-08-13 00:00:00+00:00 # What is LLM observability? Tracing where the payload is the signal LLM observability is the practice of recording what an LLM application actually did on every request - the prompt it sent, the response it got back, the tools it called, the tokens it burned - so that any single one of those runs can be reconstructed afterwards. Technically it's distributed tracing. In practice it's a different job, because of one shift: **the payload, not the timing, is the signal.** Ordinary observability tells you a request took 4.2 seconds and returned `200` . For an LLM feature that's nearly useless, because the interesting failures return `200` too: the model answers fluently and wrongly, or the agent calls the same tool eleven times and gives up, or retrieval matches nothing and the model fills the silence itself. No status code knows about any of that, and no latency histogram does either. The evidence is in the content, so that's what has to be in the trace. If you're running LLM features in production, this covers what a trace should record, how observability differs from monitoring and from evaluation, and what gets expensive once the payloads grow. It won't rank vendors for you. Evaluation frameworks and prompt management come up only in passing; both deserve their own posts. Two words do a lot of work below. A **span** is one recorded operation, say a model call or a database query, with a start time, a duration and a set of attributes. A **trace** is the tree of spans produced by a single user request. Almost everything specific to LLM observability comes down to what those spans are allowed to carry. [What LLM observability records](#what-llm-observability-records)What LLM observability records An LLM trace carries everything ordinary tracing carries - service, operation, duration, parent/child structure, errors - and then the fields that make a run explainable: | Recorded on the span | Why it has to be there | |---|---| | The rendered prompt, system prompt and injected context included | The bug is usually in what the model was actually asked, not in the code that asked it | | The completion, and the reason generation stopped | `length` and `tool_calls` are quiet failures that never raise | | Model, provider, temperature and the rest of the params | Behaviour moves when any of these move, and they move without a deploy | | Input and output token counts | The only honest basis for cost per request | | Tool calls: name, arguments, result | Where agents spend their time, and where they loop | | Retrieval: the query, what came back, the scores | Separates "the model made it up" from "the model was handed nothing" | | Eval scores, attached after the fact | Quality gets computed later; it isn't a status code | | A conversation or session id | A multi-turn chat is several traces that belong together | The last two rows have no equivalent in ordinary observability, and the payload rows are far larger than anything a normal span carries. Those two facts are what make it awkward to operate, and the rest follows from them. [One user action is a tree, not a request](#one-user-action-is-a-tree,-not-a-request)One user action is a tree, not a request The other reason ordinary tooling struggles is structure. A user asks one question; the application answers it with a fan-out of model calls, tool calls, retrievals and retries. The unit you debug is the whole tree, not the span that happened to be slow. Two things in that picture matter. First, the `GET /flights/BA117` span is an ordinary HTTP span with no AI payload at all - a real agent trace is a mix, and a large share of it is ordinary infrastructure work. Second, the eval score hanging off the final `chat` span was not produced by the request. It was computed afterwards and attached to a span that had already finished, which is a write pattern ordinary tracing backends don't really have. [Monitoring, observability, evaluation](#monitoring,-observability,-evaluation)Monitoring, observability, evaluation These three get used interchangeably in marketing copy and they aren't the same thing. The distinction is practical, because each answers a different question and each fails you at a different moment. | Practice | Question it answers | Works on | Tells you a wrong answer was wrong | |---|---|---|---| Monitoring | Is the system healthy right now? | Aggregates: rates, p95, errors | No | Observability | What exactly happened in this run? | Individual traces and payloads | Only if you go and read it | Evaluation | Is the output any good? | Scores computed over traces | Yes, that is its whole job | Monitoring is necessary and cheap and it won't catch a hallucination. Evaluation catches quality problems but cannot tell you why the answer was bad. Observability is the layer the other two stand on: monitoring aggregates the traces, evals score them. If the traces are sampled or truncated, both of the others degrade quietly. [The data model: use the one that exists](#the-data-model-use-the-one-that-exists)The data model: use the one that exists The temptation with a new problem is to invent a schema. Don't. There's a standard already, and it's widely adopted. The [OpenTelemetry GenAI semantic conventions](https://github.com/open-telemetry/semantic-conventions-genai) define the span names and the `gen_ai.*` attributes for all of this: `gen_ai.request.model` , `gen_ai.usage.input_tokens` , `gen_ai.usage.output_tokens` , `gen_ai.response.finish_reasons` , the agent and tool span structure, and opt-in prompt and completion capture. Emitting standard attributes means you instrument once and stay free to move. Any OTLP-compatible backend will accept the data, so sending your telemetry somewhere else is a config change, not a re-instrumentation project. The conventions are still marked experimental, so pin your instrumentation versions and expect the occasional rename. Our hands-on walkthrough - auto-instrumentation, manual agent and tool spans, token usage, conversation grouping - is in [how to instrument an LLM agent with OpenTelemetry](/posts/instrument-llm-agent-opentelemetry). [Three problems every team hits about a month in](#three-problems-every-team-hits-about-a-month-in)Three problems every team hits about a month in [The bill grows with how good your agent gets](#the-bill-grows-with-how-good-your-agent-gets)The bill grows with how good your agent gets An agent that plans better, calls more tools and checks its work emits more spans per user task. So the observability bill rises as the product improves, which is a deeply unhelpful incentive. It gets worse when every span in the trace is charged at the premium AI rate, because much of an agent trace is ordinary HTTP, database and queue work that needs no AI-specific processing at all. We wrote that up separately in [the AI tax on ordinary telemetry](/posts/the-ai-tax-on-telemetry). [You are pushed to drop the data you most need](#you-are-pushed-to-drop-the-data-you-most-need)You are pushed to drop the data you most need The standard answer to volume is to sample, truncate payloads, or set a short retention. Each of those quietly removes the evidence. Sampling assumes the interesting run is statistically typical, and the run you will be asked about is the one a customer complained about by name. Truncating the prompt removes the context that explains the answer. Short retention means the regression someone noticed this week happened outside your window. Prompts and completions are kilobytes per span rather than tags, so the pressure shows up early. Giving in to it throws away the thing you started collecting for. [Your traces end up in someone else's format](#your-traces-end-up-in-someone-elses-format)Your traces end up in someone else's format Trace data has a second life as raw material for evals, fine-tuning sets, regression suites, and whatever you think to ask in six months. That only works if you can get it out and query it with ordinary tools. Telemetry locked in a proprietary indexed store is readable through one product's UI and one product's API, on that product's retention terms, and nowhere else. The alternative is to keep it in open columnar formats - [Apache Parquet](https://parquet.apache.org/) files in an [Apache Iceberg](https://iceberg.apache.org/) table on your own object storage - where SQL engines can read it directly and retention is priced like storage rather than like an index. That is the architecture behind [IceGate](https://icegate.tech), our open-source engine, and we described how the ingestion side works in [observability ingestion on S3 alone](/posts/observability-ingestion-on-s3). [The numbers behind this](#the-numbers-behind-this)The numbers behind this We can put figures on the billing-line argument, but they come out of a cost model rather than a measurement, and that's worth saying before the numbers rather than after them. On a heavy profile - 5,000,000 traces a month at 10 spans each - charging the AI rate only for AI spans and ordinary rates for the rest comes out around **2x cheaper** than pricing every span in the trace at the premium. Our own rate card has that shape: all ingested bytes are charged at €0.25/GB, and LLM-span bytes carry a further €2.50/GB on top, so the premium lands only where an AI payload actually exists. The span mix and the 20 KB per-span size in that model are assumptions we disclosed to show the shape of the problem, not measurements of anyone's production traffic. The workings are in [the AI tax on ordinary telemetry](/posts/the-ai-tax-on-telemetry). You can't check a cost figure that comes with no workload behind it, and you can't compare it against your own. Ours are above with the assumptions attached, and it's fair to want the same from anyone else. [How to start](#how-to-start)How to start **Instrument with OpenTelemetry, not a vendor SDK.** Auto-instrumentation for your model client gets you most of the`gen_ai.*` attributes without touching application code.**Make one user task one trace.** Wrap the whole agent run in a parent span so the model calls, tool calls and retrievals hang off it. A flat list of model calls isn't a trace.**Stamp a conversation id on every span, not just the top one.** Span attributes don't inherit, so multi-turn grouping needs a span processor, not a single assignment.**Turn on payload capture deliberately.** Prompts and completions are off by default for good reason. Turn them on where you need them, and work out what gets redacted first, not afterwards.**Decide retention before volume decides for you.** Work out what a month of full-fidelity traces costs at your projected traffic. If the answer forces sampling, fix the storage layer instead of the instrumentation. [Where we sit](#where-we-sit)Where we sit We build [TripleCloud](https://triplecloud.tech), an LLM observability platform on the open lakehouse: it speaks OTLP, so an existing OpenTelemetry pipeline points at it without re-instrumentation, and traces land in open Parquet and Iceberg on storage you control. AI spans are priced as AI spans and ordinary spans aren't, which is what keeps full-fidelity retention cheap enough that you don't have to sample away the runs someone will eventually ask you about. TripleCloud is rolling out with its first teams now. Join the [waitlist](https://triplecloud.tech) and we'll get in touch, or just read the code: IceGate is Apache-2.0 and [on GitHub](https://github.com/icegatetech/icegate). [FAQ](#faq)FAQ [What is LLM observability?](#what-is-llm-observability)What is LLM observability? LLM observability is the practice of recording what an LLM application did on every request - the prompt, the response, the tool calls, the retrievals and the token counts - so that an individual run can be reconstructed and judged afterwards. It's distributed tracing where the payload of each span, not its timing, carries most of the information. [How is LLM observability different from ordinary observability?](#how-is-llm-observability-different-from-ordinary-observability)How is LLM observability different from ordinary observability? Two differences. The payload matters more than the timing, because an LLM request can return `200` in normal time and still be wrong, so the prompt and completion have to be stored rather than summarised into metrics. And quality is computed after the fact by evaluation, then attached back to a trace that has already finished, which ordinary application monitoring has no equivalent for. [What is the difference between LLM monitoring and LLM observability?](#what-is-the-difference-between-llm-monitoring-and-llm-observability)What is the difference between LLM monitoring and LLM observability? Monitoring works on aggregates - request rates, latency percentiles, error ratios - and tells you when something changed. Observability works on individual traces and tells you what happened in one specific run. Monitoring is built on top of the traces observability collects, so it isn't an alternative to it. [Do I need a special SDK, or can I use OpenTelemetry?](#do-i-need-a-special-sdk,-or-can-i-use-opentelemetry)Do I need a special SDK, or can I use OpenTelemetry? OpenTelemetry is enough, and it's the better default. The GenAI semantic conventions define the `gen_ai.*` attributes for models, tokens, agents and tools, and auto-instrumentation for the common model clients emits them without application changes. Instrumenting to the standard keeps the choice of backend a configuration decision. [Do I have to store full prompts and completions?](#do-i-have-to-store-full-prompts-and-completions)Do I have to store full prompts and completions? Not always, but the debugging value collapses without them. A trace recording only that a model was called and took 3 seconds doesn't explain a wrong answer. Prompt capture is opt-in in the OpenTelemetry conventions, so treat it as a deliberate decision per service, with redaction of secrets and personal data applied before the data leaves the process. [Why does LLM observability cost so much more than ordinary tracing?](#why-does-llm-observability-cost-so-much-more-than-ordinary-tracing)Why does LLM observability cost so much more than ordinary tracing? Two reasons compound. Payloads are kilobytes per span where ordinary spans are bytes, and agent applications emit many spans per user task rather than one. On top of that, most tools charge a premium AI rate for every span in the trace, including the ordinary HTTP and database spans that carry no AI payload at all. [Can I keep using my existing observability stack?](#can-i-keep-using-my-existing-observability-stack)Can I keep using my existing observability stack? For the ordinary half of your traces, yes, that data is unchanged. What existing stacks tend to handle badly is the AI half: large payloads on every span, eval scores written back to finished traces, and retention priced in a way that makes keeping full-fidelity prompt data unattractive. Since the telemetry is OTLP either way, running both and pointing the AI-heavy volume somewhere designed for it is a routing change, not a migration. [Trademarks and disclaimer](#trademarks-and-disclaimer)Trademarks and disclaimer **Trademarks.** OpenTelemetry is a trademark of the Cloud Native Computing Foundation (CNCF) / The Linux Foundation. Apache®, Apache Iceberg™, and Apache Parquet™ are trademarks of The Apache Software Foundation. Amazon S3® is a trademark of Amazon Technologies, Inc. All other product names, logos, and brands are the property of their respective owners. Names are used solely for identification and do not imply affiliation, sponsorship, or endorsement by the trademark holders. **Disclaimer.** IceGate and TripleCloud are not affiliated with or endorsed by the Cloud Native Computing Foundation (CNCF), The Apache Software Foundation, Amazon Web Services, or any of the other organizations mentioned here. The OpenTelemetry GenAI semantic conventions are still experimental and subject to breaking change, so attribute names above may differ from the release you pin. Corrections and clarifications are welcome - we will fix any inaccuracies.