# LLM Observability: Production Best Practices for AI Agents, RAG, Tracing, Tokens, Costs, and Evaluation > Source: > Published: 2026-09-16 01:37:57+00:00 **LLM observability** is the practice of monitoring and tracing AI applications to understand LLM calls, prompts, token usage, latency, costs, tool calls, retrieval, agent trajectories, errors, and output quality in production. For production AI systems, logging only the final LLM response is not enough. You need visibility across: - LLM calls - Prompts - Tokens - Costs - Latency - Tool calls - Retrieval - Agent trajectories - Errors - Evaluations - User feedback **LLM observability** is the practice of instrumenting and monitoring generative AI systems so engineers can understand what happened during an AI request, why it happened, how long it took, how much it cost, where it failed, and whether the result was useful. Unlike traditional application observability, LLM observability must account for: ``` Prompts Model generations Token usage Costs Retrieval Tool calls Agent trajectories Evaluations User feedback ``` A production AI system is often a multi-step workflow rather than a single API call. Traditional software observability helps answer: ``` Is the service healthy? Where did the request fail? Which component is slow? How many errors are occurring? ``` LLM observability adds questions such as: ``` Why did the model produce this answer? Which prompt version was used? What context did the model receive? Which tools did the agent call? How many tokens were consumed? How much did the request cost? Why did the agent retry? Did retrieval return useful documents? Was the final answer correct? ``` Without this information, debugging production AI systems becomes guesswork. Traditional observability focuses primarily on: ``` Logs Metrics Traces Infrastructure Errors ``` LLM observability extends this model with: ``` Prompts Model generations Token usage Costs Retrieval Tool calls Agent trajectories Evaluations User feedback ``` The key difference is that an LLM request is often only one step in a larger probabilistic workflow. A production AI system should trace the complete execution path. ``` php flowchart TD U[User Request] --> T[Trace] T --> A[AI Application] A --> R[Retrieval] A --> L[LLM Generation] A --> TO[Tool Call] R --> RC[Retrieved Context] L --> LR[Model Response] TO --> TR[Tool Result] RC --> L LR --> A TR --> A A --> O[Final Output] T --> M[Metrics] T --> C[Cost Tracking] T --> E[Evaluation] M --> D[Observability Platform] C --> D E --> D ``` The key principle: **Trace the execution, not just the final response.** Production AI observability can be viewed as three complementary layers. ``` php flowchart TD A[AI Observability] A --> B[System Observability] A --> C[LLM Execution Observability] A --> D[Quality Observability] B --> B1[Infrastructure] B --> B2[HTTP] B --> B3[Database] B --> B4[Queues] C --> C1[Prompts] C --> C2[Tokens] C --> C3[Latency] C --> C4[Tool Calls] C --> C5[Agent Traces] C --> C6[Retrieval] D --> D1[Evaluation] D --> D2[Groundedness] D --> D3[Correctness] D --> D4[Task Success] D --> D5[User Feedback] ``` Monitors the software around the AI: ``` CPU Memory Kubernetes Database Network HTTP Queues External APIs ``` Monitors how the AI system executes: ``` Prompts LLM generations Tokens Latency Costs Retrieval Tool calls Agent trajectories ``` Measures whether the AI system actually works: ``` Correctness Relevance Groundedness Task success User feedback Evaluation scores ``` A mature AI system needs all three. A useful trace contains multiple types of observations. ``` php flowchart LR T[Trace] T --> P[Prompt] T --> G[LLM Generation] T --> R[Retrieval] T --> TC[Tool Call] T --> A[Agent Step] T --> E[Evaluation] P --> PV[Prompt Version] G --> TOK[Tokens] G --> LAT[Latency] G --> COST[Cost] G --> MODEL[Model] R --> DOCS[Documents] R --> SCORE[Retrieval Scores] TC --> ARG[Arguments] TC --> RESULT[Result] TC --> ERR[Errors] A --> ITER[Iterations] A --> TRAJ[Trajectory] E --> QUALITY[Quality Score] ``` Useful request metadata: ``` trace_id session_id request_id user_id tenant_id feature environment region timestamp ``` For example: ``` { "trace_id": "tr_8f31", "session_id": "session_921", "feature": "document_qa", "environment": "production", "tenant_id": "company_42" } ``` Use metadata to answer: ``` Which feature is expensive? Which tenant generates the most traffic? Which workflow has the highest failure rate? Is production slower than staging? ``` An agent trajectory is the sequence of decisions and actions taken by an AI agent. ``` sequenceDiagram participant User participant Agent participant LLM participant Search participant API User->>Agent: User task Agent->>LLM: Generate next action LLM-->>Agent: Search required Agent->>Search: search(query) Search-->>Agent: Search results Agent->>LLM: Analyze results LLM-->>Agent: Call API Agent->>API: execute(parameters) API-->>Agent: API result Agent->>LLM: Generate final response LLM-->>Agent: Final answer Agent-->>User: Response ``` For production agents, the trace should preserve this entire sequence. A useful agent trace might look like: ``` Agent ├── LLM generation #1 ├── Tool: search_documents ├── Retrieval ├── LLM generation #2 ├── Tool: create_ticket ├── Tool: send_email └── Final generation ``` This is much more useful than: ``` Agent └── LLM call ``` Prompts should be treated like production code. Track: ``` prompt_name prompt_version system_prompt user_prompt template_variables model temperature max_tokens response_format ``` Prefer: ``` customer-support-agent:v12 ``` over: ``` customer-support-agent ``` This allows you to correlate: ``` php flowchart LR PV[Prompt Version] PV --> P1[v1] PV --> P2[v2] PV --> P3[v3] P1 --> T1[Production Traces] P2 --> T2[Production Traces] P3 --> T3[Production Traces] T1 --> E1[Evaluation] T2 --> E2[Evaluation] T3 --> E3[Evaluation] E1 --> F[Feedback] E2 --> F E3 --> F F --> N[Next Prompt Version] ``` Always make it possible to answer: **Which prompt version produced this response?** Treat prompts like code: - Version them - Test them - Evaluate them - Track them in production - Avoid silent production changes Every LLM invocation should expose useful usage information. ``` php flowchart TD R[User Request] R --> L1[LLM Call 1] R --> L2[LLM Call 2] R --> L3[LLM Call 3] L1 --> T1[Tokens + Cost] L2 --> T2[Tokens + Cost] L3 --> T3[Tokens + Cost] T1 --> TOTAL[Trace Total] T2 --> TOTAL T3 --> TOTAL TOTAL --> F[Feature] TOTAL --> U[User] TOTAL --> TEN[Tenant] TOTAL --> MODEL[Model] ``` Track where available: ``` input_tokens output_tokens cached_tokens reasoning_tokens ``` Calculate: ``` Cost / request Cost / trace Cost / user Cost / tenant Cost / feature Cost / model Cost / day ``` Do not assume token counts alone tell you the cost. Different models and token categories can have different prices. Keep model pricing configuration versioned so historical cost calculations remain reproducible. End-to-end latency alone is not enough. ``` php flowchart LR START[Request] --> RET[Retrieval] RET --> PROMPT[Prompt Construction] PROMPT --> LLM[LLM Request] LLM --> TTFT[Time to First Token] TTFT --> GEN[Generation] GEN --> TOOL[Tool Execution] TOOL --> POST[Post Processing] POST --> END[Response] ``` Break latency down into: ``` Total latency Retrieval latency Prompt construction LLM queue time Time to first token LLM generation time Tool execution Database time Post-processing ``` For example: ``` Total latency: 8.4s Retrieval: 420ms Prompt construction: 80ms LLM #1: 1.8s Tool call: 950ms LLM #2: 2.4s Tool call: 1.1s Final generation: 1.7s ``` This makes performance bottlenecks visible. Retrieval-Augmented Generation needs its own observability layer. ``` php flowchart TD Q[User Query] Q --> EMB[Embedding] EMB --> VS[Vector Search] VS --> DOCS[Top K Documents] DOCS --> RR[Reranker] RR --> CTX[Final Context] CTX --> LLM[LLM] LLM --> ANSWER[Answer] VS --> RM[Retrieval Metrics] RR --> RM RM --> E[Evaluation] ANSWER --> E ``` Trace: ``` query embedding model retriever top_k documents retrieved similarity scores reranker final context retrieval latency empty-result rate ``` This helps distinguish: ``` Bad answer because of the model ``` from: ``` Bad answer because the model received bad context ``` Tools should be treated as first-class observations. ``` php flowchart TD A[Agent] A --> TC[Tool Call] TC --> NAME[Tool Name] TC --> VER[Tool Version] TC --> ARG[Arguments] TC --> EXEC[Execution] EXEC --> RESULT[Result] EXEC --> LAT[Latency] EXEC --> STATUS[Status] EXEC --> ERROR[Error] ERROR --> RETRY[Retry] RETRY --> TC ``` Track: ``` tool_name tool_version arguments result latency status error retry_count ``` This lets you answer: ``` Which tools are called most frequently? Which tools fail most often? Which tools are slow? Which tools cause retries? Which tools are incorrectly selected? ``` Agent loops are an important production failure mode. ``` php flowchart TD START[Agent Step] --> LLM[LLM] LLM --> TOOL[Tool Call] TOOL --> RESULT[Tool Result] RESULT --> CHECK{Progress?} CHECK -->|Yes| LLM CHECK -->|No| LOOP[Potential Loop] LOOP --> LIMIT{Execution Limit?} LIMIT -->|No| RECOVERY[Recovery Strategy] LIMIT -->|Yes| STOP[Stop Agent] ``` Monitor: ``` iterations_per_trace tool_calls_per_trace retries_per_trace duplicate_tool_calls maximum_depth execution_time cost_per_trace ``` Production agents should have explicit limits, for example: ``` max_iterations = 20 max_tool_calls = 50 max_execution_time = 60s max_cost = €0.50 ``` Observability should help enforce and investigate these limits. LLM applications have multiple failure surfaces. ``` php flowchart TD REQUEST[Request] REQUEST --> LLM[LLM] REQUEST --> RET[Retrieval] REQUEST --> TOOL[Tool] REQUEST --> DB[Database] REQUEST --> EXT[External API] LLM --> ERR[Failure] RET --> ERR TOOL --> ERR DB --> ERR EXT --> ERR ERR --> CLASSIFY[Error Classification] CLASSIFY --> TIMEOUT[Timeout] CLASSIFY --> RATE[Rate Limit] CLASSIFY --> AUTH[Authentication] CLASSIFY --> CONTEXT[Context Limit] CLASSIFY --> SCHEMA[Schema Validation] CLASSIFY --> PROVIDER[Provider Failure] CLASSIFY --> AGENT[Agent Loop] ``` Capture structured failure information: ``` provider model status_code error_type error_message retry_count trace_id span_id ``` Useful categories include: ``` Timeout Rate limit Authentication Invalid request Context window exceeded Tool failure Retrieval failure Schema validation failure Agent loop Guardrail violation Provider outage ``` If an LLM generates JSON or another structured format, trace validation separately. ``` php flowchart LR LLM[LLM] --> JSON[Structured Response] JSON --> VALIDATE[Schema Validation] VALIDATE -->|Valid| CONTINUE[Continue] VALIDATE -->|Invalid| REPAIR[Retry / Repair] REPAIR --> LLM ``` Record: ``` schema_name schema_version validation_status validation_errors repair_attempts ``` This is particularly important for: - AI APIs - Agent tools - Data extraction - Workflow automation - Healthcare systems - Financial systems Observability and evaluation answer different questions. ``` php flowchart LR SYSTEM[AI System] SYSTEM --> TRACE[Observability] TRACE --> WHAT[What happened?] TRACE --> WHY[Why did it happen?] TRACE --> COST[What did it cost?] TRACE --> LAT[How long did it take?] SYSTEM --> EVAL[Evaluation] EVAL --> QUALITY[Was it correct?] EVAL --> RELEVANCE[Was it relevant?] EVAL --> GROUND[Was it grounded?] EVAL --> SUCCESS[Was the task successful?] ``` Answers: ``` What happened? Where did it happen? How long did it take? What did it cost? ``` Answers: ``` Was the answer correct? Was it relevant? Was it grounded? Was the task successful? ``` The strongest AI engineering workflows connect both. Connect user feedback to traces. For example: ``` 👍 👎 ``` or: ``` rating = 1..5 ``` Associate feedback with: ``` trace_id model prompt_version feature agent retrieval configuration ``` This makes it possible to investigate: ``` Which prompt version receives the most negative feedback? Which workflow has the highest task-success rate? Which model produces more successful outcomes? ``` **OpenTelemetry** is an important vendor-neutral foundation for production observability. Use it for: ``` Distributed traces Metrics Logs Context propagation Vendor-neutral instrumentation ``` For AI systems, use standardized GenAI semantic conventions where practical instead of inventing application-specific attribute names for common telemetry. A useful architecture is: ``` php flowchart TD APP[AI Application] APP --> OTEL[OpenTelemetry] OTEL --> TRACE[Traces] OTEL --> METRICS[Metrics] OTEL --> LOGS[Logs] TRACE --> COLLECTOR[OpenTelemetry Collector] METRICS --> COLLECTOR LOGS --> COLLECTOR COLLECTOR --> AI[AI Observability Platform] COLLECTOR --> MON[Infrastructure Monitoring] AI --> LLMTRACE[LLM Tracing] AI --> EVAL[Evaluation] AI --> PROMPTS[Prompt Management] AI --> COST[Cost Tracking] MON --> DASH[Infrastructure Dashboards] ``` This helps reduce vendor lock-in. Use OpenTelemetry as the instrumentation and transport layer, then use specialized AI observability platforms where you need LLM-specific analysis. There is no universal best observability tool. Choose based on your architecture and requirements. | Tool | Typical use | |---|---| | **OpenTelemetry** | Vendor-neutral telemetry and distributed tracing | | **Langfuse** | LLM tracing, prompts, costs, evaluations, agent observability | | **Arize Phoenix** | LLM/RAG/agent tracing and evaluation | | **Prometheus** | Metrics | | **Grafana** | Dashboards and metrics visualization | | **Datadog** | Full-stack observability | | **New Relic** | Application and infrastructure observability | | **Elastic** | Logs, traces, metrics, and search | Choose tools based on requirements such as: ``` Self-hosting Data residency Privacy requirements OpenTelemetry support Evaluation capabilities Prompt management Cost visibility Agent tracing RAG debugging Existing observability stack ``` Observability creates another data surface. ``` php flowchart LR APP[AI Application] APP --> TEL[Telemetry] TEL --> REDACT[Redaction] REDACT --> PII[PII Filtering] REDACT --> SECRET[Secret Filtering] REDACT --> POLICY[Retention Policy] PII --> STORE[Observability Storage] SECRET --> STORE POLICY --> STORE STORE --> ACCESS[Access Control] ``` Potentially sensitive data includes: ``` User prompts LLM responses Retrieved documents Tool arguments API responses Personal data Financial data Internal company information ``` Never blindly log: ``` API keys Access tokens Passwords Session cookies Authorization headers Private credentials ``` Consider: ``` PII redaction Field-level masking Data retention policies Encryption Access control Sampling Payload filtering Tenant isolation ``` Observability must not become a new data-leakage surface. Not every production trace needs the same level of detail. ``` php flowchart TD REQUEST[Request] REQUEST --> CHECK{Important Trace?} CHECK -->|Error| FULL[100% Trace] CHECK -->|Slow| FULL CHECK -->|Expensive| FULL CHECK -->|Critical Workflow| FULL CHECK -->|Normal| SAMPLE[Sample] SAMPLE --> LOW[Partial / Reduced Trace] ``` A practical strategy might be: ``` Errors: 100% Slow requests: 100% Expensive requests: 100% Critical workflows: 100% Normal traffic: 5–20% ``` The exact rate should depend on traffic volume, storage cost, privacy requirements, and debugging needs. A useful AI observability dashboard should cover four dimensions. ``` mindmap root((LLM Observability)) Reliability Error rate Timeout rate Retry rate Agent failures Tool failures Performance P50 latency P95 latency P99 latency TTFT Tool latency Retrieval latency Cost Cost/request Cost/user Cost/tenant Cost/feature Cost/model Quality Correctness Relevance Groundedness Task success User feedback ``` Alert on meaningful changes such as: ``` LLM error rate increases P95 latency increases Cost per request increases Token usage suddenly increases Tool failure rate increases Agent iteration count increases Retrieval empty-result rate increases Task-success rate decreases Evaluation score regresses ``` Prefer alerts based on meaningful baselines and business impact rather than arbitrary thresholds. If you are starting from zero, do not try to implement everything at once. Start with: 1. Trace ID for every AI request 2. Individual LLM generation spans 3. Model name 4. Prompt version 5. Input/output tokens 6. Latency 7. Cost 8. Error tracking 9. Tool-call tracing for agents 10. Basic evaluation or user feedback A practical implementation path: ``` Phase 1 → Traces + errors Phase 2 → Tokens + cost + latency Phase 3 → Tools + retrieval Phase 4 → Evaluation + feedback Phase 5 → Automated regression, cost, and quality alerts ``` This gives you useful observability early without over-engineering the system. A useful way to think about observability maturity is: ``` php flowchart LR A[Level 1
Logging] --> B[Level 2
Tracing] B --> C[Level 3
Cost & Performance] C --> D[Level 4
Evaluation] D --> E[Level 5
Continuous Optimization] Errors Requests Basic model information LLM calls Tool calls Retrieval Agent steps Tokens Costs Latency TTFT Resource usage Correctness Relevance Groundedness Task success User feedback Automated regression detection Prompt experiments Model experiments Quality alerts Cost optimization Continuous evaluation ``` The goal is not necessarily to implement every feature immediately. The maturity model helps prioritize what to build next. - Every important request has a trace ID - Every LLM call is traced - Tool calls are traced - Retrieval is traced - Agent iterations are visible - External API calls are correlated - Prompts are versioned - Prompt version is stored in traces - Prompt changes can be compared - Input tokens are tracked - Output tokens are tracked - Cached/reasoning tokens are tracked when available - Cost is calculated - Cost can be grouped by feature, model, and tenant - P50/P95/P99 latency is monitored - Time-to-first-token is tracked - Slow spans are identifiable - Retrieval and tool latency are visible - Agent trajectories are traceable - Tool calls are visible - Retries are tracked - Loops are detectable - Maximum iterations are enforced - Maximum execution time is enforced - Cost limits exist - Retrieval queries are traced - Retrieved documents are visible - Similarity scores are available - Reranking is observable - Context size is tracked - Empty retrieval results are monitored - Secrets are removed - PII is protected - Trace access is controlled - Retention policies exist - Sensitive payloads are sampled or redacted - Traces can be evaluated - User feedback is connected to traces - Failed traces can become evaluation datasets - Prompt/model changes can be evaluated - Quality regressions can trigger alerts The ultimate goal is not simply collecting traces. It is continuous improvement. ``` php flowchart TD PROD[Production AI System] PROD --> TRACE[Tracing] TRACE --> EVAL[Evaluation] EVAL --> ANALYZE[Failure Analysis] ANALYZE --> DATASET[Evaluation Dataset] DATASET --> EXP[Experiment] EXP --> PROMPT[Prompt Change] EXP --> MODEL[Model Change] EXP --> RETRIEVAL[Retrieval Change] EXP --> TOOL[Tool Change] PROMPT --> DEPLOY[Deploy] MODEL --> DEPLOY RETRIEVAL --> DEPLOY TOOL --> DEPLOY DEPLOY --> PROD ``` The engineering loop becomes: ``` Observe ↓ Evaluate ↓ Diagnose ↓ Improve ↓ Deploy ↓ Observe ``` This is where observability becomes an AI engineering capability rather than just another monitoring dashboard. **LLM observability is the foundation for operating AI systems in production.** Traditional observability tells you whether your service is running. LLM observability tells you: ``` What happened? Why did it happen? Which model was used? Which prompt version was used? What context did it receive? Which tools were called? How many tokens were consumed? How much did it cost? How long did it take? Where did it fail? Was the result actually good? ``` For simple LLM applications: ``` OpenTelemetry + LLM tracing + Token tracking + Cost tracking + Error monitoring ``` For RAG: ``` Retrieval tracing + Context inspection + RAG evaluation ``` For AI agents: ``` Agent trajectory tracing + Tool-call observability + Loop detection + Evaluation + Cost controls ``` The goal is not to collect more telemetry. The goal is to make **AI behavior observable, measurable, debuggable, secure, and continuously improvable.** **If you cannot reconstruct what an AI system did, you cannot reliably debug, optimize, or operate it in production.** `LLM Observability` · `AI Observability` · `LLM Tracing` · `AI Agents` · `RAG` · `OpenTelemetry` · `LLM Evaluation` · `LLM Cost Tracking` · `Production AI`