LLM Observability: Production Best Practices for AI Agents, RAG, Tracing, Tokens, Costs, and Evaluation A developer outlined production best practices for LLM observability, arguing that logging only the final model response is insufficient for AI systems. The guidance recommends tracing the full execution path—prompts, token usage, costs, latency, retrieval, tool calls, agent trajectories, and evaluations—across three layers: system, LLM execution, and quality observability. The key principle stated is to "trace the execution, not just the final response. LLM observability is the practice of monitoring and tracing AI applications to understand LLM calls, prompts, token usage, latency, costs, tool calls, retrieval, agent trajectories, errors, and output quality in production. For production AI systems, logging only the final LLM response is not enough. You need visibility across: - LLM calls - Prompts - Tokens - Costs - Latency - Tool calls - Retrieval - Agent trajectories - Errors - Evaluations - User feedback LLM observability is the practice of instrumenting and monitoring generative AI systems so engineers can understand what happened during an AI request, why it happened, how long it took, how much it cost, where it failed, and whether the result was useful. Unlike traditional application observability, LLM observability must account for: Prompts Model generations Token usage Costs Retrieval Tool calls Agent trajectories Evaluations User feedback A production AI system is often a multi-step workflow rather than a single API call. Traditional software observability helps answer: Is the service healthy? Where did the request fail? Which component is slow? How many errors are occurring? LLM observability adds questions such as: Why did the model produce this answer? Which prompt version was used? What context did the model receive? Which tools did the agent call? How many tokens were consumed? How much did the request cost? Why did the agent retry? Did retrieval return useful documents? Was the final answer correct? Without this information, debugging production AI systems becomes guesswork. Traditional observability focuses primarily on: Logs Metrics Traces Infrastructure Errors LLM observability extends this model with: Prompts Model generations Token usage Costs Retrieval Tool calls Agent trajectories Evaluations User feedback The key difference is that an LLM request is often only one step in a larger probabilistic workflow. A production AI system should trace the complete execution path. php flowchart TD U User Request -- T Trace T -- A AI Application A -- R Retrieval A -- L LLM Generation A -- TO Tool Call R -- RC Retrieved Context L -- LR Model Response TO -- TR Tool Result RC -- L LR -- A TR -- A A -- O Final Output T -- M Metrics T -- C Cost Tracking T -- E Evaluation M -- D Observability Platform C -- D E -- D The key principle: Trace the execution, not just the final response. Production AI observability can be viewed as three complementary layers. php flowchart TD A AI Observability A -- B System Observability A -- C LLM Execution Observability A -- D Quality Observability B -- B1 Infrastructure B -- B2 HTTP B -- B3 Database B -- B4 Queues C -- C1 Prompts C -- C2 Tokens C -- C3 Latency C -- C4 Tool Calls C -- C5 Agent Traces C -- C6 Retrieval D -- D1 Evaluation D -- D2 Groundedness D -- D3 Correctness D -- D4 Task Success D -- D5 User Feedback Monitors the software around the AI: CPU Memory Kubernetes Database Network HTTP Queues External APIs Monitors how the AI system executes: Prompts LLM generations Tokens Latency Costs Retrieval Tool calls Agent trajectories Measures whether the AI system actually works: Correctness Relevance Groundedness Task success User feedback Evaluation scores A mature AI system needs all three. A useful trace contains multiple types of observations. php flowchart LR T Trace T -- P Prompt T -- G LLM Generation T -- R Retrieval T -- TC Tool Call T -- A Agent Step T -- E Evaluation P -- PV Prompt Version G -- TOK Tokens G -- LAT Latency G -- COST Cost G -- MODEL Model R -- DOCS Documents R -- SCORE Retrieval Scores TC -- ARG Arguments TC -- RESULT Result TC -- ERR Errors A -- ITER Iterations A -- TRAJ Trajectory E -- QUALITY Quality Score Useful request metadata: trace id session id request id user id tenant id feature environment region timestamp For example: { "trace id": "tr 8f31", "session id": "session 921", "feature": "document qa", "environment": "production", "tenant id": "company 42" } Use metadata to answer: Which feature is expensive? Which tenant generates the most traffic? Which workflow has the highest failure rate? Is production slower than staging? An agent trajectory is the sequence of decisions and actions taken by an AI agent. sequenceDiagram participant User participant Agent participant LLM participant Search participant API User- Agent: User task Agent- LLM: Generate next action LLM-- Agent: Search required Agent- Search: search query Search-- Agent: Search results Agent- LLM: Analyze results LLM-- Agent: Call API Agent- API: execute parameters API-- Agent: API result Agent- LLM: Generate final response LLM-- Agent: Final answer Agent-- User: Response For production agents, the trace should preserve this entire sequence. A useful agent trace might look like: Agent ├── LLM generation 1 ├── Tool: search documents ├── Retrieval ├── LLM generation 2 ├── Tool: create ticket ├── Tool: send email └── Final generation This is much more useful than: Agent └── LLM call Prompts should be treated like production code. Track: prompt name prompt version system prompt user prompt template variables model temperature max tokens response format Prefer: customer-support-agent:v12 over: customer-support-agent This allows you to correlate: php flowchart LR PV Prompt Version PV -- P1 v1 PV -- P2 v2 PV -- P3 v3 P1 -- T1 Production Traces P2 -- T2 Production Traces P3 -- T3 Production Traces T1 -- E1 Evaluation T2 -- E2 Evaluation T3 -- E3 Evaluation E1 -- F Feedback E2 -- F E3 -- F F -- N Next Prompt Version Always make it possible to answer: Which prompt version produced this response? Treat prompts like code: - Version them - Test them - Evaluate them - Track them in production - Avoid silent production changes Every LLM invocation should expose useful usage information. php flowchart TD R User Request R -- L1 LLM Call 1 R -- L2 LLM Call 2 R -- L3 LLM Call 3 L1 -- T1 Tokens + Cost L2 -- T2 Tokens + Cost L3 -- T3 Tokens + Cost T1 -- TOTAL Trace Total T2 -- TOTAL T3 -- TOTAL TOTAL -- F Feature TOTAL -- U User TOTAL -- TEN Tenant TOTAL -- MODEL Model Track where available: input tokens output tokens cached tokens reasoning tokens Calculate: Cost / request Cost / trace Cost / user Cost / tenant Cost / feature Cost / model Cost / day Do not assume token counts alone tell you the cost. Different models and token categories can have different prices. Keep model pricing configuration versioned so historical cost calculations remain reproducible. End-to-end latency alone is not enough. php flowchart LR START Request -- RET Retrieval RET -- PROMPT Prompt Construction PROMPT -- LLM LLM Request LLM -- TTFT Time to First Token TTFT -- GEN Generation GEN -- TOOL Tool Execution TOOL -- POST Post Processing POST -- END Response Break latency down into: Total latency Retrieval latency Prompt construction LLM queue time Time to first token LLM generation time Tool execution Database time Post-processing For example: Total latency: 8.4s Retrieval: 420ms Prompt construction: 80ms LLM 1: 1.8s Tool call: 950ms LLM 2: 2.4s Tool call: 1.1s Final generation: 1.7s This makes performance bottlenecks visible. Retrieval-Augmented Generation needs its own observability layer. php flowchart TD Q User Query Q -- EMB Embedding EMB -- VS Vector Search VS -- DOCS Top K Documents DOCS -- RR Reranker RR -- CTX Final Context CTX -- LLM LLM LLM -- ANSWER Answer VS -- RM Retrieval Metrics RR -- RM RM -- E Evaluation ANSWER -- E Trace: query embedding model retriever top k documents retrieved similarity scores reranker final context retrieval latency empty-result rate This helps distinguish: Bad answer because of the model from: Bad answer because the model received bad context Tools should be treated as first-class observations. php flowchart TD A Agent A -- TC Tool Call TC -- NAME Tool Name TC -- VER Tool Version TC -- ARG Arguments TC -- EXEC Execution EXEC -- RESULT Result EXEC -- LAT Latency EXEC -- STATUS Status EXEC -- ERROR Error ERROR -- RETRY Retry RETRY -- TC Track: tool name tool version arguments result latency status error retry count This lets you answer: Which tools are called most frequently? Which tools fail most often? Which tools are slow? Which tools cause retries? Which tools are incorrectly selected? Agent loops are an important production failure mode. php flowchart TD START Agent Step -- LLM LLM LLM -- TOOL Tool Call TOOL -- RESULT Tool Result RESULT -- CHECK{Progress?} CHECK -- |Yes| LLM CHECK -- |No| LOOP Potential Loop LOOP -- LIMIT{Execution Limit?} LIMIT -- |No| RECOVERY Recovery Strategy LIMIT -- |Yes| STOP Stop Agent Monitor: iterations per trace tool calls per trace retries per trace duplicate tool calls maximum depth execution time cost per trace Production agents should have explicit limits, for example: max iterations = 20 max tool calls = 50 max execution time = 60s max cost = €0.50 Observability should help enforce and investigate these limits. LLM applications have multiple failure surfaces. php flowchart TD REQUEST Request REQUEST -- LLM LLM REQUEST -- RET Retrieval REQUEST -- TOOL Tool REQUEST -- DB Database REQUEST -- EXT External API LLM -- ERR Failure RET -- ERR TOOL -- ERR DB -- ERR EXT -- ERR ERR -- CLASSIFY Error Classification CLASSIFY -- TIMEOUT Timeout CLASSIFY -- RATE Rate Limit CLASSIFY -- AUTH Authentication CLASSIFY -- CONTEXT Context Limit CLASSIFY -- SCHEMA Schema Validation CLASSIFY -- PROVIDER Provider Failure CLASSIFY -- AGENT Agent Loop Capture structured failure information: provider model status code error type error message retry count trace id span id Useful categories include: Timeout Rate limit Authentication Invalid request Context window exceeded Tool failure Retrieval failure Schema validation failure Agent loop Guardrail violation Provider outage If an LLM generates JSON or another structured format, trace validation separately. php flowchart LR LLM LLM -- JSON Structured Response JSON -- VALIDATE Schema Validation VALIDATE -- |Valid| CONTINUE Continue VALIDATE -- |Invalid| REPAIR Retry / Repair REPAIR -- LLM Record: schema name schema version validation status validation errors repair attempts This is particularly important for: - AI APIs - Agent tools - Data extraction - Workflow automation - Healthcare systems - Financial systems Observability and evaluation answer different questions. php flowchart LR SYSTEM AI System SYSTEM -- TRACE Observability TRACE -- WHAT What happened? TRACE -- WHY Why did it happen? TRACE -- COST What did it cost? TRACE -- LAT How long did it take? SYSTEM -- EVAL Evaluation EVAL -- QUALITY Was it correct? EVAL -- RELEVANCE Was it relevant? EVAL -- GROUND Was it grounded? EVAL -- SUCCESS Was the task successful? Answers: What happened? Where did it happen? How long did it take? What did it cost? Answers: Was the answer correct? Was it relevant? Was it grounded? Was the task successful? The strongest AI engineering workflows connect both. Connect user feedback to traces. For example: 👍 👎 or: rating = 1..5 Associate feedback with: trace id model prompt version feature agent retrieval configuration This makes it possible to investigate: Which prompt version receives the most negative feedback? Which workflow has the highest task-success rate? Which model produces more successful outcomes? OpenTelemetry is an important vendor-neutral foundation for production observability. Use it for: Distributed traces Metrics Logs Context propagation Vendor-neutral instrumentation For AI systems, use standardized GenAI semantic conventions where practical instead of inventing application-specific attribute names for common telemetry. A useful architecture is: php flowchart TD APP AI Application APP -- OTEL OpenTelemetry OTEL -- TRACE Traces OTEL -- METRICS Metrics OTEL -- LOGS Logs TRACE -- COLLECTOR OpenTelemetry Collector METRICS -- COLLECTOR LOGS -- COLLECTOR COLLECTOR -- AI AI Observability Platform COLLECTOR -- MON Infrastructure Monitoring AI -- LLMTRACE LLM Tracing AI -- EVAL Evaluation AI -- PROMPTS Prompt Management AI -- COST Cost Tracking MON -- DASH Infrastructure Dashboards This helps reduce vendor lock-in. Use OpenTelemetry as the instrumentation and transport layer, then use specialized AI observability platforms where you need LLM-specific analysis. There is no universal best observability tool. Choose based on your architecture and requirements. | Tool | Typical use | |---|---| | OpenTelemetry | Vendor-neutral telemetry and distributed tracing | | Langfuse | LLM tracing, prompts, costs, evaluations, agent observability | | Arize Phoenix | LLM/RAG/agent tracing and evaluation | | Prometheus | Metrics | | Grafana | Dashboards and metrics visualization | | Datadog | Full-stack observability | | New Relic | Application and infrastructure observability | | Elastic | Logs, traces, metrics, and search | Choose tools based on requirements such as: Self-hosting Data residency Privacy requirements OpenTelemetry support Evaluation capabilities Prompt management Cost visibility Agent tracing RAG debugging Existing observability stack Observability creates another data surface. php flowchart LR APP AI Application APP -- TEL Telemetry TEL -- REDACT Redaction REDACT -- PII PII Filtering REDACT -- SECRET Secret Filtering REDACT -- POLICY Retention Policy PII -- STORE Observability Storage SECRET -- STORE POLICY -- STORE STORE -- ACCESS Access Control Potentially sensitive data includes: User prompts LLM responses Retrieved documents Tool arguments API responses Personal data Financial data Internal company information Never blindly log: API keys Access tokens Passwords Session cookies Authorization headers Private credentials Consider: PII redaction Field-level masking Data retention policies Encryption Access control Sampling Payload filtering Tenant isolation Observability must not become a new data-leakage surface. Not every production trace needs the same level of detail. php flowchart TD REQUEST Request REQUEST -- CHECK{Important Trace?} CHECK -- |Error| FULL 100% Trace CHECK -- |Slow| FULL CHECK -- |Expensive| FULL CHECK -- |Critical Workflow| FULL CHECK -- |Normal| SAMPLE Sample SAMPLE -- LOW Partial / Reduced Trace A practical strategy might be: Errors: 100% Slow requests: 100% Expensive requests: 100% Critical workflows: 100% Normal traffic: 5–20% The exact rate should depend on traffic volume, storage cost, privacy requirements, and debugging needs. A useful AI observability dashboard should cover four dimensions. mindmap root LLM Observability Reliability Error rate Timeout rate Retry rate Agent failures Tool failures Performance P50 latency P95 latency P99 latency TTFT Tool latency Retrieval latency Cost Cost/request Cost/user Cost/tenant Cost/feature Cost/model Quality Correctness Relevance Groundedness Task success User feedback Alert on meaningful changes such as: LLM error rate increases P95 latency increases Cost per request increases Token usage suddenly increases Tool failure rate increases Agent iteration count increases Retrieval empty-result rate increases Task-success rate decreases Evaluation score regresses Prefer alerts based on meaningful baselines and business impact rather than arbitrary thresholds. If you are starting from zero, do not try to implement everything at once. Start with: 1. Trace ID for every AI request 2. Individual LLM generation spans 3. Model name 4. Prompt version 5. Input/output tokens 6. Latency 7. Cost 8. Error tracking 9. Tool-call tracing for agents 10. Basic evaluation or user feedback A practical implementation path: Phase 1 → Traces + errors Phase 2 → Tokens + cost + latency Phase 3 → Tools + retrieval Phase 4 → Evaluation + feedback Phase 5 → Automated regression, cost, and quality alerts This gives you useful observability early without over-engineering the system. A useful way to think about observability maturity is: php flowchart LR A Level 1