{"slug": "llm-observability-production-best-practices-for-ai-agents-rag-tracing-tokens-and", "title": "LLM Observability: Production Best Practices for AI Agents, RAG, Tracing, Tokens, Costs, and Evaluation", "summary": "A developer outlined production best practices for LLM observability, arguing that logging only the final model response is insufficient for AI systems. The guidance recommends tracing the full execution path—prompts, token usage, costs, latency, retrieval, tool calls, agent trajectories, and evaluations—across three layers: system, LLM execution, and quality observability. The key principle stated is to \"trace the execution, not just the final response.", "body_md": "**LLM observability** is the practice of monitoring and tracing AI applications to understand LLM calls, prompts, token usage, latency, costs, tool calls, retrieval, agent trajectories, errors, and output quality in production.\n\nFor production AI systems, logging only the final LLM response is not enough.\n\nYou need visibility across:\n\n- LLM calls\n- Prompts\n- Tokens\n- Costs\n- Latency\n- Tool calls\n- Retrieval\n- Agent trajectories\n- Errors\n- Evaluations\n- User feedback\n\n**LLM observability** is the practice of instrumenting and monitoring generative AI systems so engineers can understand what happened during an AI request, why it happened, how long it took, how much it cost, where it failed, and whether the result was useful.\n\nUnlike traditional application observability, LLM observability must account for:\n\n```\nPrompts\nModel generations\nToken usage\nCosts\nRetrieval\nTool calls\nAgent trajectories\nEvaluations\nUser feedback\n```\n\nA production AI system is often a multi-step workflow rather than a single API call.\n\nTraditional software observability helps answer:\n\n```\nIs the service healthy?\nWhere did the request fail?\nWhich component is slow?\nHow many errors are occurring?\n```\n\nLLM observability adds questions such as:\n\n```\nWhy did the model produce this answer?\nWhich prompt version was used?\nWhat context did the model receive?\nWhich tools did the agent call?\nHow many tokens were consumed?\nHow much did the request cost?\nWhy did the agent retry?\nDid retrieval return useful documents?\nWas the final answer correct?\n```\n\nWithout this information, debugging production AI systems becomes guesswork.\n\nTraditional observability focuses primarily on:\n\n```\nLogs\nMetrics\nTraces\nInfrastructure\nErrors\n```\n\nLLM observability extends this model with:\n\n```\nPrompts\nModel generations\nToken usage\nCosts\nRetrieval\nTool calls\nAgent trajectories\nEvaluations\nUser feedback\n```\n\nThe key difference is that an LLM request is often only one step in a larger probabilistic workflow.\n\nA production AI system should trace the complete execution path.\n\n``` php\nflowchart TD\n    U[User Request] --> T[Trace]\n\n    T --> A[AI Application]\n\n    A --> R[Retrieval]\n    A --> L[LLM Generation]\n    A --> TO[Tool Call]\n\n    R --> RC[Retrieved Context]\n    L --> LR[Model Response]\n    TO --> TR[Tool Result]\n\n    RC --> L\n    LR --> A\n    TR --> A\n\n    A --> O[Final Output]\n\n    T --> M[Metrics]\n    T --> C[Cost Tracking]\n    T --> E[Evaluation]\n\n    M --> D[Observability Platform]\n    C --> D\n    E --> D\n```\n\nThe key principle:\n\n**Trace the execution, not just the final response.**\n\nProduction AI observability can be viewed as three complementary layers.\n\n``` php\nflowchart TD\n    A[AI Observability]\n\n    A --> B[System Observability]\n    A --> C[LLM Execution Observability]\n    A --> D[Quality Observability]\n\n    B --> B1[Infrastructure]\n    B --> B2[HTTP]\n    B --> B3[Database]\n    B --> B4[Queues]\n\n    C --> C1[Prompts]\n    C --> C2[Tokens]\n    C --> C3[Latency]\n    C --> C4[Tool Calls]\n    C --> C5[Agent Traces]\n    C --> C6[Retrieval]\n\n    D --> D1[Evaluation]\n    D --> D2[Groundedness]\n    D --> D3[Correctness]\n    D --> D4[Task Success]\n    D --> D5[User Feedback]\n```\n\nMonitors the software around the AI:\n\n```\nCPU\nMemory\nKubernetes\nDatabase\nNetwork\nHTTP\nQueues\nExternal APIs\n```\n\nMonitors how the AI system executes:\n\n```\nPrompts\nLLM generations\nTokens\nLatency\nCosts\nRetrieval\nTool calls\nAgent trajectories\n```\n\nMeasures whether the AI system actually works:\n\n```\nCorrectness\nRelevance\nGroundedness\nTask success\nUser feedback\nEvaluation scores\n```\n\nA mature AI system needs all three.\n\nA useful trace contains multiple types of observations.\n\n``` php\nflowchart LR\n    T[Trace]\n\n    T --> P[Prompt]\n    T --> G[LLM Generation]\n    T --> R[Retrieval]\n    T --> TC[Tool Call]\n    T --> A[Agent Step]\n    T --> E[Evaluation]\n\n    P --> PV[Prompt Version]\n\n    G --> TOK[Tokens]\n    G --> LAT[Latency]\n    G --> COST[Cost]\n    G --> MODEL[Model]\n\n    R --> DOCS[Documents]\n    R --> SCORE[Retrieval Scores]\n\n    TC --> ARG[Arguments]\n    TC --> RESULT[Result]\n    TC --> ERR[Errors]\n\n    A --> ITER[Iterations]\n    A --> TRAJ[Trajectory]\n\n    E --> QUALITY[Quality Score]\n```\n\nUseful request metadata:\n\n```\ntrace_id\nsession_id\nrequest_id\nuser_id\ntenant_id\nfeature\nenvironment\nregion\ntimestamp\n```\n\nFor example:\n\n```\n{\n  \"trace_id\": \"tr_8f31\",\n  \"session_id\": \"session_921\",\n  \"feature\": \"document_qa\",\n  \"environment\": \"production\",\n  \"tenant_id\": \"company_42\"\n}\n```\n\nUse metadata to answer:\n\n```\nWhich feature is expensive?\nWhich tenant generates the most traffic?\nWhich workflow has the highest failure rate?\nIs production slower than staging?\n```\n\nAn agent trajectory is the sequence of decisions and actions taken by an AI agent.\n\n```\nsequenceDiagram\n    participant User\n    participant Agent\n    participant LLM\n    participant Search\n    participant API\n\n    User->>Agent: User task\n\n    Agent->>LLM: Generate next action\n    LLM-->>Agent: Search required\n\n    Agent->>Search: search(query)\n    Search-->>Agent: Search results\n\n    Agent->>LLM: Analyze results\n    LLM-->>Agent: Call API\n\n    Agent->>API: execute(parameters)\n    API-->>Agent: API result\n\n    Agent->>LLM: Generate final response\n    LLM-->>Agent: Final answer\n\n    Agent-->>User: Response\n```\n\nFor production agents, the trace should preserve this entire sequence.\n\nA useful agent trace might look like:\n\n```\nAgent\n├── LLM generation #1\n├── Tool: search_documents\n├── Retrieval\n├── LLM generation #2\n├── Tool: create_ticket\n├── Tool: send_email\n└── Final generation\n```\n\nThis is much more useful than:\n\n```\nAgent\n└── LLM call\n```\n\nPrompts should be treated like production code.\n\nTrack:\n\n```\nprompt_name\nprompt_version\nsystem_prompt\nuser_prompt\ntemplate_variables\nmodel\ntemperature\nmax_tokens\nresponse_format\n```\n\nPrefer:\n\n```\ncustomer-support-agent:v12\n```\n\nover:\n\n```\ncustomer-support-agent\n```\n\nThis allows you to correlate:\n\n``` php\nflowchart LR\n    PV[Prompt Version]\n\n    PV --> P1[v1]\n    PV --> P2[v2]\n    PV --> P3[v3]\n\n    P1 --> T1[Production Traces]\n    P2 --> T2[Production Traces]\n    P3 --> T3[Production Traces]\n\n    T1 --> E1[Evaluation]\n    T2 --> E2[Evaluation]\n    T3 --> E3[Evaluation]\n\n    E1 --> F[Feedback]\n    E2 --> F\n    E3 --> F\n\n    F --> N[Next Prompt Version]\n```\n\nAlways make it possible to answer:\n\n**Which prompt version produced this response?**\n\nTreat prompts like code:\n\n- Version them\n- Test them\n- Evaluate them\n- Track them in production\n- Avoid silent production changes\n\nEvery LLM invocation should expose useful usage information.\n\n``` php\nflowchart TD\n    R[User Request]\n\n    R --> L1[LLM Call 1]\n    R --> L2[LLM Call 2]\n    R --> L3[LLM Call 3]\n\n    L1 --> T1[Tokens + Cost]\n    L2 --> T2[Tokens + Cost]\n    L3 --> T3[Tokens + Cost]\n\n    T1 --> TOTAL[Trace Total]\n    T2 --> TOTAL\n    T3 --> TOTAL\n\n    TOTAL --> F[Feature]\n    TOTAL --> U[User]\n    TOTAL --> TEN[Tenant]\n    TOTAL --> MODEL[Model]\n```\n\nTrack where available:\n\n```\ninput_tokens\noutput_tokens\ncached_tokens\nreasoning_tokens\n```\n\nCalculate:\n\n```\nCost / request\nCost / trace\nCost / user\nCost / tenant\nCost / feature\nCost / model\nCost / day\n```\n\nDo not assume token counts alone tell you the cost. Different models and token categories can have different prices.\n\nKeep model pricing configuration versioned so historical cost calculations remain reproducible.\n\nEnd-to-end latency alone is not enough.\n\n``` php\nflowchart LR\n    START[Request] --> RET[Retrieval]\n    RET --> PROMPT[Prompt Construction]\n    PROMPT --> LLM[LLM Request]\n\n    LLM --> TTFT[Time to First Token]\n    TTFT --> GEN[Generation]\n\n    GEN --> TOOL[Tool Execution]\n    TOOL --> POST[Post Processing]\n    POST --> END[Response]\n```\n\nBreak latency down into:\n\n```\nTotal latency\nRetrieval latency\nPrompt construction\nLLM queue time\nTime to first token\nLLM generation time\nTool execution\nDatabase time\nPost-processing\n```\n\nFor example:\n\n```\nTotal latency:        8.4s\nRetrieval:            420ms\nPrompt construction:   80ms\nLLM #1:               1.8s\nTool call:            950ms\nLLM #2:               2.4s\nTool call:            1.1s\nFinal generation:     1.7s\n```\n\nThis makes performance bottlenecks visible.\n\nRetrieval-Augmented Generation needs its own observability layer.\n\n``` php\nflowchart TD\n    Q[User Query]\n\n    Q --> EMB[Embedding]\n    EMB --> VS[Vector Search]\n\n    VS --> DOCS[Top K Documents]\n\n    DOCS --> RR[Reranker]\n\n    RR --> CTX[Final Context]\n\n    CTX --> LLM[LLM]\n\n    LLM --> ANSWER[Answer]\n\n    VS --> RM[Retrieval Metrics]\n    RR --> RM\n\n    RM --> E[Evaluation]\n    ANSWER --> E\n```\n\nTrace:\n\n```\nquery\nembedding model\nretriever\ntop_k\ndocuments retrieved\nsimilarity scores\nreranker\nfinal context\nretrieval latency\nempty-result rate\n```\n\nThis helps distinguish:\n\n```\nBad answer because of the model\n```\n\nfrom:\n\n```\nBad answer because the model received bad context\n```\n\nTools should be treated as first-class observations.\n\n``` php\nflowchart TD\n    A[Agent]\n\n    A --> TC[Tool Call]\n\n    TC --> NAME[Tool Name]\n    TC --> VER[Tool Version]\n    TC --> ARG[Arguments]\n\n    TC --> EXEC[Execution]\n\n    EXEC --> RESULT[Result]\n    EXEC --> LAT[Latency]\n    EXEC --> STATUS[Status]\n    EXEC --> ERROR[Error]\n\n    ERROR --> RETRY[Retry]\n\n    RETRY --> TC\n```\n\nTrack:\n\n```\ntool_name\ntool_version\narguments\nresult\nlatency\nstatus\nerror\nretry_count\n```\n\nThis lets you answer:\n\n```\nWhich tools are called most frequently?\nWhich tools fail most often?\nWhich tools are slow?\nWhich tools cause retries?\nWhich tools are incorrectly selected?\n```\n\nAgent loops are an important production failure mode.\n\n``` php\nflowchart TD\n    START[Agent Step] --> LLM[LLM]\n\n    LLM --> TOOL[Tool Call]\n    TOOL --> RESULT[Tool Result]\n\n    RESULT --> CHECK{Progress?}\n\n    CHECK -->|Yes| LLM\n    CHECK -->|No| LOOP[Potential Loop]\n\n    LOOP --> LIMIT{Execution Limit?}\n\n    LIMIT -->|No| RECOVERY[Recovery Strategy]\n    LIMIT -->|Yes| STOP[Stop Agent]\n```\n\nMonitor:\n\n```\niterations_per_trace\ntool_calls_per_trace\nretries_per_trace\nduplicate_tool_calls\nmaximum_depth\nexecution_time\ncost_per_trace\n```\n\nProduction agents should have explicit limits, for example:\n\n```\nmax_iterations = 20\nmax_tool_calls = 50\nmax_execution_time = 60s\nmax_cost = €0.50\n```\n\nObservability should help enforce and investigate these limits.\n\nLLM applications have multiple failure surfaces.\n\n``` php\nflowchart TD\n    REQUEST[Request]\n\n    REQUEST --> LLM[LLM]\n    REQUEST --> RET[Retrieval]\n    REQUEST --> TOOL[Tool]\n    REQUEST --> DB[Database]\n    REQUEST --> EXT[External API]\n\n    LLM --> ERR[Failure]\n    RET --> ERR\n    TOOL --> ERR\n    DB --> ERR\n    EXT --> ERR\n\n    ERR --> CLASSIFY[Error Classification]\n\n    CLASSIFY --> TIMEOUT[Timeout]\n    CLASSIFY --> RATE[Rate Limit]\n    CLASSIFY --> AUTH[Authentication]\n    CLASSIFY --> CONTEXT[Context Limit]\n    CLASSIFY --> SCHEMA[Schema Validation]\n    CLASSIFY --> PROVIDER[Provider Failure]\n    CLASSIFY --> AGENT[Agent Loop]\n```\n\nCapture structured failure information:\n\n```\nprovider\nmodel\nstatus_code\nerror_type\nerror_message\nretry_count\ntrace_id\nspan_id\n```\n\nUseful categories include:\n\n```\nTimeout\nRate limit\nAuthentication\nInvalid request\nContext window exceeded\nTool failure\nRetrieval failure\nSchema validation failure\nAgent loop\nGuardrail violation\nProvider outage\n```\n\nIf an LLM generates JSON or another structured format, trace validation separately.\n\n``` php\nflowchart LR\n    LLM[LLM] --> JSON[Structured Response]\n    JSON --> VALIDATE[Schema Validation]\n\n    VALIDATE -->|Valid| CONTINUE[Continue]\n    VALIDATE -->|Invalid| REPAIR[Retry / Repair]\n\n    REPAIR --> LLM\n```\n\nRecord:\n\n```\nschema_name\nschema_version\nvalidation_status\nvalidation_errors\nrepair_attempts\n```\n\nThis is particularly important for:\n\n- AI APIs\n- Agent tools\n- Data extraction\n- Workflow automation\n- Healthcare systems\n- Financial systems\n\nObservability and evaluation answer different questions.\n\n``` php\nflowchart LR\n    SYSTEM[AI System]\n\n    SYSTEM --> TRACE[Observability]\n\n    TRACE --> WHAT[What happened?]\n    TRACE --> WHY[Why did it happen?]\n    TRACE --> COST[What did it cost?]\n    TRACE --> LAT[How long did it take?]\n\n    SYSTEM --> EVAL[Evaluation]\n\n    EVAL --> QUALITY[Was it correct?]\n    EVAL --> RELEVANCE[Was it relevant?]\n    EVAL --> GROUND[Was it grounded?]\n    EVAL --> SUCCESS[Was the task successful?]\n```\n\nAnswers:\n\n```\nWhat happened?\nWhere did it happen?\nHow long did it take?\nWhat did it cost?\n```\n\nAnswers:\n\n```\nWas the answer correct?\nWas it relevant?\nWas it grounded?\nWas the task successful?\n```\n\nThe strongest AI engineering workflows connect both.\n\nConnect user feedback to traces.\n\nFor example:\n\n```\n👍\n👎\n```\n\nor:\n\n```\nrating = 1..5\n```\n\nAssociate feedback with:\n\n```\ntrace_id\nmodel\nprompt_version\nfeature\nagent\nretrieval configuration\n```\n\nThis makes it possible to investigate:\n\n```\nWhich prompt version receives the most negative feedback?\nWhich workflow has the highest task-success rate?\nWhich model produces more successful outcomes?\n```\n\n**OpenTelemetry** is an important vendor-neutral foundation for production observability.\n\nUse it for:\n\n```\nDistributed traces\nMetrics\nLogs\nContext propagation\nVendor-neutral instrumentation\n```\n\nFor AI systems, use standardized GenAI semantic conventions where practical instead of inventing application-specific attribute names for common telemetry.\n\nA useful architecture is:\n\n``` php\nflowchart TD\n    APP[AI Application]\n\n    APP --> OTEL[OpenTelemetry]\n\n    OTEL --> TRACE[Traces]\n    OTEL --> METRICS[Metrics]\n    OTEL --> LOGS[Logs]\n\n    TRACE --> COLLECTOR[OpenTelemetry Collector]\n    METRICS --> COLLECTOR\n    LOGS --> COLLECTOR\n\n    COLLECTOR --> AI[AI Observability Platform]\n    COLLECTOR --> MON[Infrastructure Monitoring]\n\n    AI --> LLMTRACE[LLM Tracing]\n    AI --> EVAL[Evaluation]\n    AI --> PROMPTS[Prompt Management]\n    AI --> COST[Cost Tracking]\n\n    MON --> DASH[Infrastructure Dashboards]\n```\n\nThis helps reduce vendor lock-in.\n\nUse OpenTelemetry as the instrumentation and transport layer, then use specialized AI observability platforms where you need LLM-specific analysis.\n\nThere is no universal best observability tool.\n\nChoose based on your architecture and requirements.\n\n| Tool | Typical use | \n|---|---|\n| **OpenTelemetry** | Vendor-neutral telemetry and distributed tracing | \n| **Langfuse** | LLM tracing, prompts, costs, evaluations, agent observability | \n| **Arize Phoenix** | LLM/RAG/agent tracing and evaluation | \n| **Prometheus** | Metrics | \n| **Grafana** | Dashboards and metrics visualization | \n| **Datadog** | Full-stack observability | \n| **New Relic** | Application and infrastructure observability | \n| **Elastic** | Logs, traces, metrics, and search | \n\nChoose tools based on requirements such as:\n\n```\nSelf-hosting\nData residency\nPrivacy requirements\nOpenTelemetry support\nEvaluation capabilities\nPrompt management\nCost visibility\nAgent tracing\nRAG debugging\nExisting observability stack\n```\n\nObservability creates another data surface.\n\n``` php\nflowchart LR\n    APP[AI Application]\n\n    APP --> TEL[Telemetry]\n\n    TEL --> REDACT[Redaction]\n\n    REDACT --> PII[PII Filtering]\n    REDACT --> SECRET[Secret Filtering]\n    REDACT --> POLICY[Retention Policy]\n\n    PII --> STORE[Observability Storage]\n    SECRET --> STORE\n    POLICY --> STORE\n\n    STORE --> ACCESS[Access Control]\n```\n\nPotentially sensitive data includes:\n\n```\nUser prompts\nLLM responses\nRetrieved documents\nTool arguments\nAPI responses\nPersonal data\nFinancial data\nInternal company information\n```\n\nNever blindly log:\n\n```\nAPI keys\nAccess tokens\nPasswords\nSession cookies\nAuthorization headers\nPrivate credentials\n```\n\nConsider:\n\n```\nPII redaction\nField-level masking\nData retention policies\nEncryption\nAccess control\nSampling\nPayload filtering\nTenant isolation\n```\n\nObservability must not become a new data-leakage surface.\n\nNot every production trace needs the same level of detail.\n\n``` php\nflowchart TD\n    REQUEST[Request]\n\n    REQUEST --> CHECK{Important Trace?}\n\n    CHECK -->|Error| FULL[100% Trace]\n    CHECK -->|Slow| FULL\n    CHECK -->|Expensive| FULL\n    CHECK -->|Critical Workflow| FULL\n\n    CHECK -->|Normal| SAMPLE[Sample]\n\n    SAMPLE --> LOW[Partial / Reduced Trace]\n```\n\nA practical strategy might be:\n\n```\nErrors:              100%\nSlow requests:       100%\nExpensive requests:  100%\nCritical workflows:  100%\nNormal traffic:       5–20%\n```\n\nThe exact rate should depend on traffic volume, storage cost, privacy requirements, and debugging needs.\n\nA useful AI observability dashboard should cover four dimensions.\n\n```\nmindmap\n  root((LLM Observability))\n    Reliability\n      Error rate\n      Timeout rate\n      Retry rate\n      Agent failures\n      Tool failures\n    Performance\n      P50 latency\n      P95 latency\n      P99 latency\n      TTFT\n      Tool latency\n      Retrieval latency\n    Cost\n      Cost/request\n      Cost/user\n      Cost/tenant\n      Cost/feature\n      Cost/model\n    Quality\n      Correctness\n      Relevance\n      Groundedness\n      Task success\n      User feedback\n```\n\nAlert on meaningful changes such as:\n\n```\nLLM error rate increases\nP95 latency increases\nCost per request increases\nToken usage suddenly increases\nTool failure rate increases\nAgent iteration count increases\nRetrieval empty-result rate increases\nTask-success rate decreases\nEvaluation score regresses\n```\n\nPrefer alerts based on meaningful baselines and business impact rather than arbitrary thresholds.\n\nIf you are starting from zero, do not try to implement everything at once.\n\nStart with:\n\n1. Trace ID for every AI request\n2. Individual LLM generation spans\n3. Model name\n4. Prompt version\n5. Input/output tokens\n6. Latency\n7. Cost\n8. Error tracking\n9. Tool-call tracing for agents\n10. Basic evaluation or user feedback\n\nA practical implementation path:\n\n```\nPhase 1 → Traces + errors\nPhase 2 → Tokens + cost + latency\nPhase 3 → Tools + retrieval\nPhase 4 → Evaluation + feedback\nPhase 5 → Automated regression, cost, and quality alerts\n```\n\nThis gives you useful observability early without over-engineering the system.\n\nA useful way to think about observability maturity is:\n\n``` php\nflowchart LR\n    A[Level 1<br/>Logging] --> B[Level 2<br/>Tracing]\n    B --> C[Level 3<br/>Cost & Performance]\n    C --> D[Level 4<br/>Evaluation]\n    D --> E[Level 5<br/>Continuous Optimization]\nErrors\nRequests\nBasic model information\nLLM calls\nTool calls\nRetrieval\nAgent steps\nTokens\nCosts\nLatency\nTTFT\nResource usage\nCorrectness\nRelevance\nGroundedness\nTask success\nUser feedback\nAutomated regression detection\nPrompt experiments\nModel experiments\nQuality alerts\nCost optimization\nContinuous evaluation\n```\n\nThe goal is not necessarily to implement every feature immediately. The maturity model helps prioritize what to build next.\n\n- Every important request has a trace ID\n- Every LLM call is traced\n- Tool calls are traced\n- Retrieval is traced\n- Agent iterations are visible\n- External API calls are correlated\n\n- Prompts are versioned\n- Prompt version is stored in traces\n- Prompt changes can be compared\n\n- Input tokens are tracked\n- Output tokens are tracked\n- Cached/reasoning tokens are tracked when available\n- Cost is calculated\n- Cost can be grouped by feature, model, and tenant\n\n- P50/P95/P99 latency is monitored\n- Time-to-first-token is tracked\n- Slow spans are identifiable\n- Retrieval and tool latency are visible\n\n- Agent trajectories are traceable\n- Tool calls are visible\n- Retries are tracked\n- Loops are detectable\n- Maximum iterations are enforced\n- Maximum execution time is enforced\n- Cost limits exist\n\n- Retrieval queries are traced\n- Retrieved documents are visible\n- Similarity scores are available\n- Reranking is observable\n- Context size is tracked\n- Empty retrieval results are monitored\n\n- Secrets are removed\n- PII is protected\n- Trace access is controlled\n- Retention policies exist\n- Sensitive payloads are sampled or redacted\n\n- Traces can be evaluated\n- User feedback is connected to traces\n- Failed traces can become evaluation datasets\n- Prompt/model changes can be evaluated\n- Quality regressions can trigger alerts\n\nThe ultimate goal is not simply collecting traces.\n\nIt is continuous improvement.\n\n``` php\nflowchart TD\n    PROD[Production AI System]\n\n    PROD --> TRACE[Tracing]\n    TRACE --> EVAL[Evaluation]\n\n    EVAL --> ANALYZE[Failure Analysis]\n\n    ANALYZE --> DATASET[Evaluation Dataset]\n\n    DATASET --> EXP[Experiment]\n\n    EXP --> PROMPT[Prompt Change]\n    EXP --> MODEL[Model Change]\n    EXP --> RETRIEVAL[Retrieval Change]\n    EXP --> TOOL[Tool Change]\n\n    PROMPT --> DEPLOY[Deploy]\n    MODEL --> DEPLOY\n    RETRIEVAL --> DEPLOY\n    TOOL --> DEPLOY\n\n    DEPLOY --> PROD\n```\n\nThe engineering loop becomes:\n\n```\nObserve\n   ↓\nEvaluate\n   ↓\nDiagnose\n   ↓\nImprove\n   ↓\nDeploy\n   ↓\nObserve\n```\n\nThis is where observability becomes an AI engineering capability rather than just another monitoring dashboard.\n\n**LLM observability is the foundation for operating AI systems in production.**\n\nTraditional observability tells you whether your service is running.\n\nLLM observability tells you:\n\n```\nWhat happened?\nWhy did it happen?\nWhich model was used?\nWhich prompt version was used?\nWhat context did it receive?\nWhich tools were called?\nHow many tokens were consumed?\nHow much did it cost?\nHow long did it take?\nWhere did it fail?\nWas the result actually good?\n```\n\nFor simple LLM applications:\n\n```\nOpenTelemetry\n+\nLLM tracing\n+\nToken tracking\n+\nCost tracking\n+\nError monitoring\n```\n\nFor RAG:\n\n```\nRetrieval tracing\n+\nContext inspection\n+\nRAG evaluation\n```\n\nFor AI agents:\n\n```\nAgent trajectory tracing\n+\nTool-call observability\n+\nLoop detection\n+\nEvaluation\n+\nCost controls\n```\n\nThe goal is not to collect more telemetry.\n\nThe goal is to make **AI behavior observable, measurable, debuggable, secure, and continuously improvable.**\n\n**If you cannot reconstruct what an AI system did, you cannot reliably debug, optimize, or operate it in production.**\n\n`LLM Observability` · `AI Observability` · `LLM Tracing` · `AI Agents` · `RAG` · `OpenTelemetry` · `LLM Evaluation` · `LLM Cost Tracking` · `Production AI`", "url": "https://wpnews.pro/news/llm-observability-production-best-practices-for-ai-agents-rag-tracing-tokens-and", "canonical_source": "https://gist.github.com/meghrazchi/d2d71d89b8bf230543ddc8e68fbb610a", "published_at": "2026-09-16 01:37:57+00:00", "updated_at": "2026-09-16 11:41:08.891650+00:00", "lang": "en", "topics": ["ai-agents", "mlops", "large-language-models", "ai-tools", "developer-tools"], "entities": [], "alternates": {"html": "https://wpnews.pro/news/llm-observability-production-best-practices-for-ai-agents-rag-tracing-tokens-and", "markdown": "https://wpnews.pro/news/llm-observability-production-best-practices-for-ai-agents-rag-tracing-tokens-and.md", "text": "https://wpnews.pro/news/llm-observability-production-best-practices-for-ai-agents-rag-tracing-tokens-and.txt", "jsonld": "https://wpnews.pro/news/llm-observability-production-best-practices-for-ai-agents-rag-tracing-tokens-and.jsonld"}}