LLM observability is the practice of monitoring and tracing AI applications to understand LLM calls, prompts, token usage, latency, costs, tool calls, retrieval, agent trajectories, errors, and output quality in production.
For production AI systems, logging only the final LLM response is not enough.
You need visibility across:
- LLM calls
- Prompts
- Tokens
- Costs
- Latency
- Tool calls
- Retrieval
- Agent trajectories
- Errors
- Evaluations
- User feedback
LLM observability is the practice of instrumenting and monitoring generative AI systems so engineers can understand what happened during an AI request, why it happened, how long it took, how much it cost, where it failed, and whether the result was useful.
Unlike traditional application observability, LLM observability must account for:
Prompts
Model generations
Token usage
Costs
Retrieval
Tool calls
Agent trajectories
Evaluations
User feedback
A production AI system is often a multi-step workflow rather than a single API call.
Traditional software observability helps answer:
Is the service healthy?
Where did the request fail?
Which component is slow?
How many errors are occurring?
LLM observability adds questions such as:
Why did the model produce this answer?
Which prompt version was used?
What context did the model receive?
Which tools did the agent call?
How many tokens were consumed?
How much did the request cost?
Why did the agent retry?
Did retrieval return useful documents?
Was the final answer correct?
Without this information, debugging production AI systems becomes guesswork.
Traditional observability focuses primarily on:
Logs
Metrics
Traces
Infrastructure
Errors
LLM observability extends this model with:
Prompts
Model generations
Token usage
Costs
Retrieval
Tool calls
Agent trajectories
Evaluations
User feedback
The key difference is that an LLM request is often only one step in a larger probabilistic workflow.
A production AI system should trace the complete execution path.
flowchart TD
U[User Request] --> T[Trace]
T --> A[AI Application]
A --> R[Retrieval]
A --> L[LLM Generation]
A --> TO[Tool Call]
R --> RC[Retrieved Context]
L --> LR[Model Response]
TO --> TR[Tool Result]
RC --> L
LR --> A
TR --> A
A --> O[Final Output]
T --> M[Metrics]
T --> C[Cost Tracking]
T --> E[Evaluation]
M --> D[Observability Platform]
C --> D
E --> D
The key principle:
Trace the execution, not just the final response.
Production AI observability can be viewed as three complementary layers.
flowchart TD
A[AI Observability]
A --> B[System Observability]
A --> C[LLM Execution Observability]
A --> D[Quality Observability]
B --> B1[Infrastructure]
B --> B2[HTTP]
B --> B3[Database]
B --> B4[Queues]
C --> C1[Prompts]
C --> C2[Tokens]
C --> C3[Latency]
C --> C4[Tool Calls]
C --> C5[Agent Traces]
C --> C6[Retrieval]
D --> D1[Evaluation]
D --> D2[Groundedness]
D --> D3[Correctness]
D --> D4[Task Success]
D --> D5[User Feedback]
Monitors the software around the AI:
CPU
Memory
Kubernetes
Database
Network
HTTP
Queues
External APIs
Monitors how the AI system executes:
Prompts
LLM generations
Tokens
Latency
Costs
Retrieval
Tool calls
Agent trajectories
Measures whether the AI system actually works:
Correctness
Relevance
Groundedness
Task success
User feedback
Evaluation scores
A mature AI system needs all three.
A useful trace contains multiple types of observations.
flowchart LR
T[Trace]
T --> P[Prompt]
T --> G[LLM Generation]
T --> R[Retrieval]
T --> TC[Tool Call]
T --> A[Agent Step]
T --> E[Evaluation]
P --> PV[Prompt Version]
G --> TOK[Tokens]
G --> LAT[Latency]
G --> COST[Cost]
G --> MODEL[Model]
R --> DOCS[Documents]
R --> SCORE[Retrieval Scores]
TC --> ARG[Arguments]
TC --> RESULT[Result]
TC --> ERR[Errors]
A --> ITER[Iterations]
A --> TRAJ[Trajectory]
E --> QUALITY[Quality Score]
Useful request metadata:
trace_id
session_id
request_id
user_id
tenant_id
feature
environment
region
timestamp
For example:
{
"trace_id": "tr_8f31",
"session_id": "session_921",
"feature": "document_qa",
"environment": "production",
"tenant_id": "company_42"
}
Use metadata to answer:
Which feature is expensive?
Which tenant generates the most traffic?
Which workflow has the highest failure rate?
Is production slower than staging?
An agent trajectory is the sequence of decisions and actions taken by an AI agent.
sequenceDiagram
participant User
participant Agent
participant LLM
participant Search
participant API
User->>Agent: User task
Agent->>LLM: Generate next action
LLM-->>Agent: Search required
Agent->>Search: search(query)
Search-->>Agent: Search results
Agent->>LLM: Analyze results
LLM-->>Agent: Call API
Agent->>API: execute(parameters)
API-->>Agent: API result
Agent->>LLM: Generate final response
LLM-->>Agent: Final answer
Agent-->>User: Response
For production agents, the trace should preserve this entire sequence.
A useful agent trace might look like:
Agent
├── LLM generation #1
├── Tool: search_documents
├── Retrieval
├── LLM generation #2
├── Tool: create_ticket
├── Tool: send_email
└── Final generation
This is much more useful than:
Agent
└── LLM call
Prompts should be treated like production code.
Track:
prompt_name
prompt_version
system_prompt
user_prompt
template_variables
model
temperature
max_tokens
response_format
Prefer:
customer-support-agent:v12
over:
customer-support-agent
This allows you to correlate:
flowchart LR
PV[Prompt Version]
PV --> P1[v1]
PV --> P2[v2]
PV --> P3[v3]
P1 --> T1[Production Traces]
P2 --> T2[Production Traces]
P3 --> T3[Production Traces]
T1 --> E1[Evaluation]
T2 --> E2[Evaluation]
T3 --> E3[Evaluation]
E1 --> F[Feedback]
E2 --> F
E3 --> F
F --> N[Next Prompt Version]
Always make it possible to answer:
Which prompt version produced this response?
Treat prompts like code:
- Version them
- Test them
- Evaluate them
- Track them in production
- Avoid silent production changes
Every LLM invocation should expose useful usage information.
flowchart TD
R[User Request]
R --> L1[LLM Call 1]
R --> L2[LLM Call 2]
R --> L3[LLM Call 3]
L1 --> T1[Tokens + Cost]
L2 --> T2[Tokens + Cost]
L3 --> T3[Tokens + Cost]
T1 --> TOTAL[Trace Total]
T2 --> TOTAL
T3 --> TOTAL
TOTAL --> F[Feature]
TOTAL --> U[User]
TOTAL --> TEN[Tenant]
TOTAL --> MODEL[Model]
Track where available:
input_tokens
output_tokens
cached_tokens
reasoning_tokens
Calculate:
Cost / request
Cost / trace
Cost / user
Cost / tenant
Cost / feature
Cost / model
Cost / day
Do not assume token counts alone tell you the cost. Different models and token categories can have different prices.
Keep model pricing configuration versioned so historical cost calculations remain reproducible.
End-to-end latency alone is not enough.
flowchart LR
START[Request] --> RET[Retrieval]
RET --> PROMPT[Prompt Construction]
PROMPT --> LLM[LLM Request]
LLM --> TTFT[Time to First Token]
TTFT --> GEN[Generation]
GEN --> TOOL[Tool Execution]
TOOL --> POST[Post Processing]
POST --> END[Response]
Break latency down into:
Total latency
Retrieval latency
Prompt construction
LLM queue time
Time to first token
LLM generation time
Tool execution
Database time
Post-processing
For example:
Total latency: 8.4s
Retrieval: 420ms
Prompt construction: 80ms
LLM #1: 1.8s
Tool call: 950ms
LLM #2: 2.4s
Tool call: 1.1s
Final generation: 1.7s
This makes performance bottlenecks visible.
Retrieval-Augmented Generation needs its own observability layer.
flowchart TD
Q[User Query]
Q --> EMB[Embedding]
EMB --> VS[Vector Search]
VS --> DOCS[Top K Documents]
DOCS --> RR[Reranker]
RR --> CTX[Final Context]
CTX --> LLM[LLM]
LLM --> ANSWER[Answer]
VS --> RM[Retrieval Metrics]
RR --> RM
RM --> E[Evaluation]
ANSWER --> E
Trace:
query
embedding model
retriever
top_k
documents retrieved
similarity scores
reranker
final context
retrieval latency
empty-result rate
This helps distinguish:
Bad answer because of the model
from:
Bad answer because the model received bad context
Tools should be treated as first-class observations.
flowchart TD
A[Agent]
A --> TC[Tool Call]
TC --> NAME[Tool Name]
TC --> VER[Tool Version]
TC --> ARG[Arguments]
TC --> EXEC[Execution]
EXEC --> RESULT[Result]
EXEC --> LAT[Latency]
EXEC --> STATUS[Status]
EXEC --> ERROR[Error]
ERROR --> RETRY[Retry]
RETRY --> TC
Track:
tool_name
tool_version
arguments
result
latency
status
error
retry_count
This lets you answer:
Which tools are called most frequently?
Which tools fail most often?
Which tools are slow?
Which tools cause retries?
Which tools are incorrectly selected?
Agent loops are an important production failure mode.
flowchart TD
START[Agent Step] --> LLM[LLM]
LLM --> TOOL[Tool Call]
TOOL --> RESULT[Tool Result]
RESULT --> CHECK{Progress?}
CHECK -->|Yes| LLM
CHECK -->|No| LOOP[Potential Loop]
LOOP --> LIMIT{Execution Limit?}
LIMIT -->|No| RECOVERY[Recovery Strategy]
LIMIT -->|Yes| STOP[Stop Agent]
Monitor:
iterations_per_trace
tool_calls_per_trace
retries_per_trace
duplicate_tool_calls
maximum_depth
execution_time
cost_per_trace
Production agents should have explicit limits, for example:
max_iterations = 20
max_tool_calls = 50
max_execution_time = 60s
max_cost = €0.50
Observability should help enforce and investigate these limits.
LLM applications have multiple failure surfaces.
flowchart TD
REQUEST[Request]
REQUEST --> LLM[LLM]
REQUEST --> RET[Retrieval]
REQUEST --> TOOL[Tool]
REQUEST --> DB[Database]
REQUEST --> EXT[External API]
LLM --> ERR[Failure]
RET --> ERR
TOOL --> ERR
DB --> ERR
EXT --> ERR
ERR --> CLASSIFY[Error Classification]
CLASSIFY --> TIMEOUT[Timeout]
CLASSIFY --> RATE[Rate Limit]
CLASSIFY --> AUTH[Authentication]
CLASSIFY --> CONTEXT[Context Limit]
CLASSIFY --> SCHEMA[Schema Validation]
CLASSIFY --> PROVIDER[Provider Failure]
CLASSIFY --> AGENT[Agent Loop]
Capture structured failure information:
provider
model
status_code
error_type
error_message
retry_count
trace_id
span_id
Useful categories include:
Timeout
Rate limit
Authentication
Invalid request
Context window exceeded
Tool failure
Retrieval failure
Schema validation failure
Agent loop
Guardrail violation
Provider outage
If an LLM generates JSON or another structured format, trace validation separately.
flowchart LR
LLM[LLM] --> JSON[Structured Response]
JSON --> VALIDATE[Schema Validation]
VALIDATE -->|Valid| CONTINUE[Continue]
VALIDATE -->|Invalid| REPAIR[Retry / Repair]
REPAIR --> LLM
Record:
schema_name
schema_version
validation_status
validation_errors
repair_attempts
This is particularly important for:
- AI APIs
- Agent tools
- Data extraction
- Workflow automation
- Healthcare systems
- Financial systems
Observability and evaluation answer different questions.
flowchart LR
SYSTEM[AI System]
SYSTEM --> TRACE[Observability]
TRACE --> WHAT[What happened?]
TRACE --> WHY[Why did it happen?]
TRACE --> COST[What did it cost?]
TRACE --> LAT[How long did it take?]
SYSTEM --> EVAL[Evaluation]
EVAL --> QUALITY[Was it correct?]
EVAL --> RELEVANCE[Was it relevant?]
EVAL --> GROUND[Was it grounded?]
EVAL --> SUCCESS[Was the task successful?]
Answers:
What happened?
Where did it happen?
How long did it take?
What did it cost?
Answers:
Was the answer correct?
Was it relevant?
Was it grounded?
Was the task successful?
The strongest AI engineering workflows connect both.
Connect user feedback to traces.
For example:
👍
👎
or:
rating = 1..5
Associate feedback with:
trace_id
model
prompt_version
feature
agent
retrieval configuration
This makes it possible to investigate:
Which prompt version receives the most negative feedback?
Which workflow has the highest task-success rate?
Which model produces more successful outcomes?
OpenTelemetry is an important vendor-neutral foundation for production observability.
Use it for:
Distributed traces
Metrics
Logs
Context propagation
Vendor-neutral instrumentation
For AI systems, use standardized GenAI semantic conventions where practical instead of inventing application-specific attribute names for common telemetry.
A useful architecture is:
flowchart TD
APP[AI Application]
APP --> OTEL[OpenTelemetry]
OTEL --> TRACE[Traces]
OTEL --> METRICS[Metrics]
OTEL --> LOGS[Logs]
TRACE --> COLLECTOR[OpenTelemetry Collector]
METRICS --> COLLECTOR
LOGS --> COLLECTOR
COLLECTOR --> AI[AI Observability Platform]
COLLECTOR --> MON[Infrastructure Monitoring]
AI --> LLMTRACE[LLM Tracing]
AI --> EVAL[Evaluation]
AI --> PROMPTS[Prompt Management]
AI --> COST[Cost Tracking]
MON --> DASH[Infrastructure Dashboards]
This helps reduce vendor lock-in.
Use OpenTelemetry as the instrumentation and transport layer, then use specialized AI observability platforms where you need LLM-specific analysis.
There is no universal best observability tool.
Choose based on your architecture and requirements.
| Tool | Typical use |
|---|---|
| OpenTelemetry | Vendor-neutral telemetry and distributed tracing |
| Langfuse | LLM tracing, prompts, costs, evaluations, agent observability |
| Arize Phoenix | LLM/RAG/agent tracing and evaluation |
| Prometheus | Metrics |
| Grafana | Dashboards and metrics visualization |
| Datadog | Full-stack observability |
| New Relic | Application and infrastructure observability |
| Elastic | Logs, traces, metrics, and search |
Choose tools based on requirements such as:
Self-hosting
Data residency
Privacy requirements
OpenTelemetry support
Evaluation capabilities
Prompt management
Cost visibility
Agent tracing
RAG debugging
Existing observability stack
Observability creates another data surface.
flowchart LR
APP[AI Application]
APP --> TEL[Telemetry]
TEL --> REDACT[Redaction]
REDACT --> PII[PII Filtering]
REDACT --> SECRET[Secret Filtering]
REDACT --> POLICY[Retention Policy]
PII --> STORE[Observability Storage]
SECRET --> STORE
POLICY --> STORE
STORE --> ACCESS[Access Control]
Potentially sensitive data includes:
User prompts
LLM responses
Retrieved documents
Tool arguments
API responses
Personal data
Financial data
Internal company information
Never blindly log:
API keys
Access tokens
Passwords
Session cookies
Authorization headers
Private credentials
Consider:
PII redaction
Field-level masking
Data retention policies
Encryption
Access control
Sampling
Payload filtering
Tenant isolation
Observability must not become a new data-leakage surface.
Not every production trace needs the same level of detail.
flowchart TD
REQUEST[Request]
REQUEST --> CHECK{Important Trace?}
CHECK -->|Error| FULL[100% Trace]
CHECK -->|Slow| FULL
CHECK -->|Expensive| FULL
CHECK -->|Critical Workflow| FULL
CHECK -->|Normal| SAMPLE[Sample]
SAMPLE --> LOW[Partial / Reduced Trace]
A practical strategy might be:
Errors: 100%
Slow requests: 100%
Expensive requests: 100%
Critical workflows: 100%
Normal traffic: 5–20%
The exact rate should depend on traffic volume, storage cost, privacy requirements, and debugging needs.
A useful AI observability dashboard should cover four dimensions.
mindmap
root((LLM Observability))
Reliability
Error rate
Timeout rate
Retry rate
Agent failures
Tool failures
Performance
P50 latency
P95 latency
P99 latency
TTFT
Tool latency
Retrieval latency
Cost
Cost/request
Cost/user
Cost/tenant
Cost/feature
Cost/model
Quality
Correctness
Relevance
Groundedness
Task success
User feedback
Alert on meaningful changes such as:
LLM error rate increases
P95 latency increases
Cost per request increases
Token usage suddenly increases
Tool failure rate increases
Agent iteration count increases
Retrieval empty-result rate increases
Task-success rate decreases
Evaluation score regresses
Prefer alerts based on meaningful baselines and business impact rather than arbitrary thresholds.
If you are starting from zero, do not try to implement everything at once.
Start with:
- Trace ID for every AI request
- Individual LLM generation spans
- Model name
- Prompt version
- Input/output tokens
- Latency
- Cost
- Error tracking
- Tool-call tracing for agents
- Basic evaluation or user feedback
A practical implementation path:
Phase 1 → Traces + errors
Phase 2 → Tokens + cost + latency
Phase 3 → Tools + retrieval
Phase 4 → Evaluation + feedback
Phase 5 → Automated regression, cost, and quality alerts
This gives you useful observability early without over-engineering the system.
A useful way to think about observability maturity is:
flowchart LR
A[Level 1<br/>Logging] --> B[Level 2<br/>Tracing]
B --> C[Level 3<br/>Cost & Performance]
C --> D[Level 4<br/>Evaluation]
D --> E[Level 5<br/>Continuous Optimization]
Errors
Requests
Basic model information
LLM calls
Tool calls
Retrieval
Agent steps
Tokens
Costs
Latency
TTFT
Resource usage
Correctness
Relevance
Groundedness
Task success
User feedback
Automated regression detection
Prompt experiments
Model experiments
Quality alerts
Cost optimization
Continuous evaluation
The goal is not necessarily to implement every feature immediately. The maturity model helps prioritize what to build next.
-
Every important request has a trace ID
-
Every LLM call is traced
-
Tool calls are traced
-
Retrieval is traced
-
Agent iterations are visible
-
External API calls are correlated
-
Prompts are versioned
-
Prompt version is stored in traces
-
Prompt changes can be compared
-
Input tokens are tracked
-
Output tokens are tracked
-
Cached/reasoning tokens are tracked when available
-
Cost is calculated
-
Cost can be grouped by feature, model, and tenant
-
P50/P95/P99 latency is monitored
-
Time-to-first-token is tracked
-
Slow spans are identifiable
-
Retrieval and tool latency are visible
-
Agent trajectories are traceable
-
Tool calls are visible
-
Retries are tracked
-
Loops are detectable
-
Maximum iterations are enforced
-
Maximum execution time is enforced
-
Cost limits exist
-
Retrieval queries are traced
-
Retrieved documents are visible
-
Similarity scores are available
-
Reranking is observable
-
Context size is tracked
-
Empty retrieval results are monitored
-
Secrets are removed
-
PII is protected
-
Trace access is controlled
-
Retention policies exist
-
Sensitive payloads are sampled or redacted
-
Traces can be evaluated
-
User feedback is connected to traces
-
Failed traces can become evaluation datasets
-
Prompt/model changes can be evaluated
-
Quality regressions can trigger alerts
The ultimate goal is not simply collecting traces.
It is continuous improvement.
flowchart TD
PROD[Production AI System]
PROD --> TRACE[Tracing]
TRACE --> EVAL[Evaluation]
EVAL --> ANALYZE[Failure Analysis]
ANALYZE --> DATASET[Evaluation Dataset]
DATASET --> EXP[Experiment]
EXP --> PROMPT[Prompt Change]
EXP --> MODEL[Model Change]
EXP --> RETRIEVAL[Retrieval Change]
EXP --> TOOL[Tool Change]
PROMPT --> DEPLOY[Deploy]
MODEL --> DEPLOY
RETRIEVAL --> DEPLOY
TOOL --> DEPLOY
DEPLOY --> PROD
The engineering loop becomes:
Observe
↓
Evaluate
↓
Diagnose
↓
Improve
↓
Deploy
↓
Observe
This is where observability becomes an AI engineering capability rather than just another monitoring dashboard.
LLM observability is the foundation for operating AI systems in production.
Traditional observability tells you whether your service is running.
LLM observability tells you:
What happened?
Why did it happen?
Which model was used?
Which prompt version was used?
What context did it receive?
Which tools were called?
How many tokens were consumed?
How much did it cost?
How long did it take?
Where did it fail?
Was the result actually good?
For simple LLM applications:
OpenTelemetry
+
LLM tracing
+
Token tracking
+
Cost tracking
+
Error monitoring
For RAG:
Retrieval tracing
+
Context inspection
+
RAG evaluation
For AI agents:
Agent trajectory tracing
+
Tool-call observability
+
Loop detection
+
Evaluation
+
Cost controls
The goal is not to collect more telemetry.
The goal is to make AI behavior observable, measurable, debuggable, secure, and continuously improvable.
If you cannot reconstruct what an AI system did, you cannot reliably debug, optimize, or operate it in production.
LLM Observability · AI Observability · LLM Tracing · AI Agents · RAG · OpenTelemetry · LLM Evaluation · LLM Cost Tracking · Production AI