From Local Traces to Production Observability for Google AI Agents

A developer detailed a strategy for bringing observability to Google AI agents, moving from local trace trees to production monitoring via OpenTelemetry. The approach emphasizes capturing decision paths with spans and events, using application-owned attributes and reason codes to explain agent behavior without exposing sensitive chain-of-thought data.

Most difficult agent incidents begin with one question: Why did the agent do that? Why did it call this tool? Why did it retry? Why did it skip the notification? Why did it trust stale data? Why was an action blocked even though every API call succeeded? Traditional logs often answer a narrower question: what executed? agent started model called tool called tool completed response sent That timeline is useful, but it loses causation. Agent systems are decision workflows. A run may include routing, model calls, tools, validation, memory, approval checks, retries, suppressions, and user feedback. Production observability must reconstruct that decision path without turning your telemetry system into a second database of sensitive prompts. During local development, I want to see the run as a tree before I want to search a production dashboard. proactive-hotel-agent 1,842 ms ├─ load-user-policy 18 ms ├─ detect-intent 312 ms ├─ search-hotels 486 ms ├─ compare-price 201 ms ├─ notification-policy 11 ms │ └─ blocked: quiet-hours └─ final-response 604 ms The tree immediately exposes parent-child relationships, missing steps, unexpected retries, and the point where the run changed direction. This is the local-to-production path I aim for: ADK / Genkit / Gemini application │ ├── model and tool spans ├── policy decision events ├── metrics and safe logs ▼ OpenTelemetry pipeline │ ┌─────┴───────────┐ ▼ ▼ Local trace view Cloud Trace / Logging / Monitoring │ ▼ alerts, dashboards, and analytics The tools can differ between development and production. The event shape should not. Create a span for an operation with measurable duration: an agent run, model request, tool execution, memory lookup, or policy evaluation. Attach an event when something meaningful happens inside that operation: a retry is scheduled, an action is blocked, confirmation is requested, or a fallback is selected. js import { SpanStatusCode, trace } from "@opentelemetry/api"; const tracer = trace.getTracer "travel-agent" ; async function tracedToolCall<T options: { runId: string; toolName: string; risk: "read" | "write" | "irreversible"; execute: = Promise<T ; } : Promise<T { return tracer.startActiveSpan agent.tool.${options.toolName} , async span = { span.setAttributes { "app.agent.run id": options.runId, "app.agent.tool.name": options.toolName, "app.agent.tool.risk": options.risk, } ; try { const result = await options.execute ; span.setStatus { code: SpanStatusCode.OK } ; return result; } catch error { span.recordException error as Error ; span.setStatus { code: SpanStatusCode.ERROR, message: error instanceof Error ? error.message : "Tool failed", } ; throw error; } finally { span.end ; } } ; } The app.agent. attributes are deliberately application-owned. Adopt standard semantic attributes where they fit, but do not wait for every agent-specific convention to stabilize before creating a consistent internal taxonomy. Observability does not require private chain-of-thought. It requires an explanation produced by your application at consequential boundaries. span.addEvent "action.blocked", { "app.agent.reason code": "QUIET HOURS", "app.agent.policy version": "notifications-v4", "app.agent.next state": "suppressed", } ; A small vocabulary of reason codes is easier to aggregate than arbitrary text: type ReasonCode = | "USER REQUEST MATCHED" | "MISSING REQUIRED DETAIL" | "CONFIRMATION REQUIRED" | "QUIET HOURS" | "DUPLICATE ACTION" | "TOOL TIMEOUT" | "LOW CONFIDENCE" | "POLICY DENIED"; You can still include a redacted human-readable summary for debugging. The code is what makes dashboards and alerts reliable. Raw prompts and tool results may contain personal data, retrieved memory, internal identifiers, or confidential business context. "We will be careful" is not a control. Prefer metadata that answers operational questions: { "promptTemplate": "hotel-price-drop-v3", "model": "gemini-family", "tool": "search hotels", "inputClassification": "travel-preferences", "piiSentToModel": false, "reasonCode": "USER REQUEST MATCHED", "status": "success" } Genkit automatically instruments AI features and makes traces available locally in its Developer UI. Its Google Cloud telemetry configuration can also collect logs, traces, and metrics. Because input and output capture may be enabled, review the configuration rather than assuming payloads are excluded. For privacy-sensitive systems, disable input/output logging and add only approved metadata: js import { enableFirebaseTelemetry } from "@genkit-ai/firebase"; enableFirebaseTelemetry { disableLoggingInputAndOutput: true, } ; Also establish retention, sampling, and access controls. Redaction performed after export may already be too late. CPU, memory, HTTP errors, and container latency still matter. They do not tell you whether the agent is useful or safe. Add agent-level metrics: Be careful with averages. A mean of 2.1 tool calls can hide a small population of 40-call loops. Use distributions and set budgets. type RunBudget = { maxModelCalls: number; maxToolCalls: number; maxDurationMs: number; }; When a budget is reached, record a terminal state such as budget exhausted ; do not let the trace simply disappear after a timeout. An agent trace can be technically successful and still produce no value. A proactive travel agent might complete every model and tool call, send a notification, and receive an immediate dismissal. That is not necessarily an infrastructure failure. It may indicate poor timing, weak relevance, or insufficient personalization. Connect operational traces to privacy-safe outcome events: run completed → recommendation delivered → opened → accepted / dismissed / ignored This makes better questions possible: Observability should help improve the product, not merely explain incidents. Production observability and testing should form a loop. This local evidence loop is also the motivation behind AgentInspect https://github.com/rajudandigam/agent-inspect , an open-source project I created for inspecting TypeScript agent trajectories. A local debugger does not replace Cloud Trace, Genkit Monitoring, or a production observability platform. It shortens the path from "something looks wrong" to a reproducible engineering artifact. Do not begin by designing fifty dashboards. Start with one root agent.run span, child spans for models and tools, policy-decision events, terminal outcomes, and strict payload controls. Confirm that one run can be followed end to end. Then add metrics and alerts for the failure modes that matter. A useful production trace should answer: AI-agent observability is not more logging. It is preserved causation. Start locally, keep the trace shape stable, minimize sensitive payloads, and promote important failures into tests. That is how agent debugging becomes an engineering discipline instead of an exercise in guessing.