{"slug": "from-local-traces-to-production-observability-for-google-ai-agents", "title": "From Local Traces to Production Observability for Google AI Agents", "summary": "A developer detailed a strategy for bringing observability to Google AI agents, moving from local trace trees to production monitoring via OpenTelemetry. The approach emphasizes capturing decision paths with spans and events, using application-owned attributes and reason codes to explain agent behavior without exposing sensitive chain-of-thought data.", "body_md": "Most difficult agent incidents begin with one question:\n\nWhy did the agent do that?\n\nWhy did it call this tool? Why did it retry? Why did it skip the notification? Why did it trust stale data? Why was an action blocked even though every API call succeeded?\n\nTraditional logs often answer a narrower question: what executed?\n\n```\nagent started\nmodel called\ntool called\ntool completed\nresponse sent\n```\n\nThat timeline is useful, but it loses causation. Agent systems are decision workflows. A run may include routing, model calls, tools, validation, memory, approval checks, retries, suppressions, and user feedback.\n\nProduction observability must reconstruct that decision path without turning your telemetry system into a second database of sensitive prompts.\n\nDuring local development, I want to see the run as a tree before I want to search a production dashboard.\n\n```\nproactive-hotel-agent                         1,842 ms\n├─ load-user-policy                             18 ms\n├─ detect-intent                               312 ms\n├─ search-hotels                               486 ms\n├─ compare-price                               201 ms\n├─ notification-policy                          11 ms\n│  └─ blocked: quiet-hours\n└─ final-response                              604 ms\n```\n\nThe tree immediately exposes parent-child relationships, missing steps, unexpected retries, and the point where the run changed direction.\n\nThis is the local-to-production path I aim for:\n\n```\nADK / Genkit / Gemini application\n             │\n             ├── model and tool spans\n             ├── policy decision events\n             ├── metrics and safe logs\n             ▼\n      OpenTelemetry pipeline\n             │\n       ┌─────┴───────────┐\n       ▼                 ▼\nLocal trace view   Cloud Trace / Logging / Monitoring\n                         │\n                         ▼\n              alerts, dashboards, and analytics\n```\n\nThe tools can differ between development and production. The event shape should not.\n\nCreate a span for an operation with measurable duration: an agent run, model request, tool execution, memory lookup, or policy evaluation.\n\nAttach an event when something meaningful happens inside that operation: a retry is scheduled, an action is blocked, confirmation is requested, or a fallback is selected.\n\n``` js\nimport { SpanStatusCode, trace } from \"@opentelemetry/api\";\n\nconst tracer = trace.getTracer(\"travel-agent\");\n\nasync function tracedToolCall<T>(options: {\n  runId: string;\n  toolName: string;\n  risk: \"read\" | \"write\" | \"irreversible\";\n  execute: () => Promise<T>;\n}): Promise<T> {\n  return tracer.startActiveSpan(`agent.tool.${options.toolName}`, async (span) => {\n    span.setAttributes({\n      \"app.agent.run_id\": options.runId,\n      \"app.agent.tool.name\": options.toolName,\n      \"app.agent.tool.risk\": options.risk,\n    });\n\n    try {\n      const result = await options.execute();\n      span.setStatus({ code: SpanStatusCode.OK });\n      return result;\n    } catch (error) {\n      span.recordException(error as Error);\n      span.setStatus({\n        code: SpanStatusCode.ERROR,\n        message: error instanceof Error ? error.message : \"Tool failed\",\n      });\n      throw error;\n    } finally {\n      span.end();\n    }\n  });\n}\n```\n\nThe `app.agent.*`\n\nattributes are deliberately application-owned. Adopt standard semantic attributes where they fit, but do not wait for every agent-specific convention to stabilize before creating a consistent internal taxonomy.\n\nObservability does not require private chain-of-thought. It requires an explanation produced by your application at consequential boundaries.\n\n```\nspan.addEvent(\"action.blocked\", {\n  \"app.agent.reason_code\": \"QUIET_HOURS\",\n  \"app.agent.policy_version\": \"notifications-v4\",\n  \"app.agent.next_state\": \"suppressed\",\n});\n```\n\nA small vocabulary of reason codes is easier to aggregate than arbitrary text:\n\n```\ntype ReasonCode =\n  | \"USER_REQUEST_MATCHED\"\n  | \"MISSING_REQUIRED_DETAIL\"\n  | \"CONFIRMATION_REQUIRED\"\n  | \"QUIET_HOURS\"\n  | \"DUPLICATE_ACTION\"\n  | \"TOOL_TIMEOUT\"\n  | \"LOW_CONFIDENCE\"\n  | \"POLICY_DENIED\";\n```\n\nYou can still include a redacted human-readable summary for debugging. The code is what makes dashboards and alerts reliable.\n\nRaw prompts and tool results may contain personal data, retrieved memory, internal identifiers, or confidential business context. \"We will be careful\" is not a control.\n\nPrefer metadata that answers operational questions:\n\n```\n{\n  \"promptTemplate\": \"hotel-price-drop-v3\",\n  \"model\": \"gemini-family\",\n  \"tool\": \"search_hotels\",\n  \"inputClassification\": \"travel-preferences\",\n  \"piiSentToModel\": false,\n  \"reasonCode\": \"USER_REQUEST_MATCHED\",\n  \"status\": \"success\"\n}\n```\n\nGenkit automatically instruments AI features and makes traces available locally in its Developer UI. Its Google Cloud telemetry configuration can also collect logs, traces, and metrics. Because input and output capture may be enabled, review the configuration rather than assuming payloads are excluded.\n\nFor privacy-sensitive systems, disable input/output logging and add only approved metadata:\n\n``` js\nimport { enableFirebaseTelemetry } from \"@genkit-ai/firebase\";\n\nenableFirebaseTelemetry({\n  disableLoggingInputAndOutput: true,\n});\n```\n\nAlso establish retention, sampling, and access controls. Redaction performed after export may already be too late.\n\nCPU, memory, HTTP errors, and container latency still matter. They do not tell you whether the agent is useful or safe.\n\nAdd agent-level metrics:\n\nBe careful with averages. A mean of 2.1 tool calls can hide a small population of 40-call loops. Use distributions and set budgets.\n\n```\ntype RunBudget = {\n  maxModelCalls: number;\n  maxToolCalls: number;\n  maxDurationMs: number;\n};\n```\n\nWhen a budget is reached, record a terminal state such as `budget_exhausted`\n\n; do not let the trace simply disappear after a timeout.\n\nAn agent trace can be technically successful and still produce no value.\n\nA proactive travel agent might complete every model and tool call, send a notification, and receive an immediate dismissal. That is not necessarily an infrastructure failure. It may indicate poor timing, weak relevance, or insufficient personalization.\n\nConnect operational traces to privacy-safe outcome events:\n\n```\nrun completed\n  → recommendation delivered\n    → opened\n      → accepted / dismissed / ignored\n```\n\nThis makes better questions possible:\n\nObservability should help improve the product, not merely explain incidents.\n\nProduction observability and testing should form a loop.\n\nThis local evidence loop is also the motivation behind [AgentInspect](https://github.com/rajudandigam/agent-inspect), an open-source project I created for inspecting TypeScript agent trajectories. A local debugger does not replace Cloud Trace, Genkit Monitoring, or a production observability platform. It shortens the path from \"something looks wrong\" to a reproducible engineering artifact.\n\nDo not begin by designing fifty dashboards.\n\nStart with one root `agent.run`\n\nspan, child spans for models and tools, policy-decision events, terminal outcomes, and strict payload controls. Confirm that one run can be followed end to end. Then add metrics and alerts for the failure modes that matter.\n\nA useful production trace should answer:\n\nAI-agent observability is not more logging. It is preserved causation.\n\nStart locally, keep the trace shape stable, minimize sensitive payloads, and promote important failures into tests. That is how agent debugging becomes an engineering discipline instead of an exercise in guessing.", "url": "https://wpnews.pro/news/from-local-traces-to-production-observability-for-google-ai-agents", "canonical_source": "https://dev.to/raju_dandigam/from-local-traces-to-production-observability-for-google-ai-agents-1l50", "published_at": "2026-09-02 13:54:54+00:00", "updated_at": "2026-09-02 14:25:51.944623+00:00", "lang": "en", "topics": ["ai-agents", "developer-tools", "mlops", "ai-infrastructure"], "entities": ["Google", "OpenTelemetry", "ADK", "Genkit", "Gemini", "Cloud Trace"], "alternates": {"html": "https://wpnews.pro/news/from-local-traces-to-production-observability-for-google-ai-agents", "markdown": "https://wpnews.pro/news/from-local-traces-to-production-observability-for-google-ai-agents.md", "text": "https://wpnews.pro/news/from-local-traces-to-production-observability-for-google-ai-agents.txt", "jsonld": "https://wpnews.pro/news/from-local-traces-to-production-observability-for-google-ai-agents.jsonld"}}