{"slug": "part-6-observability-for-ai-agents-tracing-metrics-and-drift", "title": "Part 6: Observability for AI Agents: Tracing, Metrics, and Drift", "summary": "Developer Akash Pal's open-source 'agent-from-scratch' project demonstrates a framework-free approach to observability for AI agents, emphasizing structured tracing with hashed arguments and truncated summaries to ensure safe, readable logs. The trace log doubles as CLI output, and the design anticipates future online metrics and drift detection without requiring changes to the tracing code.", "body_md": "*Part 6 of a series building a support-ticket agent with no framework. Previous: Part 5 (guardrails). Repo: github.com/akash-pal/agent-from-scratch*\n\n\"Run the eval set\" and \"is this agent healthy right now\" are different questions, and it's easy to only build infrastructure for the first one. Eval sets run offline, on cases you already thought of. Production traffic doesn't ask permission to send you a ticket type you didn't anticipate. Observability is what tells you when that's happening — and it's also, unglamorously, what makes offline evaluation possible in the first place: you can't debug a failing eval case without knowing what the agent actually did, step by step.\n\nEvery tool call in this build logs a structured record — [ src/trace.ts](https://github.com/akash-pal/agent-from-scratch/blob/main/src/trace.ts):\n\n```\nexport interface TraceStep {\n  trace_id: string;\n  step_id: number;\n  tool_name: string;\n  args_hash: string;        // hashed, never raw args\n  duration_ms: number;\n  result_summary: string;\n  model: string;\n  token_usage: { input: number; output: number };\n}\n```\n\nTwo details here that look small and aren't:\n\n** args_hash, not raw args.** This trace log is meant to be safe to keep around, ship to a monitoring system, or paste into a bug report — none of which should require thinking about what secrets might be embedded in a tool call's arguments. Hashing means you can still confirm two calls used identical arguments (for debugging idempotency, for instance) without ever persisting the actual values:\n\n```\nexport function hashArgs(args: Record<string, unknown>): string {\n  return \"sha256:\" + createHash(\"sha256\").update(JSON.stringify(args)).digest(\"hex\").slice(0, 8);\n}\n```\n\n** result_summary, truncated.** Full tool results can be large (a\n\n`kb_search`\n\nreturning full article bodies, for instance) — logging the whole thing on every step makes trace output unreadable and bloats whatever's storing it. `summarizeResult`\n\ntakes the first few fields and truncates long values:\n\n``` js\nconst MAX_FIELD_LEN = 70;\nexport function summarizeResult(result: Record<string, unknown>): string {\n  const entries = Object.entries(result).slice(0, 4);\n  return entries.map(([k, v]) => `${k}=${truncate(JSON.stringify(v))}`).join(\"  \");\n}\n```\n\nThe trace log doubles as CLI output — this build's whole point is being inspectable, so watching an agent run in real time matters. Early on, that meant a raw JSON blob per line, which is technically complete and practically unreadable. The fix was a small formatting pass, not a new logging system:\n\n``` js\nexport function logTrace(step: TraceStep): void {\n  const timing = `${step.duration_ms}ms, ${step.token_usage.input}→${step.token_usage.output} tok`;\n  console.log(`  ${DIM}[${step.step_id}]${RESET} ${CYAN}${step.tool_name}${RESET} ${DIM}(${timing})${RESET}`);\n  console.log(`      ${step.result_summary}`);\n}\n```\n\nOutput in an actual terminal:\n\n```\n  [1] order_lookup (0ms, 1077→23 tok)\n      order_id=\"ord_1005\"  status=\"processing\"  items=[...]  total_usd=45\n  [2] kb_search (1ms, 1268→20 tok)\n      articles=[...]  relevance_scores=[0.48,0.24,0.24]\n```\n\nColors auto-disable when `stdout`\n\nisn't a real TTY (`process.stdout.isTTY`\n\n), so piping this to a file or a CI log doesn't leave you with literal escape-code garbage — small thing, but the kind of small thing that makes the difference between a trace log people actually read and one they ignore.\n\nTrace-per-step is the foundation, but three distinct layers sit on top of it, each answering a different question:\n\nThis repo doesn't implement online metrics or drift detection — it's a CLI reference build with no persistent request volume to aggregate — but the trace payload is written specifically so those layers could be built on top of it without changing the tracing code itself. That's the actual design goal: the minimum payload isn't \"the metrics you need now,\" it's \"the raw material any metrics system would need later.\"\n\nGo back to Part 3's eval failure:\n\n```\n[FAIL] hard_04 (hard)\n    - trajectory: expected [order_lookup, refund_eligibility, issue_refund] as a subsequence, got [order_lookup, refund_eligibility]\n```\n\nThat failure message exists because of tracing, full stop. Without a structured, step-by-step record of what tools got called, \"the eval failed\" would be all you'd know — not *why*. The actual bug behind that failure (Part 5's phantom refund proposal) was findable specifically because the trajectory was visible, not just the final pass/fail.\n\n** Part 7: Iterating to Green: Real Bugs, and When You'd Actually Reach for a Framework →** closes the series: the full iteration log — every real bug found running this agent against the eval set, what fixed each one, and when you'd actually reach for a framework instead of this raw loop.", "url": "https://wpnews.pro/news/part-6-observability-for-ai-agents-tracing-metrics-and-drift", "canonical_source": "https://dev.to/akashpal/part-6-observability-for-ai-agents-tracing-metrics-and-drift-2pgh", "published_at": "2026-08-11 18:43:58+00:00", "updated_at": "2026-08-11 18:49:11.668242+00:00", "lang": "en", "topics": ["ai-agents", "developer-tools", "ai-infrastructure", "mlops"], "entities": ["Akash Pal", "agent-from-scratch", "github.com/akash-pal/agent-from-scratch"], "alternates": {"html": "https://wpnews.pro/news/part-6-observability-for-ai-agents-tracing-metrics-and-drift", "markdown": "https://wpnews.pro/news/part-6-observability-for-ai-agents-tracing-metrics-and-drift.md", "text": "https://wpnews.pro/news/part-6-observability-for-ai-agents-tracing-metrics-and-drift.txt", "jsonld": "https://wpnews.pro/news/part-6-observability-for-ai-agents-tracing-metrics-and-drift.jsonld"}}