Part 6: Observability for AI Agents: Tracing, Metrics, and Drift

Developer Akash Pal's open-source 'agent-from-scratch' project demonstrates a framework-free approach to observability for AI agents, emphasizing structured tracing with hashed arguments and truncated summaries to ensure safe, readable logs. The trace log doubles as CLI output, and the design anticipates future online metrics and drift detection without requiring changes to the tracing code.

Part 6 of a series building a support-ticket agent with no framework. Previous: Part 5 guardrails . Repo: github.com/akash-pal/agent-from-scratch "Run the eval set" and "is this agent healthy right now" are different questions, and it's easy to only build infrastructure for the first one. Eval sets run offline, on cases you already thought of. Production traffic doesn't ask permission to send you a ticket type you didn't anticipate. Observability is what tells you when that's happening — and it's also, unglamorously, what makes offline evaluation possible in the first place: you can't debug a failing eval case without knowing what the agent actually did, step by step. Every tool call in this build logs a structured record — src/trace.ts https://github.com/akash-pal/agent-from-scratch/blob/main/src/trace.ts : export interface TraceStep { trace id: string; step id: number; tool name: string; args hash: string; // hashed, never raw args duration ms: number; result summary: string; model: string; token usage: { input: number; output: number }; } Two details here that look small and aren't: args hash, not raw args. This trace log is meant to be safe to keep around, ship to a monitoring system, or paste into a bug report — none of which should require thinking about what secrets might be embedded in a tool call's arguments. Hashing means you can still confirm two calls used identical arguments for debugging idempotency, for instance without ever persisting the actual values: export function hashArgs args: Record<string, unknown : string { return "sha256:" + createHash "sha256" .update JSON.stringify args .digest "hex" .slice 0, 8 ; } result summary, truncated. Full tool results can be large a kb search returning full article bodies, for instance — logging the whole thing on every step makes trace output unreadable and bloats whatever's storing it. summarizeResult takes the first few fields and truncates long values: js const MAX FIELD LEN = 70; export function summarizeResult result: Record<string, unknown : string { const entries = Object.entries result .slice 0, 4 ; return entries.map k, v = ${k}=${truncate JSON.stringify v } .join " " ; } The trace log doubles as CLI output — this build's whole point is being inspectable, so watching an agent run in real time matters. Early on, that meant a raw JSON blob per line, which is technically complete and practically unreadable. The fix was a small formatting pass, not a new logging system: js export function logTrace step: TraceStep : void { const timing = ${step.duration ms}ms, ${step.token usage.input}→${step.token usage.output} tok ; console.log ${DIM} ${step.step id} ${RESET} ${CYAN}${step.tool name}${RESET} ${DIM} ${timing} ${RESET} ; console.log ${step.result summary} ; } Output in an actual terminal: 1 order lookup 0ms, 1077→23 tok order id="ord 1005" status="processing" items= ... total usd=45 2 kb search 1ms, 1268→20 tok articles= ... relevance scores= 0.48,0.24,0.24 Colors auto-disable when stdout isn't a real TTY process.stdout.isTTY , so piping this to a file or a CI log doesn't leave you with literal escape-code garbage — small thing, but the kind of small thing that makes the difference between a trace log people actually read and one they ignore. Trace-per-step is the foundation, but three distinct layers sit on top of it, each answering a different question: This repo doesn't implement online metrics or drift detection — it's a CLI reference build with no persistent request volume to aggregate — but the trace payload is written specifically so those layers could be built on top of it without changing the tracing code itself. That's the actual design goal: the minimum payload isn't "the metrics you need now," it's "the raw material any metrics system would need later." Go back to Part 3's eval failure: FAIL hard 04 hard - trajectory: expected order lookup, refund eligibility, issue refund as a subsequence, got order lookup, refund eligibility That failure message exists because of tracing, full stop. Without a structured, step-by-step record of what tools got called, "the eval failed" would be all you'd know — not why . The actual bug behind that failure Part 5's phantom refund proposal was findable specifically because the trajectory was visible, not just the final pass/fail. Part 7: Iterating to Green: Real Bugs, and When You'd Actually Reach for a Framework → closes the series: the full iteration log — every real bug found running this agent against the eval set, what fixed each one, and when you'd actually reach for a framework instead of this raw loop.