Part 6 of a series building a support-ticket agent with no framework. Previous: Part 5 (guardrails). Repo: github.com/akash-pal/agent-from-scratch
"Run the eval set" and "is this agent healthy right now" are different questions, and it's easy to only build infrastructure for the first one. Eval sets run offline, on cases you already thought of. Production traffic doesn't ask permission to send you a ticket type you didn't anticipate. Observability is what tells you when that's happening — and it's also, unglamorously, what makes offline evaluation possible in the first place: you can't debug a failing eval case without knowing what the agent actually did, step by step.
Every tool call in this build logs a structured record — src/trace.ts:
export interface TraceStep {
trace_id: string;
step_id: number;
tool_name: string;
args_hash: string; // hashed, never raw args
duration_ms: number;
result_summary: string;
model: string;
token_usage: { input: number; output: number };
}
Two details here that look small and aren't:
** args_hash, not raw args.** This trace log is meant to be safe to keep around, ship to a monitoring system, or paste into a bug report — none of which should require thinking about what secrets might be embedded in a tool call's arguments. Hashing means you can still confirm two calls used identical arguments (for debugging idempotency, for instance) without ever persisting the actual values:
export function hashArgs(args: Record<string, unknown>): string {
return "sha256:" + createHash("sha256").update(JSON.stringify(args)).digest("hex").slice(0, 8);
}
** result_summary, truncated.** Full tool results can be large (a
kb_search
returning full article bodies, for instance) — logging the whole thing on every step makes trace output unreadable and bloats whatever's storing it. summarizeResult
takes the first few fields and truncates long values:
const MAX_FIELD_LEN = 70;
export function summarizeResult(result: Record<string, unknown>): string {
const entries = Object.entries(result).slice(0, 4);
return entries.map(([k, v]) => `${k}=${truncate(JSON.stringify(v))}`).join(" ");
}
The trace log doubles as CLI output — this build's whole point is being inspectable, so watching an agent run in real time matters. Early on, that meant a raw JSON blob per line, which is technically complete and practically unreadable. The fix was a small formatting pass, not a new logging system:
export function logTrace(step: TraceStep): void {
const timing = `${step.duration_ms}ms, ${step.token_usage.input}→${step.token_usage.output} tok`;
console.log(` ${DIM}[${step.step_id}]${RESET} ${CYAN}${step.tool_name}${RESET} ${DIM}(${timing})${RESET}`);
console.log(` ${step.result_summary}`);
}
Output in an actual terminal:
[1] order_lookup (0ms, 1077→23 tok)
order_id="ord_1005" status="processing" items=[...] total_usd=45
[2] kb_search (1ms, 1268→20 tok)
articles=[...] relevance_scores=[0.48,0.24,0.24]
Colors auto-disable when stdout
isn't a real TTY (process.stdout.isTTY
), so piping this to a file or a CI log doesn't leave you with literal escape-code garbage — small thing, but the kind of small thing that makes the difference between a trace log people actually read and one they ignore.
Trace-per-step is the foundation, but three distinct layers sit on top of it, each answering a different question:
This repo doesn't implement online metrics or drift detection — it's a CLI reference build with no persistent request volume to aggregate — but the trace payload is written specifically so those layers could be built on top of it without changing the tracing code itself. That's the actual design goal: the minimum payload isn't "the metrics you need now," it's "the raw material any metrics system would need later."
Go back to Part 3's eval failure:
[FAIL] hard_04 (hard)
- trajectory: expected [order_lookup, refund_eligibility, issue_refund] as a subsequence, got [order_lookup, refund_eligibility]
That failure message exists because of tracing, full stop. Without a structured, step-by-step record of what tools got called, "the eval failed" would be all you'd know — not why. The actual bug behind that failure (Part 5's phantom refund proposal) was findable specifically because the trajectory was visible, not just the final pass/fail.
** Part 7: Iterating to Green: Real Bugs, and When You'd Actually Reach for a Framework →** closes the series: the full iteration log — every real bug found running this agent against the eval set, what fixed each one, and when you'd actually reach for a framework instead of this raw loop.