Metadata-Only Tracing: Privacy-First Observability for AI Agents

A developer proposes metadata-only tracing as a privacy-first observability approach for AI agents, recording execution structure without storing raw payloads by default. The technique uses operation-specific types and controlled vocabularies to minimize sensitive data exposure while maintaining operational visibility.

Agent tracing is useful because it reveals execution structure: which step ran, which tool failed, where a retry occurred, how long a model call took, and how the token budget changed. The easiest implementation is to capture every prompt, argument, result, and response. It is also the easiest way to turn an observability system into a second copy of sensitive application data. Metadata-only tracing takes a different approach. It records the behavior of an agent without storing its raw payloads by default. The result is not zero-risk telemetry, but it is a much smaller and more governable data surface. A useful metadata trace should answer operational questions such as: It should not answer these questions unless a separate capture policy explicitly allows it: That boundary keeps everyday traces useful without making full-fidelity capture the default. Metadata can still be sensitive. A workflow name, fine-grained location, unique identifier, decision label, or rare error category may identify a person or reveal confidential business activity. The relevant distinction is not “payload versus harmless metadata.” It is necessary, classified metadata versus unbounded content . Every field still needs a purpose, an owner, and a retention policy. Avoid free-form metadata bags. A type such as Record<string, string | number constrains value shapes, but it does not prevent a developer from adding email , prompt , or accessToken . Operation-specific types make the intended schema visible during code review and prevent arbitrary fields from spreading through the trace system. type StepMetadata = { retrieval: { source: 'knowledge base' | 'ticket index'; requestedTopK: number; resultCount: number; contextTokens: number; }; model: { provider: 'openai' | 'anthropic' | 'google' | 'other'; model: string; inputTokens: number; cachedInputTokens: number; outputTokens: number; finishReason: 'stop' | 'length' | 'tool' | 'other'; }; tool: { tool: 'lookup order' | 'search docs' | 'create ticket'; result: 'found' | 'not found' | 'created' | 'rejected'; retryCount: number; }; policy: { policy: 'input validation' | 'tool authorization' | 'output check'; outcome: 'allow' | 'block'; reason: 'valid' | 'invalid shape' | 'not authorized' | 'unsafe output'; }; }; type StepKind = keyof StepMetadata; Controlled vocabularies are intentional. They make dashboards stable, reduce high-cardinality fields, and force new data collection to be reviewed as a schema change. Model names may remain dynamic, but they should still be length-limited and normalized. User-controlled strings should not be copied into these fields. The trace envelope should support parent-child relationships without carrying application payloads. type TraceStatus = 'ok' | 'error'; type StepCompleted<K extends StepKind = StepKind = { version: 1; event: 'step completed'; traceId: string; spanId: string; parentSpanId: string | null; timestamp: string; kind: K; name: string; status: TraceStatus; durationMs: number; errorCategory?: | 'timeout' | 'rate limit' | 'validation' | 'authorization' | 'dependency' | 'unknown'; metadata?: StepMetadata K ; }; Versioning matters because trace artifacts often outlive the code that produced them. A version field lets readers migrate or reject incompatible events rather than guessing their shape. Do not include raw error messages or stack traces in the default event. Both frequently contain payload fragments, file paths, headers, or query values. Map exceptions to a controlled category and keep richer diagnostics behind a restricted capture mode. AsyncLocalStorage can carry trace and parent-span identifiers across promise chains without passing them through every function signature. The tracer below emits completion events and requires each operation to return metadata that matches its declared kind. js import { AsyncLocalStorage } from 'node:async hooks'; import { randomUUID } from 'node:crypto'; type TraceContext = { traceId: string; spanId: string | null; }; type StepOutput<T, K extends StepKind = { value: T; metadata: StepMetadata K ; }; export interface TraceSink { write event: StepCompleted : Promise<void ; } const traceContext = new AsyncLocalStorage<TraceContext ; function categorizeError error: unknown : StepCompleted 'errorCategory' { if error instanceof Error return 'unknown'; if error.name === 'AbortError' return 'timeout'; if error.name === 'ValidationError' return 'validation'; if error.name === 'AuthorizationError' return 'authorization'; return 'dependency'; } export async function runTrace<T work: = Promise<T , : Promise<T { return traceContext.run { traceId: randomUUID , spanId: null }, work, ; } export async function traceStep<K extends StepKind, T sink: TraceSink, kind: K, name: string, work: = Promise<StepOutput<T, K , : Promise<T { const parent = traceContext.getStore ; if parent throw new Error 'traceStep must run inside runTrace' ; const spanId = randomUUID ; const startedAt = Date.now ; try { const output = await traceContext.run { traceId: parent.traceId, spanId }, work, ; await sink.write { version: 1, event: 'step completed', traceId: parent.traceId, spanId, parentSpanId: parent.spanId, timestamp: new Date .toISOString , kind, name, status: 'ok', durationMs: Date.now - startedAt, metadata: output.metadata, } ; return output.value; } catch error { await sink.write { version: 1, event: 'step completed', traceId: parent.traceId, spanId, parentSpanId: parent.spanId, timestamp: new Date .toISOString , kind, name, status: 'error', durationMs: Date.now - startedAt, errorCategory: categorizeError error , } ; throw error; } } The sink can write to a local NDJSON file during development or export approved events to an observability backend. Capture policy belongs before the sink so changing destinations cannot silently increase what is collected. The model and retrieval operations can use sensitive values in memory while returning only bounded operational metadata to the tracer. js const answer = await runTrace async = { const documents = await traceStep sink, 'retrieval', 'retrieve support docs', async = { const value = await searchDocuments userQuestion ; return { value, metadata: { source: 'knowledge base', requestedTopK: 5, resultCount: value.length, contextTokens: countDocumentTokens value , }, }; }, ; return traceStep sink, 'model', 'generate support answer', async = { const response = await callModel userQuestion, documents ; return { value: response.text, metadata: { provider: 'other', model: response.model.slice 0, 80 , inputTokens: response.usage.inputTokens, cachedInputTokens: response.usage.cachedInputTokens ?? 0, outputTokens: response.usage.outputTokens, finishReason: response.finishReason, }, }; }, ; } ; This trace can reveal an empty retrieval result, an oversized context, a length-limited response, or an unexpectedly expensive model call. It never needs the user question or document text. support agent 1,184 ms ok ├─ retrieve support docs 96 ms ok │ source=knowledge base resultCount=5 contextTokens=0 └─ generate support answer 1,072 ms ok inputTokens=640 cachedInputTokens=0 outputTokens=84 finishReason=stop The zero-token retrieval context is immediately suspicious even though the trace does not expose the documents. Metadata narrows the investigation; a developer can then enable selective local capture for that step if the issue cannot be reproduced otherwise. TypeScript types disappear at runtime, and trace data may come from JavaScript adapters or external libraries. Validate events before writing them: Schema validation libraries can enforce the structural rules. The operation-specific builders should still remain the primary policy boundary. Privacy requirements are often about what must never appear. Encode those expectations in tests. js const forbiddenKeys = 'prompt', 'response', 'args', 'resultBody', 'authorization', 'cookie', 'email', as const; function assertNoForbiddenKeys event: unknown : void { const serialized = JSON.stringify event .toLowerCase ; for const key of forbiddenKeys { if serialized.includes "${key.toLowerCase }" { throw new Error Forbidden trace key: ${key} ; } } } Add representative secrets and personal data to test fixtures, run the agent, and assert that none of those values appear in the emitted trace. This is not a substitute for a broader security review, but it catches regressions when instrumentation changes. Metadata-only tracing is excellent for timing, topology, retries, token usage, policy outcomes, and broad error localization. It cannot explain every semantic failure. When exact content is necessary, use a separate diagnostic mode with these constraints: The escalation path should be obvious, but it should require intent. Start with a versioned event envelope, operation-specific metadata types, async parent-child context, controlled error categories, and runtime validation. Store timing, status, token usage, counts, and bounded labels. Keep prompts, outputs, tool payloads, retrieved text, headers, and environment values out of the default schema. Metadata-only tracing will not answer every debugging question. It will answer a large portion of them while substantially reducing the amount of sensitive data your observability system must protect. The next article will focus on the TypeScript runtime itself: async context propagation, module boundaries, serverless execution, and the hooks a tracing tool needs to handle cleanly.