{"slug": "metadata-only-tracing-privacy-first-observability-for-ai-agents", "title": "Metadata-Only Tracing: Privacy-First Observability for AI Agents", "summary": "A developer proposes metadata-only tracing as a privacy-first observability approach for AI agents, recording execution structure without storing raw payloads by default. The technique uses operation-specific types and controlled vocabularies to minimize sensitive data exposure while maintaining operational visibility.", "body_md": "Agent tracing is useful because it reveals execution structure: which step ran, which tool failed, where a retry occurred, how long a model call took, and how the token budget changed.\n\nThe easiest implementation is to capture every prompt, argument, result, and response. It is also the easiest way to turn an observability system into a second copy of sensitive application data.\n\n**Metadata-only tracing** takes a different approach. It records the behavior of an agent without storing its raw payloads by default. The result is not zero-risk telemetry, but it is a much smaller and more governable data surface.\n\nA useful metadata trace should answer operational questions such as:\n\nIt should not answer these questions unless a separate capture policy explicitly allows it:\n\nThat boundary keeps everyday traces useful without making full-fidelity capture the default.\n\nMetadata can still be sensitive. A workflow name, fine-grained location, unique identifier, decision label, or rare error category may identify a person or reveal confidential business activity.\n\nThe relevant distinction is not “payload versus harmless metadata.” It is **necessary, classified metadata versus unbounded content**. Every field still needs a purpose, an owner, and a retention policy.\n\nAvoid free-form metadata bags. A type such as `Record<string, string | number>`\n\nconstrains value shapes, but it does not prevent a developer from adding `email`\n\n, `prompt`\n\n, or `accessToken`\n\n.\n\nOperation-specific types make the intended schema visible during code review and prevent arbitrary fields from spreading through the trace system.\n\n```\ntype StepMetadata = {\n  retrieval: {\n    source: 'knowledge_base' | 'ticket_index';\n    requestedTopK: number;\n    resultCount: number;\n    contextTokens: number;\n  };\n  model: {\n    provider: 'openai' | 'anthropic' | 'google' | 'other';\n    model: string;\n    inputTokens: number;\n    cachedInputTokens: number;\n    outputTokens: number;\n    finishReason: 'stop' | 'length' | 'tool' | 'other';\n  };\n  tool: {\n    tool: 'lookup_order' | 'search_docs' | 'create_ticket';\n    result: 'found' | 'not_found' | 'created' | 'rejected';\n    retryCount: number;\n  };\n  policy: {\n    policy: 'input_validation' | 'tool_authorization' | 'output_check';\n    outcome: 'allow' | 'block';\n    reason: 'valid' | 'invalid_shape' | 'not_authorized' | 'unsafe_output';\n  };\n};\n\ntype StepKind = keyof StepMetadata;\n```\n\nControlled vocabularies are intentional. They make dashboards stable, reduce high-cardinality fields, and force new data collection to be reviewed as a schema change.\n\nModel names may remain dynamic, but they should still be length-limited and normalized. User-controlled strings should not be copied into these fields.\n\nThe trace envelope should support parent-child relationships without carrying application payloads.\n\n```\ntype TraceStatus = 'ok' | 'error';\n\ntype StepCompleted<K extends StepKind = StepKind> = {\n  version: 1;\n  event: 'step_completed';\n  traceId: string;\n  spanId: string;\n  parentSpanId: string | null;\n  timestamp: string;\n  kind: K;\n  name: string;\n  status: TraceStatus;\n  durationMs: number;\n  errorCategory?:\n    | 'timeout'\n    | 'rate_limit'\n    | 'validation'\n    | 'authorization'\n    | 'dependency'\n    | 'unknown';\n  metadata?: StepMetadata[K];\n};\n```\n\nVersioning matters because trace artifacts often outlive the code that produced them. A version field lets readers migrate or reject incompatible events rather than guessing their shape.\n\nDo not include raw error messages or stack traces in the default event. Both frequently contain payload fragments, file paths, headers, or query values. Map exceptions to a controlled category and keep richer diagnostics behind a restricted capture mode.\n\n`AsyncLocalStorage`\n\ncan carry trace and parent-span identifiers across promise chains without passing them through every function signature. The tracer below emits completion events and requires each operation to return metadata that matches its declared kind.\n\n``` js\nimport { AsyncLocalStorage } from 'node:async_hooks';\nimport { randomUUID } from 'node:crypto';\n\ntype TraceContext = {\n  traceId: string;\n  spanId: string | null;\n};\n\ntype StepOutput<T, K extends StepKind> = {\n  value: T;\n  metadata: StepMetadata[K];\n};\n\nexport interface TraceSink {\n  write(event: StepCompleted): Promise<void>;\n}\n\nconst traceContext = new AsyncLocalStorage<TraceContext>();\n\nfunction categorizeError(error: unknown): StepCompleted['errorCategory'] {\n  if (!(error instanceof Error)) return 'unknown';\n  if (error.name === 'AbortError') return 'timeout';\n  if (error.name === 'ValidationError') return 'validation';\n  if (error.name === 'AuthorizationError') return 'authorization';\n  return 'dependency';\n}\n\nexport async function runTrace<T>(\n  work: () => Promise<T>,\n): Promise<T> {\n  return traceContext.run(\n    { traceId: randomUUID(), spanId: null },\n    work,\n  );\n}\n\nexport async function traceStep<K extends StepKind, T>(\n  sink: TraceSink,\n  kind: K,\n  name: string,\n  work: () => Promise<StepOutput<T, K>>,\n): Promise<T> {\n  const parent = traceContext.getStore();\n  if (!parent) throw new Error('traceStep must run inside runTrace');\n\n  const spanId = randomUUID();\n  const startedAt = Date.now();\n\n  try {\n    const output = await traceContext.run(\n      { traceId: parent.traceId, spanId },\n      work,\n    );\n\n    await sink.write({\n      version: 1,\n      event: 'step_completed',\n      traceId: parent.traceId,\n      spanId,\n      parentSpanId: parent.spanId,\n      timestamp: new Date().toISOString(),\n      kind,\n      name,\n      status: 'ok',\n      durationMs: Date.now() - startedAt,\n      metadata: output.metadata,\n    });\n\n    return output.value;\n  } catch (error) {\n    await sink.write({\n      version: 1,\n      event: 'step_completed',\n      traceId: parent.traceId,\n      spanId,\n      parentSpanId: parent.spanId,\n      timestamp: new Date().toISOString(),\n      kind,\n      name,\n      status: 'error',\n      durationMs: Date.now() - startedAt,\n      errorCategory: categorizeError(error),\n    });\n\n    throw error;\n  }\n}\n```\n\nThe sink can write to a local NDJSON file during development or export approved events to an observability backend. Capture policy belongs before the sink so changing destinations cannot silently increase what is collected.\n\nThe model and retrieval operations can use sensitive values in memory while returning only bounded operational metadata to the tracer.\n\n``` js\nconst answer = await runTrace(async () => {\n  const documents = await traceStep(\n    sink,\n    'retrieval',\n    'retrieve_support_docs',\n    async () => {\n      const value = await searchDocuments(userQuestion);\n\n      return {\n        value,\n        metadata: {\n          source: 'knowledge_base',\n          requestedTopK: 5,\n          resultCount: value.length,\n          contextTokens: countDocumentTokens(value),\n        },\n      };\n    },\n  );\n\n  return traceStep(\n    sink,\n    'model',\n    'generate_support_answer',\n    async () => {\n      const response = await callModel(userQuestion, documents);\n\n      return {\n        value: response.text,\n        metadata: {\n          provider: 'other',\n          model: response.model.slice(0, 80),\n          inputTokens: response.usage.inputTokens,\n          cachedInputTokens: response.usage.cachedInputTokens ?? 0,\n          outputTokens: response.usage.outputTokens,\n          finishReason: response.finishReason,\n        },\n      };\n    },\n  );\n});\n```\n\nThis trace can reveal an empty retrieval result, an oversized context, a length-limited response, or an unexpectedly expensive model call. It never needs the user question or document text.\n\n```\nsupport_agent  1,184 ms  ok\n├─ retrieve_support_docs   96 ms  ok\n│  source=knowledge_base resultCount=5 contextTokens=0\n└─ generate_support_answer 1,072 ms ok\n   inputTokens=640 cachedInputTokens=0 outputTokens=84 finishReason=stop\n```\n\nThe zero-token retrieval context is immediately suspicious even though the trace does not expose the documents. Metadata narrows the investigation; a developer can then enable selective local capture for that step if the issue cannot be reproduced otherwise.\n\nTypeScript types disappear at runtime, and trace data may come from JavaScript adapters or external libraries. Validate events before writing them:\n\nSchema validation libraries can enforce the structural rules. The operation-specific builders should still remain the primary policy boundary.\n\nPrivacy requirements are often about what must never appear. Encode those expectations in tests.\n\n``` js\nconst forbiddenKeys = [\n  'prompt',\n  'response',\n  'args',\n  'resultBody',\n  'authorization',\n  'cookie',\n  'email',\n] as const;\n\nfunction assertNoForbiddenKeys(event: unknown): void {\n  const serialized = JSON.stringify(event).toLowerCase();\n\n  for (const key of forbiddenKeys) {\n    if (serialized.includes(`\"${key.toLowerCase()}\"`)) {\n      throw new Error(`Forbidden trace key: ${key}`);\n    }\n  }\n}\n```\n\nAdd representative secrets and personal data to test fixtures, run the agent, and assert that none of those values appear in the emitted trace. This is not a substitute for a broader security review, but it catches regressions when instrumentation changes.\n\nMetadata-only tracing is excellent for timing, topology, retries, token usage, policy outcomes, and broad error localization. It cannot explain every semantic failure.\n\nWhen exact content is necessary, use a separate diagnostic mode with these constraints:\n\nThe escalation path should be obvious, but it should require intent.\n\nStart with a versioned event envelope, operation-specific metadata types, async parent-child context, controlled error categories, and runtime validation. Store timing, status, token usage, counts, and bounded labels. Keep prompts, outputs, tool payloads, retrieved text, headers, and environment values out of the default schema.\n\nMetadata-only tracing will not answer every debugging question. It will answer a large portion of them while substantially reducing the amount of sensitive data your observability system must protect.\n\nThe next article will focus on the TypeScript runtime itself: async context propagation, module boundaries, serverless execution, and the hooks a tracing tool needs to handle cleanly.", "url": "https://wpnews.pro/news/metadata-only-tracing-privacy-first-observability-for-ai-agents", "canonical_source": "https://dev.to/raju_dandigam/metadata-only-tracing-privacy-first-observability-for-ai-agents-1a43", "published_at": "2026-07-21 19:14:31+00:00", "updated_at": "2026-07-21 19:21:16.968399+00:00", "lang": "en", "topics": ["ai-agents", "ai-safety", "ai-ethics", "developer-tools", "ai-infrastructure"], "entities": ["OpenAI", "Anthropic", "Google"], "alternates": {"html": "https://wpnews.pro/news/metadata-only-tracing-privacy-first-observability-for-ai-agents", "markdown": "https://wpnews.pro/news/metadata-only-tracing-privacy-first-observability-for-ai-agents.md", "text": "https://wpnews.pro/news/metadata-only-tracing-privacy-first-observability-for-ai-agents.txt", "jsonld": "https://wpnews.pro/news/metadata-only-tracing-privacy-first-observability-for-ai-agents.jsonld"}}