{"slug": "from-flat-logs-to-execution-trees-debugging-modern-ai-agents", "title": "From Flat Logs to Execution Trees: Debugging Modern AI Agents", "summary": "A developer detailed a method for reconstructing execution trees from flat event logs in AI agent debugging, addressing challenges like out-of-order events and incomplete spans. The approach uses stable trace and span identity, explicit assembly state, and diagnostics for anomalies such as duplicate starts or ends.", "body_md": "An agent trace is usually written as a sequence of events because append-only data is simple to produce:\n\n```\nspan_started\nspan_started\nspan_ended\nspan_started\nspan_ended\nspan_ended\n```\n\nDevelopers do not want to debug that sequence directly. They want to see the causal structure:\n\n```\nresearch_agent\n├─ search_web\n├─ query_database\n├─ call_finance_api\n│  ├─ attempt_1          timeout\n│  └─ attempt_2          ok\n└─ summarize_results\n```\n\nThe event stream is optimized for writing. The execution tree is optimized for understanding. Building a reliable tree requires more than sorting by timestamp: events may arrive out of order, siblings may run concurrently, spans may be incomplete, and retries may fail while the parent operation still succeeds.\n\nEvery event needs stable trace and span identity. Start events establish parentage; end events establish outcome and duration.\n\n```\ntype SpanKind = 'run' | 'model' | 'tool' | 'retrieval' | 'decision' | 'fallback';\n\ntype TraceEvent =\n  | {\n      event: 'span_started';\n      traceId: string;\n      spanId: string;\n      parentSpanId: string | null;\n      name: string;\n      kind: SpanKind;\n      timestampMs: number;\n    }\n  | {\n      event: 'span_ended';\n      traceId: string;\n      spanId: string;\n      timestampMs: number;\n      status: 'ok' | 'error' | 'cancelled';\n      errorCategory?: string;\n      metadata?: Record<string, string | number | boolean | null>;\n    };\n```\n\nTimestamps alone cannot establish parentage. Two events that occur next to each other may be siblings, unrelated concurrent work, or operations from different traces.\n\nStart and end events may not arrive together. A buffered exporter can deliver the end first, and a crashed process may never deliver an end at all.\n\nRepresent assembly state explicitly:\n\n```\ntype AssembledSpan = {\n  traceId: string;\n  spanId: string;\n  parentSpanId: string | null;\n  name: string;\n  kind: SpanKind;\n  startedAtMs: number;\n  endedAtMs?: number;\n  status: 'open' | 'ok' | 'error' | 'cancelled';\n  errorCategory?: string;\n  metadata: Record<string, string | number | boolean | null>;\n};\n\ntype AssemblyDiagnostic = {\n  code:\n    | 'duplicate_start'\n    | 'duplicate_end'\n    | 'end_without_start'\n    | 'span_left_open';\n  spanId: string;\n};\n\nclass SpanAssembler {\n  private readonly spans = new Map<string, AssembledSpan>();\n  private readonly pendingEnds = new Map<\n    string,\n    Extract<TraceEvent, { event: 'span_ended' }>\n  >();\n  readonly diagnostics: AssemblyDiagnostic[] = [];\n\n  accept(event: TraceEvent): void {\n    if (event.event === 'span_started') {\n      if (this.spans.has(event.spanId)) {\n        this.diagnostics.push({ code: 'duplicate_start', spanId: event.spanId });\n        return;\n      }\n\n      const span: AssembledSpan = {\n        traceId: event.traceId,\n        spanId: event.spanId,\n        parentSpanId: event.parentSpanId,\n        name: event.name,\n        kind: event.kind,\n        startedAtMs: event.timestampMs,\n        status: 'open',\n        metadata: {},\n      };\n\n      this.spans.set(event.spanId, span);\n\n      const pending = this.pendingEnds.get(event.spanId);\n      if (pending) {\n        this.pendingEnds.delete(event.spanId);\n        this.applyEnd(span, pending);\n      }\n\n      return;\n    }\n\n    const span = this.spans.get(event.spanId);\n    if (!span) {\n      if (this.pendingEnds.has(event.spanId)) {\n        this.diagnostics.push({ code: 'duplicate_end', spanId: event.spanId });\n        return;\n      }\n\n      this.pendingEnds.set(event.spanId, event);\n      return;\n    }\n\n    if (span.status !== 'open') {\n      this.diagnostics.push({ code: 'duplicate_end', spanId: event.spanId });\n      return;\n    }\n\n    this.applyEnd(span, event);\n  }\n\n  private applyEnd(\n    span: AssembledSpan,\n    event: Extract<TraceEvent, { event: 'span_ended' }>,\n  ): void {\n    span.endedAtMs = Math.max(span.startedAtMs, event.timestampMs);\n    span.status = event.status;\n    span.errorCategory = event.errorCategory;\n    span.metadata = event.metadata ?? {};\n  }\n\n  finish(): { spans: AssembledSpan[]; diagnostics: AssemblyDiagnostic[] } {\n    for (const spanId of this.pendingEnds.keys()) {\n      this.diagnostics.push({ code: 'end_without_start', spanId });\n    }\n\n    for (const span of this.spans.values()) {\n      if (span.status === 'open') {\n        this.diagnostics.push({ code: 'span_left_open', spanId: span.spanId });\n      }\n    }\n\n    return {\n      spans: [...this.spans.values()],\n      diagnostics: [...this.diagnostics],\n    };\n  }\n}\n```\n\nThis assembler buffers an end event until its start arrives. At finalization, unmatched ends and open spans remain visible as diagnostics. It does not invent timestamps or mark incomplete work successful.\n\nFor an unbounded live stream, pending events need a size limit and expiration policy. Otherwise malformed or hostile input can create an unbounded map.\n\nA valid trace normally has one root, but a renderer should handle multiple roots and orphans without crashing.\n\n```\ntype SpanNode = AssembledSpan & { children: SpanNode[] };\n\ntype TraceForest = {\n  roots: SpanNode[];\n  orphans: SpanNode[];\n  duplicateIds: string[];\n};\n\nfunction buildForest(spans: AssembledSpan[]): TraceForest {\n  const nodes = new Map<string, SpanNode>();\n  const duplicateIds: string[] = [];\n\n  for (const span of spans) {\n    if (nodes.has(span.spanId)) {\n      duplicateIds.push(span.spanId);\n      continue;\n    }\n\n    nodes.set(span.spanId, { ...span, children: [] });\n  }\n\n  const roots: SpanNode[] = [];\n  const orphans: SpanNode[] = [];\n\n  for (const node of nodes.values()) {\n    if (node.parentSpanId === null) {\n      roots.push(node);\n      continue;\n    }\n\n    const parent = nodes.get(node.parentSpanId);\n    if (!parent) {\n      orphans.push(node);\n      continue;\n    }\n\n    parent.children.push(node);\n  }\n\n  const sortChildren = (node: SpanNode): void => {\n    node.children.sort((a, b) => {\n      return a.startedAtMs - b.startedAtMs || a.spanId.localeCompare(b.spanId);\n    });\n    node.children.forEach(sortChildren);\n  };\n\n  roots.sort((a, b) => a.startedAtMs - b.startedAtMs);\n  roots.forEach(sortChildren);\n\n  return { roots, orphans, duplicateIds };\n}\n```\n\nValidate parent links for cycles before recursively sorting or rendering. A malformed trace where A is the parent of B and B is the parent of A can otherwise cause infinite recursion. Cycle detection can use a depth-first search with `visiting`\n\nand `visited`\n\nsets.\n\nOrphans should appear in a separate “unattached spans” section with diagnostics. Hiding them makes instrumentation gaps look like missing work.\n\nSort siblings by start time for a stable display, but do not imply that one caused the next. Parallel children can overlap completely.\n\nA useful UI combines two views:\n\n```\ntree                              timeline\nresearch_agent                    |----------------------|\n├─ search_web                     |-----|\n├─ query_database                 |-------------|\n├─ finance_api                    |--------|\n│  └─ retry                           |----|\n└─ summarize_results                            |------|\n```\n\nThe tree explains why work happened. The timeline explains when it happened.\n\nAdding every span duration usually overstates trace time for two reasons:\n\nIf a root lasts 2 seconds and contains two parallel 1-second tools, summing all three spans reports 4 seconds. The wall-clock run still lasted 2 seconds.\n\nUse separate metrics:\n\nComputing a true critical path requires dependency semantics, not just parentage. Sibling tools may all be required, any one may be sufficient, or one may be cancelled after another succeeds. The trace schema needs to represent those decision rules before a UI can label a critical path confidently.\n\nRetries are separate attempts with separate outcomes:\n\n```\nload_pricing                  ok\n├─ attempt_1                  error: timeout\n├─ attempt_2                  error: invalid_response\n└─ fallback_to_cache          ok: age_minutes=18\n```\n\nThe parent can legitimately succeed even when children fail. Do not automatically propagate the worst child status to the parent. The parent’s status should describe whether the operation fulfilled its contract; child statuses explain how.\n\nQuality gates can still warn when a successful parent depended on stale fallback data or exceeded an attempt budget.\n\nProcesses crash, clients disconnect, serverless invocations end, and exporters drop events. An open span is evidence, not clutter.\n\nRender incomplete spans distinctly and include the reason when known:\n\n```\ngenerate_answer     open: completion event missing\nstream_response     cancelled: client disconnected\ntool_call           unknown: adapter ended before callback\n```\n\nDo not silently close every open span at the trace’s last timestamp. That invents duration and status. A UI may estimate a visible range, but it should label the estimate.\n\nExecution trees are useful regression artifacts, but exact snapshots are brittle. Compare durable properties:\n\nIgnore random IDs and normalize timestamps. For concurrent siblings, compare sets or parentage rather than one exact ordering.\n\nNewline-delimited JSON works well for local traces because each event is independently appendable and replayable. A reader can stream events through the assembler rather than loading every run into memory.\n\nIndex or partition by trace ID when files become large. Bound retention and avoid storing raw prompts, tool payloads, retrieved documents, credentials, or user data by default. Tree reconstruction needs identity and lifecycle, not complete application content.\n\nBefore trusting an execution tree, verify:\n\nValidation errors should be separate from agent errors. A malformed trace may describe a successful agent run while still being unusable for debugging.\n\nFlat events are not the enemy; they are the practical storage format. The mistake is treating their arrival order as the execution model.\n\nAssemble lifecycle events into spans, validate identity and parentage, preserve incomplete data honestly, and render both tree and timeline views. Then retries, fallbacks, parallel work, and silent failures become properties of a system you can inspect rather than clues scattered through a terminal transcript.\n\nThat is the real shift from flat logs to execution trees: not more telemetry, but trustworthy structure.", "url": "https://wpnews.pro/news/from-flat-logs-to-execution-trees-debugging-modern-ai-agents", "canonical_source": "https://dev.to/raju_dandigam/from-flat-logs-to-execution-trees-debugging-modern-ai-agents-1mop", "published_at": "2026-08-19 23:40:08+00:00", "updated_at": "2026-08-19 23:43:59.169439+00:00", "lang": "en", "topics": ["developer-tools", "ai-agents"], "entities": [], "alternates": {"html": "https://wpnews.pro/news/from-flat-logs-to-execution-trees-debugging-modern-ai-agents", "markdown": "https://wpnews.pro/news/from-flat-logs-to-execution-trees-debugging-modern-ai-agents.md", "text": "https://wpnews.pro/news/from-flat-logs-to-execution-trees-debugging-modern-ai-agents.txt", "jsonld": "https://wpnews.pro/news/from-flat-logs-to-execution-trees-debugging-modern-ai-agents.jsonld"}}