{"slug": "we-built-a-flight-recorder-for-ai-coding-agents-here-s-what-signoz-taught-us", "title": "We Built a Flight Recorder for AI Coding Agents: Here's What SigNoz Taught Us About Watching Them Think", "summary": "SigNoz built AXRAY, an observability layer for AI coding agents, using OpenTelemetry and SigNoz as its backbone. The system revealed that 78% of agent latency came from LLM thinking time, not Docker or shell commands, enabling targeted optimizations.", "body_md": "Our AI agent was writing code, running tests, and opening pull requests, but I had absolutely no idea **why some runs took 8 seconds while others took 45**.\n\nWas the LLM thinking too long?\n\nWas it stuck retrying a failing shell command?\n\nWas Docker slow?\n\nI couldn't tell.\n\nAll I had was a loading spinner and a final result.\n\nThat gap between knowing an agent is *doing something* and actually knowing **what** it's doing is exactly why we built **AXRAY**, using **SigNoz** as our observability backbone.\n\nThis is the story of building it, the architecture behind it, and the deployment pitfalls that nearly broke everything.\n\nAutonomous coding agents don't behave like traditional microservices.\n\nA single agent turn might involve:\n\nEvery one of those actions has its own latency.\n\nEvery one can fail independently.\n\nTraditional application logs flatten all of this into an unreadable wall of text.\n\nYou can't tell whether the agent was:\n\nThat distinction matters.\n\nIf latency comes from the LLM generating thousands of reasoning tokens, the solution is prompt optimization.\n\nIf latency comes from a hanging shell command, the solution is a timeout or sandbox fix.\n\nWithout separating those two, you're simply guessing.\n\nAXRAY instruments every agent turn using **OpenTelemetry**, following the official **GenAI Semantic Conventions** instead of inventing our own telemetry schema.\n\nExamples include:\n\n`gen_ai.request.model`\n\n`gen_ai.usage.input_tokens`\n\n`gen_ai.usage.output_tokens`\n\nEvery span is tagged with a simple phase:\n\n`llm`\n\n`tool`\n\nFrom that we calculate two metrics for every agent turn.\n\nHow long the LLM spent thinking, reasoning, and generating tokens.\n\nHow long Docker actually spent executing commands.\n\nThose two numbers unlock almost everything.\n\nWe also compute an overall execution efficiency score:\n\n``` js\nconst efficiencyScore = Math.max(\n  45,\n  Math.min(98, 100 - (0.35 * brainPercent + 0.10 * envPercent))\n);\n```\n\nThe first time I watched a real execution session, I noticed something surprising.\n\nAlmost **78% of the latency** was inside the LLM.\n\nOur Docker sandbox wasn't slow.\n\nOur shell commands weren't slow.\n\nThe context window was simply too large.\n\nThat wasn't speculation.\n\nIt came directly from a SigNoz trace query against:\n\n`signoz_traces.signoz_index_v3`\n\nSigNoz ended up powering three completely different layers of AXRAY.\n\nEvery LLM request and every tool invocation exports spans through OTLP (`:4318`\n\n).\n\nOnce we mapped our attributes onto the official GenAI semantic conventions, everything started fitting naturally into the existing observability ecosystem.\n\nNo custom telemetry format required.\n\nWe wanted sub-turn latency breakdowns that standard dashboards don't expose directly.\n\nBecause SigNoz stores traces inside ClickHouse, we could run custom SQL like this:\n\n```\nSELECT\n    attributes_string['tool.name'] AS toolName,\n    avg(durationNano) AS avgDurationNano,\n    count() AS executionCount\nFROM signoz_traces.signoz_index_v3\nWHERE name = 'tool.call'\nAND attributes_string['axray.session.id'] = 'sess_42'\nGROUP BY toolName\nORDER BY avgDurationNano DESC;\n```\n\nThat query instantly showed which tools consumed the most execution time across an entire session.\n\nOne feature we really wanted was live alert visibility inside AXRAY itself.\n\nInstead of rebuilding an alerting system, we connected directly to SigNoz's MCP server.\n\nCalling:\n\n```\nsignoz_list_alerts\n```\n\nthrough `StreamableHTTPClientTransport`\n\ngave us structured JSON containing active alert rules.\n\nThis meant:\n\ncould appear directly inside AXRAY's UI without duplicating any of SigNoz's alerting logic.\n\nHere's the part I wish someone had warned me about.\n\nMost tutorials still describe the classic:\n\ninstallation flow.\n\nThat isn't the recommended deployment anymore.\n\nSigNoz has moved to **Foundry**, driven by an extremely small YAML manifest.\n\n```\napiVersion: v1alpha1\nkind: Installation\n\nmetadata:\n  name: signoz\n\nspec:\n  deployment:\n    flavor: compose\n    mode: docker\n\n  mcp:\n    spec:\n      enabled: true\n```\n\nDeployment becomes a single command:\n\n```\nfoundryctl cast -f casting.yaml\n```\n\nFoundry:\n\nIt's actually simpler than the old approach.\n\nI just lost an afternoon following outdated tutorials before discovering it.\n\nThe second bug was much sneakier.\n\nI wrote a script that automatically imported dashboards and alert rules into SigNoz's Postgres metadata database.\n\nEverything worked perfectly.\n\nOn **my machine.**\n\nWhy?\n\nBecause I had accidentally hardcoded my own:\n\n`org_id`\n\n`user_id`\n\nEvery fresh SigNoz installation generates completely different UUIDs.\n\nThat meant my setup script silently failed on a clean installation.\n\nThe fix was querying them dynamically.\n\n``` js\nfunction getOrgAndUser(container) {\n  const orgId = runSql(\n    container,\n    \"SELECT id FROM organizations LIMIT 1;\"\n  ).stdout.trim();\n\n  const userId = runSql(\n    container,\n    \"SELECT id FROM users LIMIT 1;\"\n  ).stdout.trim();\n\n  return { orgId, userId };\n}\n```\n\nTiny change.\n\nHuge difference.\n\nIt's exactly the kind of bug that only appears when someone else runs your project.\n\nAXRAY started with one simple idea:\n\n\"Let's add some logging to our AI agent.\"\n\nIt ended up becoming something much bigger.\n\nA strong argument that AI agents should be treated exactly like any production backend:\n\nUsing SigNoz's OpenTelemetry-native architecture meant we didn't have to invent our own telemetry format, and that decision paid off far more than we expected.\n\nIf you're building AI agents that combine LLM reasoning with real system execution, the very first metric I'd instrument is:\n\nTime-in-Brain vs Time-in-Environment\n\nIt's inexpensive to add.\n\nAnd it immediately answers the only question that matters when an agent feels slow:\n\n**Was it thinking… or was it stuck?**\n\n[WeMakeDevs](https://wemakedevs.org/) × [SigNoz](https://signoz.io/) — [Agents of SigNoz Hackathon](https://wemakedevs.org/events/agents-of-signoz)\n\nIf you found this interesting, I'd love your feedback!\n\n*Thanks for reading! *", "url": "https://wpnews.pro/news/we-built-a-flight-recorder-for-ai-coding-agents-here-s-what-signoz-taught-us", "canonical_source": "https://dev.to/hussain_jamal_/we-built-a-flight-recorder-for-ai-coding-agents-heres-what-signoz-taught-us-about-watching-them-28mc", "published_at": "2026-07-26 07:15:43+00:00", "updated_at": "2026-07-26 07:29:26.944408+00:00", "lang": "en", "topics": ["ai-agents", "developer-tools", "ai-infrastructure"], "entities": ["SigNoz", "AXRAY", "OpenTelemetry", "ClickHouse", "Docker", "LLM"], "alternates": {"html": "https://wpnews.pro/news/we-built-a-flight-recorder-for-ai-coding-agents-here-s-what-signoz-taught-us", "markdown": "https://wpnews.pro/news/we-built-a-flight-recorder-for-ai-coding-agents-here-s-what-signoz-taught-us.md", "text": "https://wpnews.pro/news/we-built-a-flight-recorder-for-ai-coding-agents-here-s-what-signoz-taught-us.txt", "jsonld": "https://wpnews.pro/news/we-built-a-flight-recorder-for-ai-coding-agents-here-s-what-signoz-taught-us.jsonld"}}