{"slug": "i-built-tracegate-because-my-ai-agent-demo-passed-but-the-traces-told-a-story", "title": "I built TraceGate because my AI agent demo passed, but the traces told a different story", "summary": "A developer built TraceGate, a release gate for AI agents that uses OpenTelemetry and SigNoz to enforce observability contracts before deployment. The tool runs agent scenarios, sends telemetry to SigNoz, and blocks releases if the run lacks sufficient evidence, catching issues like excessive tool retries or missing cost metadata that demos might miss.", "body_md": "Most AI agent demos stop at the final answer.\n\nMine did too at first. I had a support agent that could answer a refund question, call a policy tool, and avoid a prompt-injection path. From the outside, it looked fine.\n\nThen I asked a less comfortable question: if this agent broke in production, would I actually know what happened?\n\nThat question became TraceGate.\n\nTraceGate is a release gate for AI agents built with OpenTelemetry and SigNoz. It runs agent scenarios, sends telemetry to SigNoz, then checks whether the run produced enough evidence to safely ship.\n\nAI agents can pass a demo while still being hard to debug.\n\nA tool can retry three times and still return a successful answer. An LLM call can happen without cost metadata. A prompt-injection test can pass, but leave no trace of which safety path was used. Those are not product failures in the usual sense. They are observability failures.\n\nThat is what I wanted TraceGate to catch.\n\nThe idea is simple: before an agent ships, it should satisfy an observability contract. If the contract fails, the release is blocked.\n\nTraceGate uses a YAML contract to describe what a safe release run should include.\n\nHere is part of the contract I used:\n\n```\nname: TraceGate AI Agent Release Contract\nserviceName: tracegate-demo-agent\n\nbudgets:\n  maxRunCostUsd: 0.005\n  maxP95LatencyMs: 2000\n  maxToolRetries: 1\n\nchecks:\n  - id: span-agent-run\n    type: required-span\n    spanName: agent.run\n    severity: critical\n\n  - id: span-llm-call\n    type: required-span\n    spanName: llm.call\n    severity: critical\n\n  - id: attr-llm-model\n    type: required-attribute\n    spanName: llm.call\n    attribute: gen_ai.request.model\n    severity: critical\n\n  - id: retry-budget-trace-lookup\n    type: max-tool-retries\n    toolName: trace.lookup\n    maxRetries: 1\n    severity: critical\n```\n\nThe default demo intentionally fails one check. The `trace.lookup`\n\ntool retries three times, but the contract only allows one retry.\n\nThat makes the demo useful because TraceGate is not pretending everything is healthy. It blocks the release and explains why.\n\n```\nTraceGate FAIL\nChecks: 7/8 passed\nCritical failures: 1\nFAIL retry-budget-trace-lookup - Worst retry count for 'trace.lookup' was 3; limit 1.\n```\n\nThe stack is:\n\n```\nVite\nReact\nTypeScript\nNode.js\nOpenTelemetry\nSigNoz\nOpenAI\nYAML contracts\n```\n\nThe flow looks like this:\n\n``` php\nScenario -> Agent runner -> OpenTelemetry -> SigNoz -> TraceGate contract evaluator -> Pass or block\n```\n\nThe Node runner executes the scenario. OpenTelemetry records spans, metrics, and logs. SigNoz receives the telemetry through OTLP. TraceGate reads the run result and evaluates it against the contract.\n\nFor the OpenTelemetry setup, I used the OTLP HTTP exporters:\n\n``` js\nconst endpoint =\n  process.env.OTEL_EXPORTER_OTLP_ENDPOINT ?? \"http://localhost:4318\";\n\nconst sdk = new NodeSDK({\n  traceExporter: new OTLPTraceExporter({\n    url: `${endpoint}/v1/traces`\n  }),\n  logRecordProcessor: new BatchLogRecordProcessor(\n    new OTLPLogExporter({\n      url: `${endpoint}/v1/logs`\n    })\n  ),\n  metricReader: new PeriodicExportingMetricReader({\n    exporter: new OTLPMetricExporter({\n      url: `${endpoint}/v1/metrics`\n    }),\n    exportIntervalMillis: 1000\n  })\n});\n```\n\nIn my local setup, SigNoz received spans for:\n\n```\nagent.run\nllm.call\ntool.ticket.lookup\ntool.policy.search\ntool.trace.lookup\n```\n\nThe part that took the most debugging was the local SigNoz setup.\n\nI used Foundry to run SigNoz locally. The first time I started everything, the UI was up, but the telemetry path was not useful. The collector looked alive, but the pipeline was effectively not sending the data I expected.\n\nThe issue was in the generated local configuration around the OpAMP and MCP service wiring. I patched the generated setup so the ingester pointed to the actual SigNoz service and the MCP server pointed to the SigNoz UI service. After that, the OTLP export worked and I could verify spans in ClickHouse.\n\nThat was the most “real” part of the project for me. The code for the gate was straightforward compared to proving that telemetry actually reached the backend.\n\nTraceGate has a landing page and a workbench.\n\nThe landing page explains the idea: ship agents only when the traces agree.\n\nThe workbench shows:\n\n```\nlatest release verdict\nchecks passed\ncritical failures\ncost\nlatency\ngate matrix\nfailed evidence\nSigNoz investigation query\n```\n\nThe gate matrix is the part I care about most. It turns vague release confidence into specific checks.\n\nFor example:\n\n```\nRoot agent run span: pass\nLLM calls traced: pass\nModel attribute present: pass\nCost attribute present: pass\nCost budget: pass\nTrace lookup retry budget: fail\nRefund support scenario: pass\nPrompt injection scenario: pass\n```\n\nThat gives a reviewer a much better starting point than “the demo worked on my machine.”\n\nI also added a judge-operated test mode because I did not want the hosted app to feel like a static mockup.\n\nA judge can enter:\n\n```\nservice name\nscenario\ntool name\nretry budget\nobserved retries\ncost\nlatency\n```\n\nThen TraceGate generates a new verdict.\n\nFor example:\n\n```\nservice: checkout-agent\ntool: inventory.lookup\nallowed retries: 1\nobserved retries: 3\n```\n\nThat blocks the release.\n\nIf the observed retries are changed to `0`\n\n, the release becomes ready.\n\nThis makes the product easier to test without asking someone to set up my whole local environment first.\n\nThe main thing I learned is that observability is more useful when it is tied to a decision.\n\nBefore this project, I mostly treated observability as something I would open after a bug. With AI agents, that feels too late. If an agent takes a risky path, skips metadata, hides cost, or retries a flaky tool several times, I want to know during release validation.\n\nThe second thing I learned is that “the agent worked” is too broad. I now split it into two questions:\n\n```\nDid the agent produce the expected outcome?\nDid the agent produce enough evidence to debug that outcome?\n```\n\nTraceGate focuses on the second question.\n\nThe next version should have a visual contract editor. Writing YAML is fine for a hackathon, but a team should be able to create a release gate by choosing required spans, attributes, budgets, and scenarios in the UI.\n\nI would also add GitHub Actions support. The natural place for TraceGate is in CI, where it can block an agent release before merge.\n\nThe final improvement would be deeper SigNoz artifacts. Each contract should generate a matching dashboard, alert, and investigation prompt so the release gate and the debugging workflow stay connected.\n\nTraceGate started from one uncomfortable question: if my AI agent fails later, will I have the evidence to understand it?\n\nSigNoz already gives teams a strong place to inspect telemetry. TraceGate adds a release workflow on top of that telemetry.\n\nThe project is small, but the idea feels useful: agents should not ship only because the final answer looked good. They should ship when their behavior is traceable enough to trust.", "url": "https://wpnews.pro/news/i-built-tracegate-because-my-ai-agent-demo-passed-but-the-traces-told-a-story", "canonical_source": "https://dev.to/codeswithroh/i-built-tracegate-because-my-ai-agent-demo-passed-but-the-traces-told-a-different-story-36c2", "published_at": "2026-07-26 12:46:53+00:00", "updated_at": "2026-07-26 13:30:32.709864+00:00", "lang": "en", "topics": ["ai-agents", "developer-tools", "ai-infrastructure", "ai-safety", "mlops"], "entities": ["TraceGate", "OpenTelemetry", "SigNoz", "OpenAI", "ClickHouse", "Foundry"], "alternates": {"html": "https://wpnews.pro/news/i-built-tracegate-because-my-ai-agent-demo-passed-but-the-traces-told-a-story", "markdown": "https://wpnews.pro/news/i-built-tracegate-because-my-ai-agent-demo-passed-but-the-traces-told-a-story.md", "text": "https://wpnews.pro/news/i-built-tracegate-because-my-ai-agent-demo-passed-but-the-traces-told-a-story.txt", "jsonld": "https://wpnews.pro/news/i-built-tracegate-because-my-ai-agent-demo-passed-but-the-traces-told-a-story.jsonld"}}