I built TraceGate because my AI agent demo passed, but the traces told a different story

A developer built TraceGate, a release gate for AI agents that uses OpenTelemetry and SigNoz to enforce observability contracts before deployment. The tool runs agent scenarios, sends telemetry to SigNoz, and blocks releases if the run lacks sufficient evidence, catching issues like excessive tool retries or missing cost metadata that demos might miss.

Most AI agent demos stop at the final answer. Mine did too at first. I had a support agent that could answer a refund question, call a policy tool, and avoid a prompt-injection path. From the outside, it looked fine. Then I asked a less comfortable question: if this agent broke in production, would I actually know what happened? That question became TraceGate. TraceGate is a release gate for AI agents built with OpenTelemetry and SigNoz. It runs agent scenarios, sends telemetry to SigNoz, then checks whether the run produced enough evidence to safely ship. AI agents can pass a demo while still being hard to debug. A tool can retry three times and still return a successful answer. An LLM call can happen without cost metadata. A prompt-injection test can pass, but leave no trace of which safety path was used. Those are not product failures in the usual sense. They are observability failures. That is what I wanted TraceGate to catch. The idea is simple: before an agent ships, it should satisfy an observability contract. If the contract fails, the release is blocked. TraceGate uses a YAML contract to describe what a safe release run should include. Here is part of the contract I used: name: TraceGate AI Agent Release Contract serviceName: tracegate-demo-agent budgets: maxRunCostUsd: 0.005 maxP95LatencyMs: 2000 maxToolRetries: 1 checks: - id: span-agent-run type: required-span spanName: agent.run severity: critical - id: span-llm-call type: required-span spanName: llm.call severity: critical - id: attr-llm-model type: required-attribute spanName: llm.call attribute: gen ai.request.model severity: critical - id: retry-budget-trace-lookup type: max-tool-retries toolName: trace.lookup maxRetries: 1 severity: critical The default demo intentionally fails one check. The trace.lookup tool retries three times, but the contract only allows one retry. That makes the demo useful because TraceGate is not pretending everything is healthy. It blocks the release and explains why. TraceGate FAIL Checks: 7/8 passed Critical failures: 1 FAIL retry-budget-trace-lookup - Worst retry count for 'trace.lookup' was 3; limit 1. The stack is: Vite React TypeScript Node.js OpenTelemetry SigNoz OpenAI YAML contracts The flow looks like this: php Scenario - Agent runner - OpenTelemetry - SigNoz - TraceGate contract evaluator - Pass or block The Node runner executes the scenario. OpenTelemetry records spans, metrics, and logs. SigNoz receives the telemetry through OTLP. TraceGate reads the run result and evaluates it against the contract. For the OpenTelemetry setup, I used the OTLP HTTP exporters: js const endpoint = process.env.OTEL EXPORTER OTLP ENDPOINT ?? "http://localhost:4318"; const sdk = new NodeSDK { traceExporter: new OTLPTraceExporter { url: ${endpoint}/v1/traces } , logRecordProcessor: new BatchLogRecordProcessor new OTLPLogExporter { url: ${endpoint}/v1/logs } , metricReader: new PeriodicExportingMetricReader { exporter: new OTLPMetricExporter { url: ${endpoint}/v1/metrics } , exportIntervalMillis: 1000 } } ; In my local setup, SigNoz received spans for: agent.run llm.call tool.ticket.lookup tool.policy.search tool.trace.lookup The part that took the most debugging was the local SigNoz setup. I used Foundry to run SigNoz locally. The first time I started everything, the UI was up, but the telemetry path was not useful. The collector looked alive, but the pipeline was effectively not sending the data I expected. The issue was in the generated local configuration around the OpAMP and MCP service wiring. I patched the generated setup so the ingester pointed to the actual SigNoz service and the MCP server pointed to the SigNoz UI service. After that, the OTLP export worked and I could verify spans in ClickHouse. That was the most “real” part of the project for me. The code for the gate was straightforward compared to proving that telemetry actually reached the backend. TraceGate has a landing page and a workbench. The landing page explains the idea: ship agents only when the traces agree. The workbench shows: latest release verdict checks passed critical failures cost latency gate matrix failed evidence SigNoz investigation query The gate matrix is the part I care about most. It turns vague release confidence into specific checks. For example: Root agent run span: pass LLM calls traced: pass Model attribute present: pass Cost attribute present: pass Cost budget: pass Trace lookup retry budget: fail Refund support scenario: pass Prompt injection scenario: pass That gives a reviewer a much better starting point than “the demo worked on my machine.” I also added a judge-operated test mode because I did not want the hosted app to feel like a static mockup. A judge can enter: service name scenario tool name retry budget observed retries cost latency Then TraceGate generates a new verdict. For example: service: checkout-agent tool: inventory.lookup allowed retries: 1 observed retries: 3 That blocks the release. If the observed retries are changed to 0 , the release becomes ready. This makes the product easier to test without asking someone to set up my whole local environment first. The main thing I learned is that observability is more useful when it is tied to a decision. Before this project, I mostly treated observability as something I would open after a bug. With AI agents, that feels too late. If an agent takes a risky path, skips metadata, hides cost, or retries a flaky tool several times, I want to know during release validation. The second thing I learned is that “the agent worked” is too broad. I now split it into two questions: Did the agent produce the expected outcome? Did the agent produce enough evidence to debug that outcome? TraceGate focuses on the second question. The next version should have a visual contract editor. Writing YAML is fine for a hackathon, but a team should be able to create a release gate by choosing required spans, attributes, budgets, and scenarios in the UI. I would also add GitHub Actions support. The natural place for TraceGate is in CI, where it can block an agent release before merge. The final improvement would be deeper SigNoz artifacts. Each contract should generate a matching dashboard, alert, and investigation prompt so the release gate and the debugging workflow stay connected. TraceGate started from one uncomfortable question: if my AI agent fails later, will I have the evidence to understand it? SigNoz already gives teams a strong place to inspect telemetry. TraceGate adds a release workflow on top of that telemetry. The project is small, but the idea feels useful: agents should not ship only because the final answer looked good. They should ship when their behavior is traceable enough to trust.