Most AI agent demos stop at the final answer.
Mine did too at first. I had a support agent that could answer a refund question, call a policy tool, and avoid a prompt-injection path. From the outside, it looked fine.
Then I asked a less comfortable question: if this agent broke in production, would I actually know what happened?
That question became TraceGate.
TraceGate is a release gate for AI agents built with OpenTelemetry and SigNoz. It runs agent scenarios, sends telemetry to SigNoz, then checks whether the run produced enough evidence to safely ship.
AI agents can pass a demo while still being hard to debug.
A tool can retry three times and still return a successful answer. An LLM call can happen without cost metadata. A prompt-injection test can pass, but leave no trace of which safety path was used. Those are not product failures in the usual sense. They are observability failures.
That is what I wanted TraceGate to catch.
The idea is simple: before an agent ships, it should satisfy an observability contract. If the contract fails, the release is blocked.
TraceGate uses a YAML contract to describe what a safe release run should include.
Here is part of the contract I used:
name: TraceGate AI Agent Release Contract
serviceName: tracegate-demo-agent
budgets:
maxRunCostUsd: 0.005
maxP95LatencyMs: 2000
maxToolRetries: 1
checks:
- id: span-agent-run
type: required-span
spanName: agent.run
severity: critical
- id: span-llm-call
type: required-span
spanName: llm.call
severity: critical
- id: attr-llm-model
type: required-attribute
spanName: llm.call
attribute: gen_ai.request.model
severity: critical
- id: retry-budget-trace-lookup
type: max-tool-retries
toolName: trace.lookup
maxRetries: 1
severity: critical
The default demo intentionally fails one check. The trace.lookup
tool retries three times, but the contract only allows one retry.
That makes the demo useful because TraceGate is not pretending everything is healthy. It blocks the release and explains why.
TraceGate FAIL
Checks: 7/8 passed
Critical failures: 1
FAIL retry-budget-trace-lookup - Worst retry count for 'trace.lookup' was 3; limit 1.
The stack is:
Vite
React
TypeScript
Node.js
OpenTelemetry
SigNoz
OpenAI
YAML contracts
The flow looks like this:
Scenario -> Agent runner -> OpenTelemetry -> SigNoz -> TraceGate contract evaluator -> Pass or block
The Node runner executes the scenario. OpenTelemetry records spans, metrics, and logs. SigNoz receives the telemetry through OTLP. TraceGate reads the run result and evaluates it against the contract.
For the OpenTelemetry setup, I used the OTLP HTTP exporters:
const endpoint =
process.env.OTEL_EXPORTER_OTLP_ENDPOINT ?? "http://localhost:4318";
const sdk = new NodeSDK({
traceExporter: new OTLPTraceExporter({
url: `${endpoint}/v1/traces`
}),
logRecordProcessor: new BatchLogRecordProcessor(
new OTLPLogExporter({
url: `${endpoint}/v1/logs`
})
),
metricReader: new PeriodicExportingMetricReader({
exporter: new OTLPMetricExporter({
url: `${endpoint}/v1/metrics`
}),
exportIntervalMillis: 1000
})
});
In my local setup, SigNoz received spans for:
agent.run
llm.call
tool.ticket.lookup
tool.policy.search
tool.trace.lookup
The part that took the most debugging was the local SigNoz setup.
I used Foundry to run SigNoz locally. The first time I started everything, the UI was up, but the telemetry path was not useful. The collector looked alive, but the pipeline was effectively not sending the data I expected.
The issue was in the generated local configuration around the OpAMP and MCP service wiring. I patched the generated setup so the ingester pointed to the actual SigNoz service and the MCP server pointed to the SigNoz UI service. After that, the OTLP export worked and I could verify spans in ClickHouse.
That was the most “real” part of the project for me. The code for the gate was straightforward compared to proving that telemetry actually reached the backend.
TraceGate has a landing page and a workbench.
The landing page explains the idea: ship agents only when the traces agree.
The workbench shows:
latest release verdict
checks passed
critical failures
cost
latency
gate matrix
failed evidence
SigNoz investigation query
The gate matrix is the part I care about most. It turns vague release confidence into specific checks.
For example:
Root agent run span: pass
LLM calls traced: pass
Model attribute present: pass
Cost attribute present: pass
Cost budget: pass
Trace lookup retry budget: fail
Refund support scenario: pass
Prompt injection scenario: pass
That gives a reviewer a much better starting point than “the demo worked on my machine.”
I also added a judge-operated test mode because I did not want the hosted app to feel like a static mockup.
A judge can enter:
service name
scenario
tool name
retry budget
observed retries
cost
latency
Then TraceGate generates a new verdict.
For example:
service: checkout-agent
tool: inventory.lookup
allowed retries: 1
observed retries: 3
That blocks the release.
If the observed retries are changed to 0
, the release becomes ready.
This makes the product easier to test without asking someone to set up my whole local environment first.
The main thing I learned is that observability is more useful when it is tied to a decision.
Before this project, I mostly treated observability as something I would open after a bug. With AI agents, that feels too late. If an agent takes a risky path, skips metadata, hides cost, or retries a flaky tool several times, I want to know during release validation.
The second thing I learned is that “the agent worked” is too broad. I now split it into two questions:
Did the agent produce the expected outcome?
Did the agent produce enough evidence to debug that outcome?
TraceGate focuses on the second question.
The next version should have a visual contract editor. Writing YAML is fine for a hackathon, but a team should be able to create a release gate by choosing required spans, attributes, budgets, and scenarios in the UI.
I would also add GitHub Actions support. The natural place for TraceGate is in CI, where it can block an agent release before merge.
The final improvement would be deeper SigNoz artifacts. Each contract should generate a matching dashboard, alert, and investigation prompt so the release gate and the debugging workflow stay connected.
TraceGate started from one uncomfortable question: if my AI agent fails later, will I have the evidence to understand it?
SigNoz already gives teams a strong place to inspect telemetry. TraceGate adds a release workflow on top of that telemetry.
The project is small, but the idea feels useful: agents should not ship only because the final answer looked good. They should ship when their behavior is traceable enough to trust.