# When Your AI Agent Handles Money, "It Worked" Isn't Good Enough > Source: > Published: 2026-07-25 02:02:35+00:00 Three verifier agents. One milestone. Real ETH at stake. The first time I ran the consensus loop without tracing, a silent timeout in the peer broadcast caused two nodes to vote while the third sat idle. The quorum passed anyway (2-of-3), but I had no idea the third node was broken until I checked the logs manually — 40 minutes later. That's when I stopped treating observability as a nice-to-have and started treating it as the product. What I Built Weft is an autonomous milestone verifier: builders stake ETH against project deliverables, and when a deadline passes, three independent AI agents gather evidence — contract deployment, on-chain usage, GitHub commits — corroborate over encrypted channels, and execute a verdict on-chain. Capital releases or refunds automatically. The stack: Python verifier daemon 0G Chain smart contracts Zama FHE for sealed ballot privacy AXL for peer-to-peer messaging SigNoz for the entire observability layer The Instrumentation That Actually Mattered I instrumented the daemon with OpenTelemetry's Python SDK — standard stuff at first: one root span per verification cycle, child spans for each evidence-gathering step. But the useful instrumentation came from the places I didn't initially think to trace. Each verifier broadcasts its signed verdict envelope to the other two nodes via HTTP POST. I wrapped the broadcast call in a span with attributes for peer.address, envelope.evidence_root, and response.status. python with tracer.start_as_current_span( "axl.broadcast_verdict", attributes={ "peer.address": peer_url, "milestone.hash": milestone_hash, "envelope.evidence_root": evidence_root, }, ) as span: resp = requests.post(f"{peer_url}/send", json=envelope, timeout=10) span.set_attribute("http.status_code", resp.status_code) When the third node was timing out, the span showed a 30s duration with a deadline_exceeded status — something I'd never have found in stdout logs, because the daemon's retry logic silently moved on. When AXL_WAIT_FOR_PEERS=1, the daemon polls its inbox for matching peer envelopes before voting. I added a span that tracks how long consensus took to form and how many unique signers contributed. In SigNoz, this shows up as a variable-width bar in the waterfall — you immediately see whether consensus was fast (all nodes healthy) or slow (one node lagging). Zama's FHEVM operations are computationally expensive. A span around submit_encrypted_weighted_verdict revealed it was taking 4–6 seconds per call — 10x longer than the regular submitVerdict. That's fine once you know it. Without the span, the total cycle time just looked "slow," with no obvious bottleneck. What SigNoz Gave Me That Logs Didn't The dashboard I actually lived in had eight panels: Verification cycle duration (p50/p95) — one number that told me if something was degrading Evidence source availability — three gauges for GitHub API, RPC node, and 0G indexer Peer consensus formation time — how long from first broadcast to quorum Transaction success rate — did the on-chain verdict actually land Per-node vote status — which verifiers voted, which are stuck FHE encryption latency — the Zama call specifically Active milestone count — workload across pending deadlines Error rate by span — which operation is failing most The trace waterfall is where debugging actually happened. A single verification cycle produces 15–25 spans: deadline_scheduler.poll → indexer_client.get_milestone → metadata_reader.read → github_client.collect → eth_rpc.get_code → mvp_verifier.count_callers → kimi_client.generate_narrative → axl_client.broadcast → peer_inbox.wait_for_consensus → keeperhub_client.execute_verdict When something breaks, you click the trace and see exactly where. The thing that surprised me: I used trace-to-logs correlation more than I expected. The daemon emits structured JSON logs with trace IDs, so clicking a suspicious span in SigNoz jumps straight to the exact log lines from that operation. This was invaluable when debugging a case where evidence-gathering passed but the attestation JSON was malformed — the span said "success," but the log showed a missing field that only mattered downstream. What I'd Tell My Past Self Instrument the boundaries, not the internals. I wasted time tracing individual lines of business logic. The useful spans were at system boundaries: HTTP calls to peers, RPC calls to the chain, API calls to GitHub/Kimi/0G. Internal function calls are better served by logs correlated to the parent span. Set span attributes aggressively. milestone.hash, verifier.address, evidence.type, consensus.signer_count — these turn a generic waterfall into a queryable dataset. When I needed to answer "which milestone took longest to verify last week," it was a SigNoz query, not a grep. The alert that saved me: a simple condition — verification cycle > 120s — caught a regression where the 0G indexer RPC was returning stale data. The daemon kept retrying reads for metadata that hadn't propagated yet. I added exponential backoff with a ceiling, but without the alert I'd have burned through rate limits for hours before noticing. Don't trace in production without sampling in mind. Three verifier nodes, each polling every 60 seconds, each producing 15+ spans per cycle — that's roughly 2,700 spans/hour, minimum. Fine for a hackathon. For production, you'd want head-based sampling on the poll cycles and 100% sampling on the verdict-execution path. The Takeaway The point of Weft is that autonomous agents handling money must be observable — not because regulators require it (though they will), but because you need to debug it at 2am when a verdict doesn't land and there's real capital on the line. SigNoz turned "the third node seems broken" from a 40-minute log-spelunking session into a 10-second glance at a trace waterfall. If you're building agents that do anything consequential — transactions, deployments, approvals — instrument the decision path end-to-end before you ship. You'll thank yourself the first time something fails silently. Weft is open source: github.com/thisyearnofear/weft. The SigNoz dashboards are provisioned via OpenTofu in agent/scripts/weft_signoz_provision.sh.