{"slug": "when-your-ai-agent-handles-money-it-worked-isn-t-good-enough", "title": "When Your AI Agent Handles Money, \"It Worked\" Isn't Good Enough", "summary": "A developer built Weft, an autonomous milestone verifier that uses three AI agents to release or refund staked ETH based on project deliverables. The system, which relies on Python, 0G Chain smart contracts, Zama FHE, and SigNoz for observability, revealed critical bottlenecks like a 10x slowdown in FHE encryption calls only after instrumenting with OpenTelemetry spans. The developer found that tracing peer broadcast timeouts and consensus formation was essential for debugging, turning observability into the core product.", "body_md": "Three verifier agents. One milestone. Real ETH at stake.\n\nThe first time I ran the consensus loop without tracing, a silent timeout in the peer broadcast caused two nodes to vote while the third sat idle. The quorum passed anyway (2-of-3), but I had no idea the third node was broken until I checked the logs manually — 40 minutes later.\n\nThat's when I stopped treating observability as a nice-to-have and started treating it as the product.\n\nWhat I Built\n\nWeft is an autonomous milestone verifier: builders stake ETH against project deliverables, and when a deadline passes, three independent AI agents gather evidence — contract deployment, on-chain usage, GitHub commits — corroborate over encrypted channels, and execute a verdict on-chain. Capital releases or refunds automatically.\n\nThe stack:\n\nPython verifier daemon\n\n0G Chain smart contracts\n\nZama FHE for sealed ballot privacy\n\nAXL for peer-to-peer messaging\n\nSigNoz for the entire observability layer\n\nThe Instrumentation That Actually Mattered\n\nI instrumented the daemon with OpenTelemetry's Python SDK — standard stuff at first: one root span per verification cycle, child spans for each evidence-gathering step. But the useful instrumentation came from the places I didn't initially think to trace.\n\nEach verifier broadcasts its signed verdict envelope to the other two nodes via HTTP POST. I wrapped the broadcast call in a span with attributes for peer.address, envelope.evidence_root, and response.status.\n\npython\n\nwith tracer.start_as_current_span(\n\n\"axl.broadcast_verdict\",\n\nattributes={\n\n\"peer.address\": peer_url,\n\n\"milestone.hash\": milestone_hash,\n\n\"envelope.evidence_root\": evidence_root,\n\n},\n\n) as span:\n\nresp = requests.post(f\"{peer_url}/send\", json=envelope, timeout=10)\n\nspan.set_attribute(\"http.status_code\", resp.status_code)\n\nWhen the third node was timing out, the span showed a 30s duration with a deadline_exceeded status — something I'd never have found in stdout logs, because the daemon's retry logic silently moved on.\n\nWhen AXL_WAIT_FOR_PEERS=1, the daemon polls its inbox for matching peer envelopes before voting. I added a span that tracks how long consensus took to form and how many unique signers contributed. In SigNoz, this shows up as a variable-width bar in the waterfall — you immediately see whether consensus was fast (all nodes healthy) or slow (one node lagging).\n\nZama's FHEVM operations are computationally expensive. A span around submit_encrypted_weighted_verdict revealed it was taking 4–6 seconds per call — 10x longer than the regular submitVerdict. That's fine once you know it. Without the span, the total cycle time just looked \"slow,\" with no obvious bottleneck.\n\nWhat SigNoz Gave Me That Logs Didn't\n\nThe dashboard I actually lived in had eight panels:\n\nVerification cycle duration (p50/p95) — one number that told me if something was degrading\n\nEvidence source availability — three gauges for GitHub API, RPC node, and 0G indexer\n\nPeer consensus formation time — how long from first broadcast to quorum\n\nTransaction success rate — did the on-chain verdict actually land\n\nPer-node vote status — which verifiers voted, which are stuck\n\nFHE encryption latency — the Zama call specifically\n\nActive milestone count — workload across pending deadlines\n\nError rate by span — which operation is failing most\n\nThe trace waterfall is where debugging actually happened. A single verification cycle produces 15–25 spans:\n\ndeadline_scheduler.poll\n\n→ indexer_client.get_milestone\n\n→ metadata_reader.read\n\n→ github_client.collect\n\n→ eth_rpc.get_code\n\n→ mvp_verifier.count_callers\n\n→ kimi_client.generate_narrative\n\n→ axl_client.broadcast\n\n→ peer_inbox.wait_for_consensus\n\n→ keeperhub_client.execute_verdict\n\nWhen something breaks, you click the trace and see exactly where.\n\nThe thing that surprised me: I used trace-to-logs correlation more than I expected. The daemon emits structured JSON logs with trace IDs, so clicking a suspicious span in SigNoz jumps straight to the exact log lines from that operation. This was invaluable when debugging a case where evidence-gathering passed but the attestation JSON was malformed — the span said \"success,\" but the log showed a missing field that only mattered downstream.\n\nWhat I'd Tell My Past Self\n\nInstrument the boundaries, not the internals. I wasted time tracing individual lines of business logic. The useful spans were at system boundaries: HTTP calls to peers, RPC calls to the chain, API calls to GitHub/Kimi/0G. Internal function calls are better served by logs correlated to the parent span.\n\nSet span attributes aggressively. milestone.hash, verifier.address, evidence.type, consensus.signer_count — these turn a generic waterfall into a queryable dataset. When I needed to answer \"which milestone took longest to verify last week,\" it was a SigNoz query, not a grep.\n\nThe alert that saved me: a simple condition — verification cycle > 120s — caught a regression where the 0G indexer RPC was returning stale data. The daemon kept retrying reads for metadata that hadn't propagated yet. I added exponential backoff with a ceiling, but without the alert I'd have burned through rate limits for hours before noticing.\n\nDon't trace in production without sampling in mind. Three verifier nodes, each polling every 60 seconds, each producing 15+ spans per cycle — that's roughly 2,700 spans/hour, minimum. Fine for a hackathon. For production, you'd want head-based sampling on the poll cycles and 100% sampling on the verdict-execution path.\n\nThe Takeaway\n\nThe point of Weft is that autonomous agents handling money must be observable — not because regulators require it (though they will), but because you need to debug it at 2am when a verdict doesn't land and there's real capital on the line.\n\nSigNoz turned \"the third node seems broken\" from a 40-minute log-spelunking session into a 10-second glance at a trace waterfall.\n\nIf you're building agents that do anything consequential — transactions, deployments, approvals — instrument the decision path end-to-end before you ship. You'll thank yourself the first time something fails silently.\n\nWeft is open source: github.com/thisyearnofear/weft. The SigNoz dashboards are provisioned via OpenTofu in agent/scripts/weft_signoz_provision.sh.", "url": "https://wpnews.pro/news/when-your-ai-agent-handles-money-it-worked-isn-t-good-enough", "canonical_source": "https://dev.to/papajams/when-your-ai-agent-handles-money-it-worked-isnt-good-enough-1jbb", "published_at": "2026-07-25 02:02:35+00:00", "updated_at": "2026-07-25 02:28:50.424152+00:00", "lang": "en", "topics": ["ai-agents", "ai-infrastructure", "developer-tools", "artificial-intelligence"], "entities": ["Weft", "0G Chain", "Zama", "SigNoz", "OpenTelemetry", "AXL"], "alternates": {"html": "https://wpnews.pro/news/when-your-ai-agent-handles-money-it-worked-isn-t-good-enough", "markdown": "https://wpnews.pro/news/when-your-ai-agent-handles-money-it-worked-isn-t-good-enough.md", "text": "https://wpnews.pro/news/when-your-ai-agent-handles-money-it-worked-isn-t-good-enough.txt", "jsonld": "https://wpnews.pro/news/when-your-ai-agent-handles-money-it-worked-isn-t-good-enough.jsonld"}}