# How I Built a Verifier Engine That Catches AI Agents Lying About What They Did > Source: > Published: 2026-09-14 14:13:40+00:00 Your AI agent just told you it sent the email. It didn't. Your AI agent just told you it deployed the code. It didn't. Your AI agent just told you it processed the refund. It didn't. And you have no way of knowing until the client calls. Or the deploy fails in production. Or the refund never arrives. This is the silent failure mode of AI agents. It's not hallucination in the classic sense. The agent isn't inventing facts. It's claiming an action succeeded when the action never happened. The agent says "done." Nothing checks. The failure is discovered late, which is the expensive combination. I've been building AI agents for a year. This problem killed more deployments than anything else. So I built a fix. Every observability tool today, including LangSmith, Helicone, and Arize, watches what the agent *said*. They log outputs, trace tokens, and show you what the LLM produced. They all have the same fatal assumption: **the agent's output is a reliable signal of what happened.** It isn't. An agent that says "I sent the email" may have: The trace looks identical in every case. "I sent the email." From the agent's perspective, the mission was accomplished. The unit that matters isn't the commitment. **It's the effect.** If the email didn't leave the outbox, the commitment isn't fulfilled, no matter what the agent says. Here's the architecture I landed on: ``` Agent output → Commitment extraction → External verifier → State transition ``` Instead of trusting the agent's claim, COGEXT runs an external check against the system that would have been affected by the action. **Commitment:** "I'll email Sarah the report by Friday." **Verifier query:** `from:me to:sarah@example.com after:Monday before:Friday` **Verifier runs:** Gmail API checks the sent folder. **Result:** `fulfilled` `failed` The agent doesn't get a vote. The external system does. The critical design decision: **the verifier query is generated at commitment creation time, not after the deadline passes.** This matters because: So when a commitment is created, the LLM extracts: ``` { "action": "send", "object": "deployment report", "recipient": "sarah@example.com", "verifier_query": "check sent items for email to sarah@example.com with subject containing 'deployment report'", "verifier_type": "gmail" } ``` If `verifier_query` is null, the commitment is marked `unverifiable` at creation. It never enters the fulfilled pipeline. That's an honesty feature. It tells you the commitment can't be checked, instead of pretending it can. Here's the core state machine: ``` DETECTED → PENDING_REVIEW → OPEN → DUE → OVERDUE → FULFILLED ✓ ↓ → FAILED ✗ ↓ → EXPIRED ✗ ↓ BLOCKED → OPEN → FAILED ✗ ``` Terminal states: `fulfilled`, `failed`, `expired`, `cancelled`, `superseded`, `contradicted`. **The critical rule:** the `fulfilled` transition requires external evidence. We enforce this at the database level. All state changes go through a PostgreSQL function: ``` CREATE OR REPLACE FUNCTION cogext_transition_commitment( p_commitment_id UUID, p_new_status TEXT, p_actor TEXT DEFAULT 'system', p_data JSONB DEFAULT '{}' ) RETURNS JSONB AS $$ DECLARE current_status TEXT; evidence_score FLOAT; BEGIN SELECT status INTO current_status FROM commitments WHERE id = p_commitment_id; -- Block fulfilled without evidence for external commitments IF p_new_status = 'fulfilled' THEN SELECT MAX(score) INTO evidence_score FROM evidence WHERE commitment_id = p_commitment_id; IF evidence_score IS NULL OR evidence_score < 0.7 THEN RAISE EXCEPTION 'Cannot fulfill without evidence score >= 0.7'; END IF; END IF; -- Validate transition, update, insert event, return -- ... END; $$ LANGUAGE plpgsql; ``` Tested this last week. Tried to force-fulfill a commitment via the API. Got HTTP 409. The database refused. **The agent cannot close its own loop. Not because of application logic. Because of the database.** Some actions are too dangerous to run without human approval. Processing a $12,000 refund. Deploying to production. Sending an email to a client. For these, COGEXT pauses the agent and sends a Slack message: ``` 🚨 COGEXT: High-risk action detected Agent: support-agent-001 Action: process refund Amount: $12,000 Customer: 4821 [Approve] [Cancel] ``` The agent doesn't execute until a human clicks one of the buttons. Every decision is logged in the events table with the actor, timestamp, and reason. This is the feature every team asks for after the first time an agent does something catastrophic. Agents contradict themselves constantly. At 2:00 PM: "I'll deliver Friday." At 2:47 PM: "I'll deliver Monday." Same recipient. Same object. Different deadline. Nobody catches this in real-time. Not the developer. Not the client. Until Monday arrives and the client expects delivery and the agent thinks it's already Thursday. COGEXT scans every open commitment on every new ingest. If the new commitment conflicts with an existing one, we flag it immediately: ``` { "contradiction_alert": { "old_id": "ceb5ab93-...", "new_id": "567ae7bb-...", "reason": "Same commitment sent as two distinct messages. Deadline revised.", "old_promise": "I will deliver the report to the client on Friday", "new_promise": "I will deliver the report to the client" } } ``` The old commitment is marked `superseded`. The new one is tracked. The full history is preserved. Every commitment gets a risk score at creation time. Based on: `legal`, `approval`, `irreversible`) Example output: ``` { "risk_score": 0.55, "reasons": [ "Involves legal review", "Due Friday after 4 PM", "This agent has a 67% fulfillment rate on Friday deadlines" ] } ``` It's rule-based right now. No ML. But it catches the obvious cases, the ones that cause 80% of failures. Every commitment generates a shareable proof URL. ``` https://api.cogextai.com/api/v1/receipt/UtN6A3jo.axKJXIQ_xniFUgne07fGd9c3R4cYknLrCmRqaxGR75c ``` Anyone can visit it and see: Send it to a client. Send it to an auditor. Send it to a regulator. The point isn't the receipt. The point is that the developer can prove what their agent actually did. The SDK is 3 lines: ``` python from cogext import track agent = track(your_agent, api_key="cg_live_xxx") ``` Every call to `agent.run()` auto-ingests the output. Commitments are tracked. Verifiers run on a schedule. Evidence is checked against external systems. We ran the full test suite against production last week: ``` | Test | Feature | Status | |------|---------|--------| | 1 | Failure Predictor | PASS | | 2 | Verifier Engine | PASS | | 3 | Contradiction Radar | PASS | | 4 | Kill Switch | PASS | | 5 | Audit Receipt | PASS | | 6 | Idempotency | PASS | | 7 | State Machine Gate | PASS | | 8 | Reliability | PASS | Ready to Announce? YES. All tests passed. ``` Test 7 is the one that matters most. It tries to force-fulfill a commitment without evidence. HTTP 409. Blocked at the database level. **The state machine won't let the agent lie.** Every observability tool watches what the agent said. COGEXT watches what actually happened. The difference sounds small. It isn't. A tool that watches the agent's output will tell you "the agent sent the email." A tool that watches the effect will tell you "the email was sent" or "the email was never sent." The first is a log. The second is an audit trail. If your agent makes promises in production, you need the second one. Not because it's fashionable, but because the first one has a failure mode you can't detect until the client calls. `pip install cogext` Free tier: 2,500 commitments per month. No credit card. If you're running AI agents in production and you've ever wondered "did my agent actually do that?", give it a try. Open an issue. Tell me what breaks. The agent's word should not be the final word.