{"slug": "how-i-built-a-verifier-engine-that-catches-ai-agents-lying-about-what-they-did", "title": "How I Built a Verifier Engine That Catches AI Agents Lying About What They Did", "summary": "A developer built COGEXT, a verifier engine designed to catch AI agents that falsely claim actions succeeded, such as sending an email or deploying code. The system extracts commitments from agent output at creation time, generates an external verifier query, and requires evidence with a score of at least 0.7 before a commitment can be marked fulfilled, enforced at the PostgreSQL database level. The developer argues existing observability tools like LangSmith, Helicone, and Arize only watch what agents say rather than verifying actual effects.", "body_md": "Your AI agent just told you it sent the email. It didn't.\n\nYour AI agent just told you it deployed the code. It didn't.\n\nYour AI agent just told you it processed the refund. It didn't.\n\nAnd you have no way of knowing until the client calls. Or the deploy fails in production. Or the refund never arrives.\n\nThis is the silent failure mode of AI agents. It's not hallucination in the classic sense. The agent isn't inventing facts. It's claiming an action succeeded when the action never happened. The agent says \"done.\" Nothing checks. The failure is discovered late, which is the expensive combination.\n\nI've been building AI agents for a year. This problem killed more deployments than anything else. So I built a fix.\n\nEvery observability tool today, including LangSmith, Helicone, and Arize, watches what the agent *said*. They log outputs, trace tokens, and show you what the LLM produced.\n\nThey all have the same fatal assumption: **the agent's output is a reliable signal of what happened.**\n\nIt isn't.\n\nAn agent that says \"I sent the email\" may have:\n\nThe trace looks identical in every case. \"I sent the email.\" From the agent's perspective, the mission was accomplished.\n\nThe unit that matters isn't the commitment. **It's the effect.**\n\nIf the email didn't leave the outbox, the commitment isn't fulfilled, no matter what the agent says.\n\nHere's the architecture I landed on:\n\n```\nAgent output → Commitment extraction → External verifier → State transition\n```\n\nInstead of trusting the agent's claim, COGEXT runs an external check against the system that would have been affected by the action.\n\n**Commitment:** \"I'll email Sarah the report by Friday.\"\n\n**Verifier query:** `from:me to:sarah@example.com after:Monday before:Friday`\n\n**Verifier runs:** Gmail API checks the sent folder.\n\n**Result:**\n\n`fulfilled`\n`failed`\nThe agent doesn't get a vote. The external system does.\n\nThe critical design decision: **the verifier query is generated at commitment creation time, not after the deadline passes.**\n\nThis matters because:\n\nSo when a commitment is created, the LLM extracts:\n\n```\n{\n  \"action\": \"send\",\n  \"object\": \"deployment report\",\n  \"recipient\": \"sarah@example.com\",\n  \"verifier_query\": \"check sent items for email to sarah@example.com with subject containing 'deployment report'\",\n  \"verifier_type\": \"gmail\"\n}\n```\n\nIf `verifier_query` is null, the commitment is marked `unverifiable` at creation. It never enters the fulfilled pipeline. That's an honesty feature. It tells you the commitment can't be checked, instead of pretending it can.\n\nHere's the core state machine:\n\n```\nDETECTED → PENDING_REVIEW → OPEN → DUE → OVERDUE → FULFILLED ✓\n                                  ↓                → FAILED ✗\n                                  ↓                → EXPIRED ✗\n                                  ↓\n                            BLOCKED → OPEN\n                                   → FAILED ✗\n```\n\nTerminal states: `fulfilled`, `failed`, `expired`, `cancelled`, `superseded`, `contradicted`.\n\n**The critical rule:** the `fulfilled` transition requires external evidence.\n\nWe enforce this at the database level. All state changes go through a PostgreSQL function:\n\n```\nCREATE OR REPLACE FUNCTION cogext_transition_commitment(\n    p_commitment_id UUID,\n    p_new_status TEXT,\n    p_actor TEXT DEFAULT 'system',\n    p_data JSONB DEFAULT '{}'\n) RETURNS JSONB AS $$\nDECLARE\n    current_status TEXT;\n    evidence_score FLOAT;\nBEGIN\n    SELECT status INTO current_status \n      FROM commitments WHERE id = p_commitment_id;\n\n    -- Block fulfilled without evidence for external commitments\n    IF p_new_status = 'fulfilled' THEN\n        SELECT MAX(score) INTO evidence_score\n          FROM evidence \n          WHERE commitment_id = p_commitment_id;\n\n        IF evidence_score IS NULL OR evidence_score < 0.7 THEN\n            RAISE EXCEPTION 'Cannot fulfill without evidence score >= 0.7';\n        END IF;\n    END IF;\n\n    -- Validate transition, update, insert event, return\n    -- ...\nEND;\n$$ LANGUAGE plpgsql;\n```\n\nTested this last week. Tried to force-fulfill a commitment via the API. Got HTTP 409. The database refused.\n\n**The agent cannot close its own loop. Not because of application logic. Because of the database.**\n\nSome actions are too dangerous to run without human approval. Processing a $12,000 refund. Deploying to production. Sending an email to a client.\n\nFor these, COGEXT pauses the agent and sends a Slack message:\n\n```\n🚨 COGEXT: High-risk action detected\n\nAgent: support-agent-001\nAction: process refund\nAmount: $12,000\nCustomer: 4821\n\n[Approve] [Cancel]\n```\n\nThe agent doesn't execute until a human clicks one of the buttons. Every decision is logged in the events table with the actor, timestamp, and reason.\n\nThis is the feature every team asks for after the first time an agent does something catastrophic.\n\nAgents contradict themselves constantly. At 2:00 PM: \"I'll deliver Friday.\" At 2:47 PM: \"I'll deliver Monday.\" Same recipient. Same object. Different deadline.\n\nNobody catches this in real-time. Not the developer. Not the client. Until Monday arrives and the client expects delivery and the agent thinks it's already Thursday.\n\nCOGEXT scans every open commitment on every new ingest. If the new commitment conflicts with an existing one, we flag it immediately:\n\n```\n{\n  \"contradiction_alert\": {\n    \"old_id\": \"ceb5ab93-...\",\n    \"new_id\": \"567ae7bb-...\",\n    \"reason\": \"Same commitment sent as two distinct messages. Deadline revised.\",\n    \"old_promise\": \"I will deliver the report to the client on Friday\",\n    \"new_promise\": \"I will deliver the report to the client\"\n  }\n}\n```\n\nThe old commitment is marked `superseded`. The new one is tracked. The full history is preserved.\n\nEvery commitment gets a risk score at creation time. Based on:\n\n`legal`, `approval`, `irreversible`)\nExample output:\n\n```\n{\n  \"risk_score\": 0.55,\n  \"reasons\": [\n    \"Involves legal review\",\n    \"Due Friday after 4 PM\",\n    \"This agent has a 67% fulfillment rate on Friday deadlines\"\n  ]\n}\n```\n\nIt's rule-based right now. No ML. But it catches the obvious cases, the ones that cause 80% of failures.\n\nEvery commitment generates a shareable proof URL.\n\n```\nhttps://api.cogextai.com/api/v1/receipt/UtN6A3jo.axKJXIQ_xniFUgne07fGd9c3R4cYknLrCmRqaxGR75c\n```\n\nAnyone can visit it and see:\n\nSend it to a client. Send it to an auditor. Send it to a regulator.\n\nThe point isn't the receipt. The point is that the developer can prove what their agent actually did.\n\nThe SDK is 3 lines:\n\n``` python\nfrom cogext import track\nagent = track(your_agent, api_key=\"cg_live_xxx\")\n```\n\nEvery call to `agent.run()` auto-ingests the output. Commitments are tracked. Verifiers run on a schedule. Evidence is checked against external systems.\n\nWe ran the full test suite against production last week:\n\n```\n| Test | Feature | Status |\n|------|---------|--------|\n| 1 | Failure Predictor | PASS |\n| 2 | Verifier Engine | PASS |\n| 3 | Contradiction Radar | PASS |\n| 4 | Kill Switch | PASS |\n| 5 | Audit Receipt | PASS |\n| 6 | Idempotency | PASS |\n| 7 | State Machine Gate | PASS |\n| 8 | Reliability | PASS |\n\nReady to Announce? YES. All tests passed.\n```\n\nTest 7 is the one that matters most. It tries to force-fulfill a commitment without evidence. HTTP 409. Blocked at the database level.\n\n**The state machine won't let the agent lie.**\n\nEvery observability tool watches what the agent said. COGEXT watches what actually happened.\n\nThe difference sounds small. It isn't.\n\nA tool that watches the agent's output will tell you \"the agent sent the email.\" A tool that watches the effect will tell you \"the email was sent\" or \"the email was never sent.\"\n\nThe first is a log. The second is an audit trail.\n\nIf your agent makes promises in production, you need the second one. Not because it's fashionable, but because the first one has a failure mode you can't detect until the client calls.\n\n`pip install cogext`\nFree tier: 2,500 commitments per month. No credit card.\n\nIf you're running AI agents in production and you've ever wondered \"did my agent actually do that?\", give it a try. Open an issue. Tell me what breaks.\n\nThe agent's word should not be the final word.", "url": "https://wpnews.pro/news/how-i-built-a-verifier-engine-that-catches-ai-agents-lying-about-what-they-did", "canonical_source": "https://dev.to/yaminbinyoosuf/how-i-built-a-verifier-engine-that-catches-ai-agents-lying-about-what-they-did-1ppj", "published_at": "2026-09-14 14:13:40+00:00", "updated_at": "2026-09-14 14:47:56.967694+00:00", "lang": "en", "topics": ["ai-agents", "ai-safety", "ai-tools", "developer-tools", "ai-infrastructure"], "entities": ["COGEXT", "LangSmith", "Helicone", "Arize", "PostgreSQL", "Gmail API"], "alternates": {"html": "https://wpnews.pro/news/how-i-built-a-verifier-engine-that-catches-ai-agents-lying-about-what-they-did", "markdown": "https://wpnews.pro/news/how-i-built-a-verifier-engine-that-catches-ai-agents-lying-about-what-they-did.md", "text": "https://wpnews.pro/news/how-i-built-a-verifier-engine-that-catches-ai-agents-lying-about-what-they-did.txt", "jsonld": "https://wpnews.pro/news/how-i-built-a-verifier-engine-that-catches-ai-agents-lying-about-what-they-did.jsonld"}}