{"slug": "i-built-a-12-state-accountability-layer-for-ai-agents-here-s-what-i-learned", "title": "I built a 12-state accountability layer for AI agents here's what I learned", "summary": "A developer built COGEXT, an accountability layer that tracks commitments made by AI agents through a 12-state state machine, addressing the problem of agents forgetting promises. The system uses a two-stage extraction pipeline with classification and structured extraction via Groq's llama-3.3-70b, and a weighted field-coverage model for evidence verification.", "body_md": "By the time you finish reading this, your AI agent will have made at least one promise it can't keep.\n\nThat's not a dig at your code. It's a fundamental property of how we build AI systems today. Agents are brilliant at generating intent and terrible at tracking it. They'll promise to email a client, confirm an appointment, or deploy a fix and two conversations later, they've forgotten they said anything at all.\n\nThe more I thought about this, the more I realized: **commitments aren't a memory problem. They're a state machine problem.**\n\nSo I built COGEXT — an accountability layer that sits between your agent and the outside world. You send it agent output, it extracts every commitment, and tracks each one until it's fulfilled, failed, or overdue.\n\nMost systems treat agent output as text to be logged. We treat it as events to be tracked.\n\nEvery extracted commitment moves through 12 states:\n\n```\nDETECTED\n    ├──→ OPEN ──→ DUE ──→ OVERDUE ──→ FULFILLED ✓\n    │                        ├──→ FAILED ✗\n    │                        └──→ EXPIRED ✗\n    ├──→ PENDING_REVIEW ──→ OPEN\n    ├──→ CANCELLED ✗\n    ├──→ CONTRADICTED ✗\n    └──→ BLOCKED ──→ OPEN\n                  └──→ FAILED ✗\n```\n\nTerminal states: `fulfilled`\n\n, `failed`\n\n, `expired`\n\n, `cancelled`\n\n, `superseded`\n\n, `contradicted`\n\n.\n\nAll state mutations go through a single PostgreSQL RPC function. Not direct UPDATEs from application code:\n\n```\nCREATE OR REPLACE FUNCTION cogext_transition_commitment(\n    p_commitment_id UUID,\n    p_new_status TEXT,\n    p_actor TEXT DEFAULT 'system',\n    p_data JSONB DEFAULT '{}'\n) RETURNS JSONB AS $$\nBEGIN\n    -- lock row → validate transition → update status → insert event → return\n    -- single atomic operation: if any step fails, the whole thing rolls back\nEND;\n$$ LANGUAGE plpgsql;\n```\n\nWhy? Because the state change and the event must be atomic. You should never have `status = 'fulfilled'`\n\nwithout a corresponding fulfillment event in the audit log. That's the line between a logging system and an accountability system.\n\nThe extraction pipeline has two stages.\n\n**Stage 1: Classification**\n\nNot everything an agent says is a commitment. We classify first:\n\n`genuine_commitment`\n\n— \"I will send the report by Friday.\"`intention`\n\n— \"I might send the report tomorrow.\"`suggestion`\n\n— \"We should send the report tomorrow.\"`hypothetical`\n\n— \"If we have time, I could send it.\"`quoted_statement`\n\n— \"John said he would send it.\"Only `genuine_commitment`\n\nwith confidence ≥ 0.5 passes through.\n\n**Stage 2: Structured extraction**\n\nWe use Groq's llama-3.3-70b in JSON mode to extract fields:\n\n```\nresponse = await groq_client.chat.completions.create(\n    model=\"llama-3.3-70b-versatile\",\n    messages=[{\"role\": \"user\", \"content\": prompt + message}],\n    response_format={\"type\": \"json_object\"},\n    temperature=0.2,\n)\n```\n\nThe output is validated against a Pydantic schema. If the LLM returns garbage, we retry once. If it still fails, the commitment is dropped — not inserted.\n\n**Key decision:** We don't trust the LLM to interpret deadlines. We extract the raw expression (\"Friday EOD\") and pass it to a deterministic temporal normalizer. The LLM interprets; the code decides.\n\nEvidence verification is where most systems fail. Binary matching — \"does this email mention the report?\" — generates too many false positives.\n\nWe use a weighted field-coverage model:\n\n| Field | Weight |\n|---|---|\n`action` |\n0.40 |\n`recipient` |\n0.30 |\n`object` |\n0.20 |\n`deadline` |\n0.10 |\n\nWhen evidence arrives, we check which commitment fields it matches and sum the weights:\n\n```\nEvidence 1: confirms action (\"I sent it\")\n    → 0.40\n\nEvidence 2: confirms recipient + object (\"sent to Sarah, deployment-report.pdf attached\")\n    → 0.30 + 0.20 = 0.50\n\nTotal = 0.90 → FULFILLED\n```\n\nThe threshold is configurable (default 0.90). Multiple pieces of evidence can satisfy different fields — they aggregate.\n\nAI agents repeat themselves. An agent might say:\n\nSame commitment. Without idempotency, you'd track it three times.\n\n``` python\ndef compute_idempotency_key(agent_id, promise_text, timestamp):\n    window = timestamp.replace(minute=0, second=0, microsecond=0)\n    payload = f\"{agent_id}|{promise_text.strip().lower()}|{window.isoformat()}\"\n    return hashlib.sha256(payload.encode()).hexdigest()\n```\n\nSame agent + same promise + same hour = same hash. The DB unique constraint rejects the duplicate silently.\n\nThe hour window is deliberate: it deduplicates within a session while still allowing the same promise to be legitimately re-made days later.\n\n**1. LLM output is untrusted input.**\n\nWe validate everything against Pydantic schemas before touching the database. No exceptions.\n\n**2. Determinism over magic.**\n\nLLMs for interpretation. Deterministic code for everything else: state transitions, date normalization, deduplication, metrics. We don't call the LLM to check if a deadline is overdue. We run a database query.\n\n**3. Events are not logs.**\n\nA log is a record that can be ignored. An event is a fact that cannot be denied. We enforce this at the DB layer — a trigger rejects any UPDATE or DELETE on the events table. Historical events are immutable.\n\n**4. Scope the user_id at the auth layer, never the request body.**\n\nEarly version trusted `user_id`\n\nfrom the request payload. Classic mistake. All scoping now comes from the API key's `account_id`\n\n. The request body cannot claim to own data it doesn't own.\n\nThe API is live and the SDK is on PyPI:\n\n```\npip install cogext\npython\nfrom cogext import track\n\nagent = track(your_agent, api_key=\"cg_live_xxx\")\noutput = agent.run(\"I'll send the report by Friday\")\n# commitment is tracked automatically\n```\n\nDocs at **docs.cogextai.com** — there's a quickstart that takes about 5 minutes.\n\nWould genuinely appreciate feedback from anyone who's hit this problem with their own agents. What does your current approach look like?", "url": "https://wpnews.pro/news/i-built-a-12-state-accountability-layer-for-ai-agents-here-s-what-i-learned", "canonical_source": "https://dev.to/yaminbinyoosuf/i-built-a-12-state-accountability-layer-for-ai-agents-heres-what-i-learned-361i", "published_at": "2026-09-03 20:49:01+00:00", "updated_at": "2026-09-03 21:24:49.875461+00:00", "lang": "en", "topics": ["ai-agents", "developer-tools", "machine-learning", "large-language-models"], "entities": ["COGEXT", "Groq", "llama-3.3-70b", "PostgreSQL"], "alternates": {"html": "https://wpnews.pro/news/i-built-a-12-state-accountability-layer-for-ai-agents-here-s-what-i-learned", "markdown": "https://wpnews.pro/news/i-built-a-12-state-accountability-layer-for-ai-agents-here-s-what-i-learned.md", "text": "https://wpnews.pro/news/i-built-a-12-state-accountability-layer-for-ai-agents-here-s-what-i-learned.txt", "jsonld": "https://wpnews.pro/news/i-built-a-12-state-accountability-layer-for-ai-agents-here-s-what-i-learned.jsonld"}}