I built a 12-state accountability layer for AI agents here's what I learned

A developer built COGEXT, an accountability layer that tracks commitments made by AI agents through a 12-state state machine, addressing the problem of agents forgetting promises. The system uses a two-stage extraction pipeline with classification and structured extraction via Groq's llama-3.3-70b, and a weighted field-coverage model for evidence verification.

By the time you finish reading this, your AI agent will have made at least one promise it can't keep. That's not a dig at your code. It's a fundamental property of how we build AI systems today. Agents are brilliant at generating intent and terrible at tracking it. They'll promise to email a client, confirm an appointment, or deploy a fix and two conversations later, they've forgotten they said anything at all. The more I thought about this, the more I realized: commitments aren't a memory problem. They're a state machine problem. So I built COGEXT — an accountability layer that sits between your agent and the outside world. You send it agent output, it extracts every commitment, and tracks each one until it's fulfilled, failed, or overdue. Most systems treat agent output as text to be logged. We treat it as events to be tracked. Every extracted commitment moves through 12 states: DETECTED ├──→ OPEN ──→ DUE ──→ OVERDUE ──→ FULFILLED ✓ │ ├──→ FAILED ✗ │ └──→ EXPIRED ✗ ├──→ PENDING REVIEW ──→ OPEN ├──→ CANCELLED ✗ ├──→ CONTRADICTED ✗ └──→ BLOCKED ──→ OPEN └──→ FAILED ✗ Terminal states: fulfilled , failed , expired , cancelled , superseded , contradicted . All state mutations go through a single PostgreSQL RPC function. Not direct UPDATEs from application code: CREATE OR REPLACE FUNCTION cogext transition commitment p commitment id UUID, p new status TEXT, p actor TEXT DEFAULT 'system', p data JSONB DEFAULT '{}' RETURNS JSONB AS $$ BEGIN -- lock row → validate transition → update status → insert event → return -- single atomic operation: if any step fails, the whole thing rolls back END; $$ LANGUAGE plpgsql; Why? Because the state change and the event must be atomic. You should never have status = 'fulfilled' without a corresponding fulfillment event in the audit log. That's the line between a logging system and an accountability system. The extraction pipeline has two stages. Stage 1: Classification Not everything an agent says is a commitment. We classify first: genuine commitment — "I will send the report by Friday." intention — "I might send the report tomorrow." suggestion — "We should send the report tomorrow." hypothetical — "If we have time, I could send it." quoted statement — "John said he would send it."Only genuine commitment with confidence ≥ 0.5 passes through. Stage 2: Structured extraction We use Groq's llama-3.3-70b in JSON mode to extract fields: response = await groq client.chat.completions.create model="llama-3.3-70b-versatile", messages= {"role": "user", "content": prompt + message} , response format={"type": "json object"}, temperature=0.2, The output is validated against a Pydantic schema. If the LLM returns garbage, we retry once. If it still fails, the commitment is dropped — not inserted. Key decision: We don't trust the LLM to interpret deadlines. We extract the raw expression "Friday EOD" and pass it to a deterministic temporal normalizer. The LLM interprets; the code decides. Evidence verification is where most systems fail. Binary matching — "does this email mention the report?" — generates too many false positives. We use a weighted field-coverage model: | Field | Weight | |---|---| action | 0.40 | recipient | 0.30 | object | 0.20 | deadline | 0.10 | When evidence arrives, we check which commitment fields it matches and sum the weights: Evidence 1: confirms action "I sent it" → 0.40 Evidence 2: confirms recipient + object "sent to Sarah, deployment-report.pdf attached" → 0.30 + 0.20 = 0.50 Total = 0.90 → FULFILLED The threshold is configurable default 0.90 . Multiple pieces of evidence can satisfy different fields — they aggregate. AI agents repeat themselves. An agent might say: Same commitment. Without idempotency, you'd track it three times. python def compute idempotency key agent id, promise text, timestamp : window = timestamp.replace minute=0, second=0, microsecond=0 payload = f"{agent id}|{promise text.strip .lower }|{window.isoformat }" return hashlib.sha256 payload.encode .hexdigest Same agent + same promise + same hour = same hash. The DB unique constraint rejects the duplicate silently. The hour window is deliberate: it deduplicates within a session while still allowing the same promise to be legitimately re-made days later. 1. LLM output is untrusted input. We validate everything against Pydantic schemas before touching the database. No exceptions. 2. Determinism over magic. LLMs for interpretation. Deterministic code for everything else: state transitions, date normalization, deduplication, metrics. We don't call the LLM to check if a deadline is overdue. We run a database query. 3. Events are not logs. A log is a record that can be ignored. An event is a fact that cannot be denied. We enforce this at the DB layer — a trigger rejects any UPDATE or DELETE on the events table. Historical events are immutable. 4. Scope the user id at the auth layer, never the request body. Early version trusted user id from the request payload. Classic mistake. All scoping now comes from the API key's account id . The request body cannot claim to own data it doesn't own. The API is live and the SDK is on PyPI: pip install cogext python from cogext import track agent = track your agent, api key="cg live xxx" output = agent.run "I'll send the report by Friday" commitment is tracked automatically Docs at docs.cogextai.com — there's a quickstart that takes about 5 minutes. Would genuinely appreciate feedback from anyone who's hit this problem with their own agents. What does your current approach look like?