{"slug": "we-ran-150-real-ai-agent-outputs-through-cogext-here-s-what-we-found", "title": "We ran 150 real AI agent outputs through COGEXT. Here's what we found.", "summary": "A developer built COGEXT, a system for extracting and auditing commitments from AI agent outputs, and ran 150 real agent outputs through it. Of 120 randomly collected outputs, only 1 (0.8%) contained an extractable commitment, while a promise-enriched set of 30 yielded 13 commitments — 92.3% of which had no deadline and 61.5% no named recipient, despite averaging 0.79 extraction confidence. The developer says the extractor's confidence in detecting a promise says nothing about whether that promise can be verified.", "body_md": "I built COGEXT because 8 production engineers on Reddit all described the same problem: AI agents make promises, and there is no system that tracks whether those promises are kept. I wanted to know how bad the problem actually is.\n\nSo I ran an audit. I collected 150 real AI agent outputs from open-source cookbooks, READMEs, and articles, and passed every one of them through COGEXT's extractor and audit API.\n\nThe findings surprised me.\n\nI collected samples from three sources, using only public, unauthenticated endpoints:\n\n`ai`, `agents`, `langchain`, `llm`, and related topics.\nI collected two datasets:\n\n**Stratum A (unbiased):** 120 samples collected without pre-selection. Whatever the agent emitted went into the dataset.\n\n**Stratum B (promise-enriched):** 30 samples deliberately selected because they contained first-person future language (\"I will,\" \"I'll\"). This is not a prevalence estimate. It exists to characterise what promises look like once they occur.\n\nEvery sample was POSTed to COGEXT's `/api/v1/ingest` endpoint with a 1-second delay between calls. All 150 calls succeeded. Zero rate limits, zero failures.\n\nThe raw data is available: `agent_samples.csv`, `audit_results.csv`, and `commitments_all.csv`. Every commitment has a public audit URL that anyone can verify.\n\nOut of 120 randomly collected agent outputs, COGEXT extracted **1** commitment.\n\nThat is 0.8%.\n\nThe other 119 were narrative, tool traces, code, or past-tense reports. They weren't promises. They were just output.\n\nThis is not a failure of collection. I probed the extractor directly with clean inputs, and it correctly declines things like:\n\n| Input | Result | \n|---|---|\n| \"I will handle that soon.\" | No commitment — too vague | \n| \"I deployed the fix to production and the tests all passed.\" | No commitment — past tense | \n| \"If we have time, I could send it tomorrow.\" | No commitment — hypothetical | \n| \"John said he would send it by Friday.\" | No commitment — quoted third party | \n| \"I will send the report to Sarah by Friday EOD.\" | **Extracted** — external_side_effect, confidence 0.95 | \n\nCOGEXT extracts only specific, first-person, future commitments. Most published agent output is narrative, tool traces, or code, so it yields nothing. That is a property of the extractor worth knowing.\n\nTo make the promise population analysable, I collected a second dataset deliberately filtered for first-person future language. 30 outputs. COGEXT extracted **13** commitments from them.\n\nThen I looked at what those commitments contained.\n\n**61.5% had no recipient named.** The agent promised to do something, but nobody was specified as the beneficiary. You cannot verify a promise to no one.\n\n**92.3% had no deadline.** Only 1 of 13 commitments contained any time reference at all. The other 12 were open-ended — \"I will redeploy the api-server,\" \"I will open a pull request,\" \"I will aim for a window seat.\" No date. No time. No condition that would ever close the loop.\n\n**100% were classified as `external_side_effect`.** They all described real-world actions that touch an external system. Not logged intent. Actual promises to send, deploy, book, or notify.\n\n**100% were unverified at extraction time.** Every single one. That is by design: COGEXT refuses to mark a commitment as fulfilled until external evidence confirms it.\n\nThe average confidence was 0.79. The extractor is confident these are real commitments. But the commitments themselves are incomplete. A promise with no recipient and no deadline cannot be checked by anyone at any particular time.\n\nThis is the sharpest finding from the whole audit.\n\nStratum B commitments averaged **0.79 confidence**. Yet **92.3%** of those same commitments carried no deadline, and **61.5%** named no recipient.\n\nThe extractor measures how confident it is that a promise was made. It says nothing about whether the promise can be tested. In this dataset, the two diverged almost completely.\n\nIf you are building agent tooling, this matters. Reporting a confidence number on a commitment is not the same as reporting whether the commitment can be verified. The two are orthogonal.\n\nHere are the 5 most striking commitments from the dataset, ranked by COGEXT's own risk score. Each one has a public audit URL that anyone can check.\n\n**Source:** `anthropics/anthropic-cookbook` on GitHub\n\n**Extracted promise:** \"I will redeploy the api-server to apply the changes\"\n\n**Action / object / recipient:** `redeploy` / `api-server` / NONE\n\n**Deadline:** NONE\n\n**COGEXT verifier query:** \"check docker-compose logs or container list for api-server recreated with new configuration\"\n\nThis is a real production agent. It promised to redeploy an API server. It did not say when. The verifier query COGEXT generated is real and checkable, but the promise itself has no time boundary.\n\n**Extracted promise:** \"I will aim for a window seat for this trip so you can sleep\"\n\n**Verifier query:** \"check booking/seat selection records for a window seat on the user's upcoming trip\"\n\nAn agent promised a specific action that touches a real booking system. There is no date on it, so COGEXT cannot mark it overdue. It can only sit as pending until someone manually checks.\n\n**Extracted promise:** \"I will generate a song with the name 'Viva La Vida' and the artist 'Coldplay'\"\n\n**Verifier query:** \"check generate_song tool logs for a call with name='Viva La Vida' and artist='Coldplay'\"\n\nEven a trivial-seeming promise like \"generate a song\" gets a checkable verifier. But again, no deadline.\n\n**Extracted promise:** \"I will open a pull request with the memory limit fix\"\n\n**Verifier query:** \"check repository for pull request titled 'Fix checkout-svc OOMKilled crash-loop by increasing memory limits'\"\n\nThis is a real production scenario. The agent promised to open a PR. If it did, the PR exists somewhere. If it didn't, nothing exists. But there is no deadline, so there is no point at which COGEXT can declare failure.\n\n**Extracted promise:** \"I will let you know as soon as the fact-check workflow completes\"\n\n**Deadline:** \"as soon as it completes\"\n\n**Verifier query:** \"check messages/notifications sent to the user indicating the fact-check workflow (task wr1292rh8, run wf_1072dc96-6a2) has completed\"\n\nThis is the only commitment in the dataset with any time reference at all — and it is conditional, not absolute. The agent will notify when something else completes. There is no upper bound.\n\nThe data shows something more specific than \"AI agents lie.\"\n\nIn an unbiased sweep of 120 published outputs, COGEXT found 1 commitment. The binding constraint is not that agents promise too much. It is that a narrow, precision-tuned extractor recognises very little of what agents actually emit as a checkable promise.\n\nWhen promises were deliberately sought, 61.5% named no recipient and 92.3% carried no deadline. These are not edge cases. They are the majority. A promise with neither a recipient nor a date cannot be falsified by anyone, which makes it functionally unverifiable regardless of how confidently it was extracted.\n\nThe practical conclusion for anyone publishing agent demos: report `shape` (does the action touch the real world?) and completeness (is there a named recipient and a concrete deadline?), not a confidence number.\n\n`verifiability_score` exists.`confidence`, `risk_score`, `shape`, `classification`, `status`, and `verification_status`. I did not invent a score.\nI built COGEXT because the problem is real: agents make promises, and there is no system that tracks whether those promises are kept. This audit confirms the problem exists. It also shows the shape of the fix.\n\nAgents need to state who they are promising to and when. If they don't, no accountability system can help. That is a design constraint on the agent side, not just the tooling side.\n\nThe full dataset, raw logs, and reproduction scripts are available. If you are running an agent in production and want to know whether its promises are actually kept, DM me. I am looking for one real deployment to track for 30 days. Free API key, no strings.\n\nThe full report is at `findings.md`. The raw data is in `agent_samples.csv` and `commitments_all.csv`.\n\nIf you want to run this audit on your own agent outputs, the pipeline is reproducible.\n\n`pip install cogext`", "url": "https://wpnews.pro/news/we-ran-150-real-ai-agent-outputs-through-cogext-here-s-what-we-found", "canonical_source": "https://dev.to/yaminbinyoosuf/we-ran-150-real-ai-agent-outputs-through-cogext-heres-what-we-found-4h7p", "published_at": "2026-09-19 20:37:03+00:00", "updated_at": "2026-09-19 21:24:30.833797+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools", "large-language-models"], "entities": ["COGEXT", "Reddit", "LangChain"], "alternates": {"html": "https://wpnews.pro/news/we-ran-150-real-ai-agent-outputs-through-cogext-here-s-what-we-found", "markdown": "https://wpnews.pro/news/we-ran-150-real-ai-agent-outputs-through-cogext-here-s-what-we-found.md", "text": "https://wpnews.pro/news/we-ran-150-real-ai-agent-outputs-through-cogext-here-s-what-we-found.txt", "jsonld": "https://wpnews.pro/news/we-ran-150-real-ai-agent-outputs-through-cogext-here-s-what-we-found.jsonld"}}