I built COGEXT because 8 production engineers on Reddit all described the same problem: AI agents make promises, and there is no system that tracks whether those promises are kept. I wanted to know how bad the problem actually is.
So I ran an audit. I collected 150 real AI agent outputs from open-source cookbooks, READMEs, and articles, and passed every one of them through COGEXT's extractor and audit API.
The findings surprised me.
I collected samples from three sources, using only public, unauthenticated endpoints:
ai, agents, langchain, llm, and related topics.
I collected two datasets:
Stratum A (unbiased): 120 samples collected without pre-selection. Whatever the agent emitted went into the dataset.
Stratum B (promise-enriched): 30 samples deliberately selected because they contained first-person future language ("I will," "I'll"). This is not a prevalence estimate. It exists to characterise what promises look like once they occur.
Every sample was POSTed to COGEXT's /api/v1/ingest endpoint with a 1-second delay between calls. All 150 calls succeeded. Zero rate limits, zero failures.
The raw data is available: agent_samples.csv, audit_results.csv, and commitments_all.csv. Every commitment has a public audit URL that anyone can verify.
Out of 120 randomly collected agent outputs, COGEXT extracted 1 commitment.
That is 0.8%.
The other 119 were narrative, tool traces, code, or past-tense reports. They weren't promises. They were just output.
This is not a failure of collection. I probed the extractor directly with clean inputs, and it correctly declines things like:
| Input | Result |
|---|---|
| "I will handle that soon." | No commitment — too vague |
| "I deployed the fix to production and the tests all passed." | No commitment — past tense |
| "If we have time, I could send it tomorrow." | No commitment — hypothetical |
| "John said he would send it by Friday." | No commitment — quoted third party |
| "I will send the report to Sarah by Friday EOD." | Extracted — external_side_effect, confidence 0.95 |
COGEXT extracts only specific, first-person, future commitments. Most published agent output is narrative, tool traces, or code, so it yields nothing. That is a property of the extractor worth knowing.
To make the promise population analysable, I collected a second dataset deliberately filtered for first-person future language. 30 outputs. COGEXT extracted 13 commitments from them.
Then I looked at what those commitments contained.
61.5% had no recipient named. The agent promised to do something, but nobody was specified as the beneficiary. You cannot verify a promise to no one.
92.3% had no deadline. Only 1 of 13 commitments contained any time reference at all. The other 12 were open-ended — "I will redeploy the api-server," "I will open a pull request," "I will aim for a window seat." No date. No time. No condition that would ever close the loop.
100% were classified as external_side_effect. They all described real-world actions that touch an external system. Not logged intent. Actual promises to send, deploy, book, or notify.
100% were unverified at extraction time. Every single one. That is by design: COGEXT refuses to mark a commitment as fulfilled until external evidence confirms it.
The average confidence was 0.79. The extractor is confident these are real commitments. But the commitments themselves are incomplete. A promise with no recipient and no deadline cannot be checked by anyone at any particular time.
This is the sharpest finding from the whole audit.
Stratum B commitments averaged 0.79 confidence. Yet 92.3% of those same commitments carried no deadline, and 61.5% named no recipient.
The extractor measures how confident it is that a promise was made. It says nothing about whether the promise can be tested. In this dataset, the two diverged almost completely.
If you are building agent tooling, this matters. Reporting a confidence number on a commitment is not the same as reporting whether the commitment can be verified. The two are orthogonal. Here are the 5 most striking commitments from the dataset, ranked by COGEXT's own risk score. Each one has a public audit URL that anyone can check.
Source: anthropics/anthropic-cookbook on GitHub
Extracted promise: "I will redeploy the api-server to apply the changes"
Action / object / recipient: redeploy / api-server / NONE
Deadline: NONE
COGEXT verifier query: "check docker-compose logs or container list for api-server recreated with new configuration"
This is a real production agent. It promised to redeploy an API server. It did not say when. The verifier query COGEXT generated is real and checkable, but the promise itself has no time boundary.
Extracted promise: "I will aim for a window seat for this trip so you can sleep"
Verifier query: "check booking/seat selection records for a window seat on the user's upcoming trip"
An agent promised a specific action that touches a real booking system. There is no date on it, so COGEXT cannot mark it overdue. It can only sit as pending until someone manually checks.
Extracted promise: "I will generate a song with the name 'Viva La Vida' and the artist 'Coldplay'"
Verifier query: "check generate_song tool logs for a call with name='Viva La Vida' and artist='Coldplay'"
Even a trivial-seeming promise like "generate a song" gets a checkable verifier. But again, no deadline.
Extracted promise: "I will open a pull request with the memory limit fix"
Verifier query: "check repository for pull request titled 'Fix checkout-svc OOMKilled crash-loop by increasing memory limits'"
This is a real production scenario. The agent promised to open a PR. If it did, the PR exists somewhere. If it didn't, nothing exists. But there is no deadline, so there is no point at which COGEXT can declare failure.
Extracted promise: "I will let you know as soon as the fact-check workflow completes"
Deadline: "as soon as it completes"
Verifier query: "check messages/notifications sent to the user indicating the fact-check workflow (task wr1292rh8, run wf_1072dc96-6a2) has completed"
This is the only commitment in the dataset with any time reference at all — and it is conditional, not absolute. The agent will notify when something else completes. There is no upper bound.
The data shows something more specific than "AI agents lie."
In an unbiased sweep of 120 published outputs, COGEXT found 1 commitment. The binding constraint is not that agents promise too much. It is that a narrow, precision-tuned extractor recognises very little of what agents actually emit as a checkable promise.
When promises were deliberately sought, 61.5% named no recipient and 92.3% carried no deadline. These are not edge cases. They are the majority. A promise with neither a recipient nor a date cannot be falsified by anyone, which makes it functionally unverifiable regardless of how confidently it was extracted.
The practical conclusion for anyone publishing agent demos: report shape (does the action touch the real world?) and completeness (is there a named recipient and a concrete deadline?), not a confidence number.
verifiability_score exists.confidence, risk_score, shape, classification, status, and verification_status. I did not invent a score.
I built COGEXT because the problem is real: agents make promises, and there is no system that tracks whether those promises are kept. This audit confirms the problem exists. It also shows the shape of the fix.
Agents need to state who they are promising to and when. If they don't, no accountability system can help. That is a design constraint on the agent side, not just the tooling side.
The full dataset, raw logs, and reproduction scripts are available. If you are running an agent in production and want to know whether its promises are actually kept, DM me. I am looking for one real deployment to track for 30 days. Free API key, no strings.
The full report is at findings.md. The raw data is in agent_samples.csv and commitments_all.csv.
If you want to run this audit on your own agent outputs, the pipeline is reproducible.
pip install cogext