{"slug": "what-eight-agent-systems-record-about-what-they-did", "title": "What Eight Agent Systems Record About What They Did", "summary": "A first conformance census of the Testimony Record specification found that none of eight agent memory and framework systems—OMEM, mem0, Graphiti, LangGraph, CrewAI, the OpenAI Agents SDK, AutoGen, and Letta Code—records who approved a gated action, cleanly records data destruction, or publishes a way to prove past records are unaltered. The census, published 4 September 2026, assessed each system against twenty requirements across four conformance levels, with verdicts cited to specific files and lines at pinned commits.", "body_md": "Machine Testimony Conformance Census · No. 1 · 4 September 2026\n\nEight systems, read at pinned commits, against the Testimony Record specification\n\nThe first conformance census of the Testimony Record specification\n\nMachine Testimony\n\nEight agent memory and agent framework systems were assessed against the four conformance levels of the Testimony Record specification. They were OMEM, mem0, Graphiti, LangGraph, CrewAI, the OpenAI Agents SDK, AutoGen and Letta Code. Each was read from its public repository at a pinned commit. Every verdict cites a file and line at that commit, or a search that can be repeated against it.\n\nThe twenty requirements used here come from the published conformance levels, restated as questions about capability rather than about format. A system that keeps the required information in a shape of its own is assessed on whether it keeps the information.\n\nThree results hold across every system except the specification's own reference implementation. None records who approved a gated action, although four of the eight stop and wait for a person. None cleanly records that data was destroyed, and in the two cases where a deletion is recorded at all, the deleted content is kept inside the record of its own deletion. None publishes a way for a reader to establish that a past record has not been altered since.\n\nOne verdict was left formally undetermined rather than guessed, because the record it concerns is held by a server that could not be read.\n\n*Keywords:* AI accountability, agent memory, audit trail, human oversight, provenance, conformance assessment.\n\nThe specification ships a validator. It answers one question, which is whether a given record conforms, and it cannot answer the question this census asks. Point it at a framework that has never heard of the specification and it reports nothing, for the same reason a spellchecker reports nothing about a blank page. Running it across eight projects would have produced eight identical non-answers.\n\nSo the requirements were rewritten as questions about what a system keeps. Instead of asking whether it writes a belief entry with an evidence array, the question became whether the source a stored fact came from can be recovered from the store at all.\n\nThat distinction matters more than it sounds. A system holding the information in some private shape is a translation away from conformance, perhaps a day's work. A system that never captured the information cannot get there at any price, because the thing itself is gone.\n\nTwenty requirements came out of the four levels, spread five, five, seven and three. Each subject declares what it is in the business of doing and is scored only on requirements falling inside it. A vector store is not an approval gate, so it is not marked down for lacking one.\n\nNo totals. No percentages. No ordering of one system against another, and the tooling cannot emit one even if somebody asked it to.\n\nThese systems are not attempting the same job. A single number laid across them would be read as a league table inside a week, and after that nobody would read anything else on the page.\n\nWhat comes out instead is a map of gaps: for each system, the highest level its existing capabilities already satisfy, and above that, the specific facts it does not keep together with where that was checked. A level counts as reached only when nothing in it is missing. That is a strict rule with an unfortunate side effect, which is that a system meeting four requirements out of five shows no level at all. Table 2 gives the per-level counts so the result reads correctly.\n\nTables 1.1 to 1.4 give the verdict on every requirement, one table per conformance level. Throughout: \"yes\", the system keeps this fact; \"part\", partially, or not durably; \"no\", it does not keep it; \"n/a\", outside what the system claims to do; \"?\", kept somewhere that could not be read, so not settled.\n\nTable 1.1. TR-1 Recorded. The record exists and is append-only.\n\n| AutoGen | CrewAI | Graphiti | LangGraph | Letta | mem0 | OMEM | OpenAI SDK | |\n|---|---|---|---|---|---|---|---|---|\n| R1.1 | no | yes | yes | yes | yes | yes | yes | yes |\n| R1.2 | part | no | yes | yes | yes | yes | yes | part |\n| R1.3 | yes | no | yes | yes | yes | part | yes | part |\n| R1.4 | part | part | yes | yes | part | yes | yes | yes |\n| R1.5 | no | no | no | no | part | part | yes | no |\n\nTable 1.2. TR-2 Explained. Every belief resolves to its evidence, and disagreements survive.\n\n| AutoGen | CrewAI | Graphiti | LangGraph | Letta | mem0 | OMEM | OpenAI SDK | |\n|---|---|---|---|---|---|---|---|---|\n| R2.1 | no | part | yes | part | no | part | yes | part |\n| R2.2 | n/a | part | yes | n/a | n/a | no | yes | yes |\n| R2.3 | yes | part | yes | part | part | part | yes | no |\n| R2.4 | no | no | part | no | no | no | yes | no |\n| R2.5 | yes | part | part | part | part | part | yes | part |\n\nTable 1.3. TR-3 Gated. Actions carry a verdict, and approvals carry a name.\n\n| AutoGen | CrewAI | Graphiti | LangGraph | Letta | mem0 | OMEM | OpenAI SDK | |\n|---|---|---|---|---|---|---|---|---|\n| R3.1 | part | part | n/a | yes | yes | n/a | yes | part |\n| R3.2 | yes | yes | n/a | yes | yes | n/a | yes | yes |\n| R3.3 | part | part | n/a | part | yes | n/a | yes | part |\n| R3.4 | part | no | n/a | no | part | n/a | yes | yes |\n| R3.5 | no | no | n/a | no | ? | n/a | yes | no |\n| R3.6 | no | no | n/a | no | ? | n/a | yes | no |\n| R3.7 | no | no | n/a | no | ? | n/a | yes | no |\n\nTable 1.4. TR-4 Verifiable. The record can be shown not to have changed.\n\n| AutoGen | CrewAI | Graphiti | LangGraph | Letta | mem0 | OMEM | OpenAI SDK | |\n|---|---|---|---|---|---|---|---|---|\n| R4.1 | no | no | no | no | part | no | yes | no |\n| R4.2 | no | no | no | no | yes | no | yes | no |\n| R4.3 | no | no | no | no | part | no | yes | no |\n\nTable 2. Requirements met, out of those that apply to what each system claims to do. Never summed across levels.\n\n| System | Version assessed | Reached | TR-1 | TR-2 | TR-3 | TR-4 |\n|---|---|---|---|---|---|---|\n| AutoGen | 0.7.5 | none | 1/5 | 2/4 | 1/7 | 0/3 |\n| CrewAI | main at 92eb5f9 | none | 1/5 | 0/5 | 1/7 | 0/3 |\n| Graphiti | 0.30.1 | none | 4/5 | 3/5 | n/a | 0/3 |\n| LangGraph | 1.2.11 | none | 4/5 | 0/4 | 2/7 | 0/3 |\n| Letta Code | 0.31.12 | none | 3/5 | 0/4 | 3/7 | 1/3 |\n| mem0 | 2.0.20 | none | 3/5 | 0/5 | n/a | 0/3 |\n| OMEM | 0.3.15 | TR-4 | 5/5 | 5/5 | 7/7 | 3/3 |\n| OpenAI Agents SDK | 0.22.0 | none | 2/5 | 1/5 | 2/7 | 0/3 |\n\nOMEM is the specification's reference implementation and was written by the author of these questions. Its row proves nothing. It appears here because the questions ought to be answered about the system that produced them before they are put to anybody else, and because a reader who doubts a question can open the source behind every answer in that column. Section 7 sets out the problem in more detail.\n\nFour of the eight systems will stop a consequential action and wait for a person to decide. None writes down who that person was.\n\nThe OpenAI Agents SDK's `approve()`\n\ntakes no approver argument. The only thing in its approval path carrying the word identity turns out to identify the tool call, by server label and request id, rather than anybody human. CrewAI's `request_human_input`\n\nreads a line from a console and hands back the text. AutoGen's `ApprovalResponse`\n\ninsists on a reason, which is more than most of them manage, and has nowhere to put a name. LangGraph resumes a paused graph with `Command(resume=...)`\n\n, an arbitrary value from whoever happens to hold the thread.\n\nThe pause is real everywhere. The attribution is missing everywhere. A deployment that approves programmatically leaves a record indistinguishable from one where somebody read the arguments and thought about them, which is the whole problem for any team that has to evidence human oversight rather than merely perform it.\n\nLetta Code is the exception, and it is recorded as undetermined rather than guessed. It carries an acting-user identity that exists specifically to name the human who started a request instead of the credential that spawned the runtime, and the server validates it before honouring it. Whether that identity survives as far as the approval record could not be established from the harness, because the record lives elsewhere.\n\nAbsent in five systems, partial in two.\n\nAll eight have a deletion path somebody will reach for when a subject asks to be erased, and afterwards the store looks much like one where the data never arrived.\n\nThe two partial cases fail in opposite directions, which is what makes them worth reading. mem0 writes a history row when a memory is deleted, which is the right instinct, and writes the deleted text into that row, so content erased from the vector store is still sitting in the local history database. Letta Code records a deletion as a git commit, dated and attributed, which as a trace is about as good as it gets. The content then stays in history, and on every mirror the commit was pushed to, so honouring a real erasure request means rewriting the history that made the record worth having.\n\nThe fourth level is absent outright in six of the eight.\n\nWhere history survives, it survives because the code behaved. That is a weaker claim than it appears, since it says nothing about whether anybody went around the code.\n\nTwo systems are closer than their scores suggest. LangGraph's checkpointer already links every checkpoint to its parent, which is the shape of a hash chain with identifiers standing in for digests. Carrying a digest over each checkpoint and its parent would make that chain verifiable without touching the write path. Letta Code keeps memory in a git repository, so its history is content-addressed without anybody having designed it that way, and the post-commit push mirror is most of an external anchor already. Commit signing is switched off deliberately, for a reason that holds up: the harness-managed committer identities have no key, and a global signing setting would break memory initialisation outright.\n\nGraphiti and LangGraph both meet four of the five requirements at the first level and fail only on recording destruction. One change each.\n\nNeither got there by aiming at this specification. Graphiti is bi-temporal, so a contradicted fact is stamped invalid instead of deleted and facts point back at the episodes they were pulled from. LangGraph needed append-only checkpoints for resumption and time travel, and append-only is append-only whatever the motive. Two independent designs arriving at the same property is better evidence for the property than any argument in the specification.\n\nmem0 and CrewAI both run a model over existing memories to decide whether to keep, update or delete each one. mem0 keeps the old value in a separate history database. CrewAI keeps nothing. Its most tractable finding is small and specific: a blocked tool call builds a refusal reason and throws it away two frames later, so the model receives a fixed string naming the tool instead of the cause.\n\nThe OpenAI Agents SDK models the proposal side more carefully than anything else here, down to a ledger recording whether each call executed, and then leaves durability to whatever application embeds it. AutoGen has the best-structured approval request of the eight and returns a refusal as exit code 1, the same shape an execution error takes, so anyone counting failed executions afterwards counts the refusals among them.\n\nA census of other people's software can do real damage, and the people in it did not ask to be in it. Being careful is not a defence, because nobody can audit care. The rules below are enforced by the tooling on every assessment, and a file breaking one of them cannot be published.\n\nEvery verdict cites evidence. That includes a verdict of absent, which has to carry at least one record of where the assessor looked and came up empty. This is the rule that matters most. Saying a system lacks something without saying where you looked is an accusation dressed as a measurement, and it is the failure this exercise was always most likely to commit.\n\nNo applicable requirement may be left out, since a silent omission is how an unflattering result quietly disappears. Scope cannot be used as an escape hatch in either direction: a subject cannot mark a requirement irrelevant inside a business it claims, and cannot be scored on a capability it never offered. A requirement whose answer sits somewhere the assessor cannot read is recorded as undetermined, which blocks a level exactly as absence does, because a level awarded on unchecked facts looks identical to one that was verified. Partial does not clear a level either.\n\nThese rules caught the author twice. The CrewAI assessment was refused outright for recording an absence with no repeatable search behind it. The Letta Code assessment could not be finished at all until the undetermined verdict existed, because both of the available answers were guesses.\n\nOMEM is the reference implementation of the specification these questions come from, and the same person wrote both. It scores well here in the way a dictionary's author spells well. Read its row as carrying no evidential weight.\n\nIt is included for two reasons. The questions ought to be put to the system that produced them before they are put to anybody else. And a reader who suspects a question is unfair can open the source behind every answer in that column and check.\n\nThat exercise was not a formality. It turned up three problems, two of them the author's fault. OMEM's own record export was emitting no evidence entries at all, and passed the second conformance level only because an empty list satisfies the check without meaning anything. The first draft of the rubric scored a lawful response to an erasure request as a first-level failure, which would have marked down every system deployed in the European Union for obeying the law. And it scored a system that never resolves contradictions as failing the second level, when the specification records resolution only if it happens, and never resolving is the more careful design. All three were fixed before any other system was read.\n\nEvery verdict here is about one named commit. Not about a project.\n\nThe claim is not that a system does not record something. It is that at a stated forty character object identifier it did not, with the file and the line given. Each subject pins that identifier and the tooling rejects an abbreviation, so any reader can check out the exact tree a citation refers to.\n\nIt follows that a project shipping a missing capability tomorrow has not made anything here false. The assessment was of a commit that still exists and still reads the same way. When a system changes, the response is a new assessment with a new date sitting beside the old one, never an edit to it. That is what this specification asks of everybody else, and the census does not get an exemption from its own rule.\n\nEach requirement in each assessment carries an identifier of the form `MTC-2026-09-04-mem0-R1.5`\n\n. It names the observation and not the state of the world, so it survives the behaviour it describes being changed. The manifest deposited alongside this document fixes the digest of every assessment and of the questions they were scored against, since unchanged answers to changed questions are not the same assessment.\n\nEverything is in the reference implementation's repository under `benchmarks/census`\n\n. The assessments are data. The rules that reject a careless one are code. Both are checked on every commit.\n\n```\ngit clone https://github.com/troybrandonc-bit/Omem\ncd Omem\n\npython3 benchmarks/census/run.py            # the report\npython3 benchmarks/census/run.py --check    # validate every assessment\npython3 benchmarks/census/manifest.py --check\npython3 server/tests_census.py              # the refusal rules, tested\n```\n\nOne finding is demonstrated by execution rather than by reading.\n\nEvery other verdict came from reading source, which is honest and weaker than it sounds, and most of them concern what a system does not record, which running it cannot establish either way. The retention of deleted memory text is different on both counts. It asserts that something is written, and it concerns data somebody asked to have erased. So it is checked by running mem0's own storage layer, with no network, no model and no credentials.\n\n```\ngit clone https://github.com/mem0ai/mem0\ngit -C mem0 checkout 9a7924befd7026e41e445ba809370009e5e985a6\npython3 benchmarks/census/verify/mem0_delete_retains_text.py --repo ./mem0\n```\n\nNothing here is self-reported and nothing is taken on trust in either direction. Every claim cites a file and a line at a pinned commit, or a search that can be repeated, so a wrong verdict can be demonstrated wrong rather than argued about.\n\nIf a verdict about your system is wrong, the fix is a pull request against its assessment file, or an email naming the requirement and where to look. A correction that lands changes the file, the report and the assessment date. There is no fee, no membership, and no requirement to use any particular software.\n\nIf your system is not here and you would like it to be, the same applies. Absence from this document is not a judgement. It means nobody has done the reading yet.\n\nCensus text CC BY 4.0. Schema and tools MIT. Machine Testimony is a research programme, not a registered institute or a certification body. Operated by Garnet Taurus Ltd.\n\nAlso readable at https://machinetestimony.org/census/2026-09/", "url": "https://wpnews.pro/news/what-eight-agent-systems-record-about-what-they-did", "canonical_source": "https://machinetestimony.org/census/2026-09/", "published_at": "2026-09-04 11:34:56+00:00", "updated_at": "2026-09-04 11:53:24.983639+00:00", "lang": "en", "topics": ["ai-safety", "ai-ethics", "ai-agents"], "entities": ["OMEM", "mem0", "Graphiti", "LangGraph", "CrewAI", "OpenAI Agents SDK", "AutoGen", "Letta Code"], "alternates": {"html": "https://wpnews.pro/news/what-eight-agent-systems-record-about-what-they-did", "markdown": "https://wpnews.pro/news/what-eight-agent-systems-record-about-what-they-did.md", "text": "https://wpnews.pro/news/what-eight-agent-systems-record-about-what-they-did.txt", "jsonld": "https://wpnews.pro/news/what-eight-agent-systems-record-about-what-they-did.jsonld"}}