{"slug": "what-can-one-ai-audit-record-actually-prove", "title": "What Can One AI Audit Record Actually Prove?", "summary": "A developer has proposed a five-layer evidence model for AI audit records, arguing that a single signed record typically proves only one or two of five distinct claims: that the event was recorded, that the record's source and integrity hold, that execution is linked to the record, that execution paths were covered, and that the external business result was verified. Using a sample record with a continuous hash chain, the analysis shows the audit row's tool and outcome fields were empty, with the tool name recoverable only from a detail string, and the side-effect row lacked a before_snapshot — demonstrating that a valid signature authenticates bytes, not meaning.", "body_md": "An audit record is on the table: one confirmation, one execution, one side effect. The hash chain verifies.\n\nWhat does that prove?\n\nOne sentence: one record does not prove all five layers: a valid signature reaches only the second, and coverage and the external result are separate checks.\n\nMost people read it as two things: the event happened, and the record is true. There are five different questions inside that sentence, and one record usually answers only one or two of them.\n\nThis piece gives a five-layer evidence model. If you only want the conclusion, you can stop here. The rest is the failure modes and the five claims behind that sentence, and you can skip it. The bold sentence in each section carries the point.\n\nEvent record: what did the system record?\n\nSource and integrity: who produced the record, and was it altered?\n\nExecution linkage: does this record correspond to the path that actually ran?\n\nCoverage: which execution paths were verified?\n\nResult verification: what actually happened to the business side effect?\n\nThe layers build on each other, but they do not upgrade automatically. A higher layer cannot be inferred from a lower layer's fields. A signature does not add coverage. A coverage statement does not verify an external result.\n\nconfirmation #3   create_followup_task  R3  approved\n\n  decided 05:54:37.327   claimed .332   executed .365   error null\n\naudit #27         action=tool_call   tool=null   outcome=null   is_error=0\n\n  detail=\"create_followup_task({\\\"customerId\\\":1,\\\"title\\\":\\\"Followup Task\\\"})\"\n\n  hash=b0d6bf…   prev_hash=565b3f…\n\neffect #1         crm_task:10\n\n  before_snapshot=null   after_snapshot={\"id\":10,...}   revoke_class=local_compensate\n\nThe three rows together are \"one operation\".\n\nBut there is a non-obvious detail: audit row #27 has tool and outcome empty. The tool name exists only inside the detail string, and the result can only be inferred from is_error=0. A later row in the same database (#37, later that day) has tool=create_followup_task and outcome=executed. The same kind of record can look different at different times.\n\nThe record is not false. It is authentic, on the chain, and the hashes are continuous. It simply does not explain itself. A reader who does not know the detail convention cannot say what ran or what the result was.\n\nReading the record layer by layer\n\nLayer 1, event record: three rows exist, a confirmation, an audit row and a side-effect row. It does not prove the event happened as recorded; the record itself can be wrong.\n\nLayer 2, source and integrity: the audit row is on the chain, with continuous hash and prev_hash. It does not prove the tail was not deleted or that nothing was omitted. The confirmation and side-effect rows are not on this chain.\n\nLayer 3, execution linkage: the confirmation's execution axis is executed, and the effect is crm_task:10. It does not prove this effect was produced by this confirmation; the link is by tool, argument hash and time, not one foreign key.\n\nLayer 4, coverage: the package's coverage statement can be consulted separately. One record does not say other execution paths were covered.\n\nLayer 5, result verification: after_snapshot is present. It does not make the change comparable, because before_snapshot is null, and it does not prove the external target's state.\n\nThe strongest layer in this record is layer 2. The weakest is layer 5. It is a real record, not a complete piece of evidence.\n\nTwo counterexamples\n\nCounterexample A: a valid signature over an incomplete record.\n\nAudit row #27 is on the chain, with continuous hash and prev_hash. But its tool and outcome are empty, and the side-effect row has no before_snapshot. So:\n\nyou can prove the audit row was not altered\n\nyou cannot use this record alone to say which tool ran, what the result was, or what changed before it\n\nA signature authenticates bytes, not meaning. A real, signed record can still fail to answer the question it appears to answer.\n\nCounterexample B: a complete record with an unknown business result.\n\nThis is a design-level scenario, not a test of the record above. A write is sent to an external system. The local side-effect row says executed, but the external system is unreachable and has no query endpoint. The correct conclusion is unknown, not success.\n\nA complete local record only says the local side is complete. An unverified external state cannot be written as success.\n\nL1: a record exists is not the same as the event happened.\n\nL2: a valid chain or signature is not the same as no omissions and no tail deletion.\n\nL3: a record is linked to an execution is not the same as the execution matched its authorization.\n\nL4: one action was covered is not the same as every path was covered.\n\nL5: a local result was recorded is not the same as the external business state was verified.\n\nThese claims do not try to prove that AI never makes mistakes. They ask which guarantees a system can offer when AI misreads, overreaches, retries, times out or hits an exception, and how far each guarantee can be verified.\n\nThey also explain why adding fields does not add guarantees: a signature at L2 does not make coverage at L4 true, and coverage at L4 does not make an external result at L5 verified. Each layer needs its own evidence.\n\nThe test of these claims is not more terminology. The test is whether they can point at a counterexample, narrow a guarantee, or turn a vague promise into a verifiable criterion. All five are candidates, not proven theorems.\n\nBoundaries: the record above already shows three. Structured fields can be empty. A side-effect row can lack a before snapshot. The confirmation and side-effect rows are not on the audit chain. Beyond that: without an external anchor, tail completeness can only be unknown; an unreachable external result is unknown; the operation audit row scope is not yet narrowed by organisation.\n\nWhat is not done: the five-layer model is not an API or a product feature. It is a way to read evidence, which you can apply to any agent runtime without waiting for us to ship five endpoints.\n\nWhat we do: split evidence into five layers and say what each cannot prove; show a real record where \"authentic but incomplete\" is visible; keep external results and tail completeness as unknown instead of writing them as success.\n\nThe mechanism above is taken from KeelBase, an open-source runtime that sits between an AI agent and a business system, re-checks identity, authorization, scope, risk and confirmation state before a tool runs, and writes every decision into a record that can be verified offline. The five-layer model is a way to read evidence, not a shipped feature; KeelBase is its implementation and evidence carrier.\n\nRepository: [https://github.com/rain6fish/KeelBase](https://github.com/rain6fish/KeelBase)\n\nFour things stated plainly. The record above comes from a local run, not production data. The five-layer model is a method, not a shipped feature. External results and tail completeness remain unknown, and the operation audit row scope is still an open item. The five layers are candidates, and counterexamples are welcome.\n\nAn honest record does not claim all five layers are full. It says where each layer stops.\n\nDrafted with an AI assistant. The system, the positions and the mistakes are mine.", "url": "https://wpnews.pro/news/what-can-one-ai-audit-record-actually-prove", "canonical_source": "https://dev.to/rain6fish/what-can-one-ai-audit-record-actually-prove-445d", "published_at": "2026-10-26 01:06:39+00:00", "updated_at": "2026-10-26 01:13:05.106558+00:00", "lang": "en", "topics": ["ai-safety", "ai-agents", "artificial-intelligence"], "entities": [], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/what-can-one-ai-audit-record-actually-prove", "markdown": "https://wpnews.pro/news/what-can-one-ai-audit-record-actually-prove.md", "text": "https://wpnews.pro/news/what-can-one-ai-audit-record-actually-prove.txt", "jsonld": "https://wpnews.pro/news/what-can-one-ai-audit-record-actually-prove.jsonld"}}