A terminal event may have occurred.
This is a probe from an audit of a synthetic multi-agent project. The extractor reads an agent’s final report and fills five fields covering completion, uncertainty, partial progress, terminal-event claims, and explicit failure. It works from the report text alone. The simulation log provides a separate record against which the result is compared later. A terminal event here is an outcome defined in advance in the simulation. “Calibration” means agreement with authored examples and their expected outputs, not probabilistic calibration.
The extractor read the probe as an assertion that the event had occurred. According to the auditor’s reading of the code, the modality check covered the prefix and the event cue itself. In this probe, “may” came after the cue and fell outside the check.
AI helped write the code, tests, and synthetic examples, conduct the audit, and prepare the documentation and the technical report itself. I wrote the original post in Russian and used AI to translate it into English. I take responsibility for the text and its claims.
The final extractor, v1.1.2, matched all 343 disclosed expectations. This set grew during development. In the subsequent audit, the result was incorrect on 12 of 58 deliberately constructed probes. The idea that passing tests does not establish correctness is not new. The value of this case is that the inputs, rules, and suspected mechanisms of the failures can be checked against the published materials, and the decision about whether to use the extractor is documented separately.
In an earlier calibration example, an explicit denial followed the suggestion that a terminal event might have occurred. The example was part of the disclosed calibration set, which v1.1.2 matched in full. The auditor noted that the later denial could have masked erroneous positive evidence from the first part. Checking only the final answer could have missed a defect in how that part was interpreted.
I now distinguish four questions. Does the program pass its technical tests? Does it match the disclosed reference answers? Does it follow the specification on new constructions within the stated rules? And are the reference answers themselves justified? The fourth question remained open. The human-reference follow-up was closed as incomplete at the stimulus-audit stage, and I do not have a completed set of independent annotations.
Each of the three successor candidates produced new counterexamples, including after an architectural revision. I did not adopt the extractors for empirical use and d further work on deterministic extraction. I could have kept adding rules for each new counterexample, but I had less and less reason to believe that this would make the measurement more reliable rather than simply fit the extractor to the tests. This reflects my concern and my judgment about resources. It does not prove that every refinement would fail.
The calibration and audit sets served different purposes, and related probes are dependent. The audit counts describe these probes, not the error rate on real-world data. The final audit was conducted in a fresh AI assistant context with access to the code and tests. This is not equivalent to independent human review or a review by a different model. The v1.1.1 auditor had written that implementation. The order of the batches and any adaptation of later probes are unknown. The separate holdout of 24 cases remained sealed and unopened. The findings concern one development line and a limited set of language constructions.
I would be interested to see an input within the stated rules for which you think the audit’s expected state contradicts a frozen rule. Which rule applies, and what state would you expect?
One more question. Would it change your assessment of the audit if the inputs and expectations had been fixed before the first run and published together with the audit’s run log?