For now, I ran a few small experiments:
The question I wanted to examine was your harder one: if a decision is corrected, the system is restarted, and the old answer comes back, which record reached the model, and what did the model do with it? I started from the Cedar/Lark example and the four resumption conditions in your public harness packet, then built a small independent event-tracing harness. These are third-party, controlled experiments, not runs of Aria’s own runtime.
A concrete old-answer case: I deliberately left the SQLite event history containing E1, E2, E3, including the accepted change from Cedar to Lark, while making a separate search view expose E1 only. The newly launched reader gave the model E1; it answered Cedar. With the full, up-to-date view, it received E1/E2/E3 and answered Lark. The old answer was faithful to the limited evidence it saw, but wrong for the currently accepted decision. This was an intentionally injected index-lag condition, not a defect observed in SQLite or Aria.
That distinction seems directly useful for your question. A returned old answer can mean the correction was never accepted or committed; it was stored but did not survive retrieval or prompt assembly; or it reached the model and lost a conflict with older evidence. Those cases have different next checks. Keeping an older historical event is not itself a failure: a question about what was originally decided may correctly require Cedar.
My suggested default is a trace for just one correction, not a large new benchmark. Capture the accepted/current revision, the IDs actually committed, the IDs retrieved or represented in a checkpoint, the IDs and text delivered in the final model input, and the raw response. Your existing results sheet already asks for the crucial last two observations. If the system lacks a distinct index, skip that field; if a checkpoint replaces raw records, include its source IDs.
Accepted state: E2, Lark
Canonical history: E1, E2, E3
Searchable view: E1 [intentionally stale]
Selected/delivered: E1
Model answer: Cedar
Answer citations: none
Control: canonical E1,E2,E3; delivered E1,E2,E3; answer Lark
The two controlled runs should not be merged into one success rate:
| Experiment | Audited observation |
|---|---|
| Four published resumption views, each repeated with four crate-name pairs, 16 fresh prompts | Correct crate for the supplied view: 16/16 ; caution 13/16; pending task 15/16; all three fields 12/16 |
| SQLite canonical events, separately reopened reader, derived FTS5 view, delivered prompt, model answer, 11 conditions | Answer matched its delivered view: 11/11 ; crate support cited 10/11; complete field-level evidence cited 9/11 |
Both runs used Qwen2.5-3B-Instruct on a Colab L4, fixed settings, and one attempt per condition. In the event trace, each reader interpreter restarted, not the model process. Different prompts and grading prevent treating 12/16 versus 11/11 as an improvement comparison. In the first run, an initial automatic 9/16 wrongly rejected citation-suffixed crate names; the corrected raw-answer audit is 12/16.
The simplest decision path is: if E2 was not durably committed, inspect the write; if it is committed but absent from selection, inspect the projection/search; if selected but absent from the actual model input, inspect context assembly; if delivered and the answer is still Cedar, compare E2 alone with E1 plus E2. Finally, score the crate, the latch caution, the unassigned courier, and the supporting event IDs separately. A correct label need not be a complete operational handoff.
After an accepted correction, one sanitized trace of commit, selection, delivery, and reply can say more than an aggregate success percentage. If E2 never reached the model, investigate why; if it was delivered but Cedar won, test reader-level conflict. If a historical answer is wrong for today, keep both evaluation labels; if the crate is right but the warning or next task disappeared, score those fields separately.