First decide what replay is supposed to prove. Showing the same recorded steps, continuing from a saved checkpoint and rerunning the task are three different activities. A transcript can support the first while leaving the other two impossible to reproduce.
This proposed sixteen-field reference was reviewed September 12, 2026. It describes an experiment record rather than a universal agent export format. No agent run was reproduced for this article, and the worksheet contains no invented results.
- 01Name the replay mode.Recorded playback and fresh execution do not provide the same evidence.
- 02Preserve the world around the prompt.Tools, files, services and evaluation rules can change the outcome.
- 03Disclose the differences.A useful rerun may still be informative when exact reproduction is unavailable.
01 — Practical decisionChoose the claim before collecting the files #
In this guide, trace playback means displaying saved events without asking the model or tools to produce them again. Checkpoint continuation means resuming from a stored intermediate state. A fresh rerun means starting the task again from a declared initial package. These are working definitions for the comparison, not a claim that every framework uses the same terms.
LangGraph’s time-travel documentation illustrates the distinction. Nodes before a selected checkpoint use saved results, while later nodes run again, including model and API calls that may produce different results. The word replay therefore does not automatically mean repeating the original output.
Write a narrow claim such as: the saved trace shows which tool response informed the answer; or a fresh execution passes the same acceptance test. Our session export reference asks whether another environment can continue the task. This article asks what a comparison between executions can establish.
02 — Practical decisionRecord the sixteen fields that shape a rerun #
The table describes a proposed minimum inventory for a tool-using task. Store the actual artifacts where practical and give each one an immutable reference or content hash. A hash identifies bytes; it does not explain their meaning, make them available or grant permission to redistribute them.
For every field, state whether it is fixed, deliberately varied, unavailable or unknown. Keep those categories separate. An unavailable provider snapshot is a known limitation; an unrecorded snapshot is an evidence gap. The chart groups inventory fields and does not count successful replications.
| Digital Applied proposed rerun inventory, as of September 12, 2026. Sixteen fields; categories describe evidence to collect, not measured reproducibility. | ||
|---|---|---|
| Record | Group | What must be identifiable |
| --- | --- | --- |
| Task brief | Inputs | Exact initial request and acceptance target. |
| Instruction files | Inputs | Loaded contents, scope and precedence. |
| Conversation state | Inputs | Starting messages and any compacted summary. |
| Input artifacts | Inputs | File versions, hashes and accessible copies. |
| Model identity | Execution | Provider, model snapshot or unresolved alias. |
| Generation settings | Execution | Actual supported parameters and effort controls. |
| Agent harness | Execution | Code version, prompts and routing policy. |
| Runtime environment | Execution | Dependencies, operating environment and configuration. |
| Tool contracts | Outside world | Schemas, versions and argument meanings. |
| Tool responses | Outside world | Recorded fixtures versus fresh service results. |
| External data | Outside world | Snapshot, retrieval time and changed records. |
| Side effects | Outside world | Operation identifiers and verified outcomes. |
| Evaluation rules | Assessment | Checks, thresholds and evaluator version. |
| Stopping rule | Assessment | Budget, deadline and retry policy. |
| Attempt history | Assessment | Every attempted run and exclusion reason. |
| Observed differences | Assessment | Changed fields and limits on the conclusion. |
03 — Practical decisionFreeze what you control and expose what you cannot #
A code task needs the repository state that the agent actually saw. A commit alone omits uncommitted changes and untracked inputs. Keep a patch or an equivalent complete workspace snapshot, dependency resolution and the test fixtures that define the starting condition.
For a research task, record retrieved documents and the retrieval time. A live URL can change while keeping the same address. If the question concerns current information, fetching a newer page is reasonable, but that changes the experiment. Record the difference instead of quietly describing the answer as a reproduction. LangGraph persistence documentation separates thread checkpoints from application-defined stores. That is a useful reminder that one saved state object may not contain all the information a run used. External accounts, long-term memory and search results can remain outside it.
04 — Practical decisionSeparate recorded tools from live side effects #
A replay using saved tool responses asks how the agent behaves given that recorded evidence. A replay calling live tools asks how it behaves in the current environment. Both can be useful, but a result from one cannot silently stand in for the other.
Consider an illustrative task that creates a support draft and uploads an attachment. Feeding back the recorded upload response checks the later decision path without up again. Re-executing the real upload may create a second artifact. The test record should identify which operations were simulated, which were read-only and which actually changed external state.
Keep operation identifiers and outcome checks with the trace. An interrupted request may already have completed outside the agent. The test-data reference helps decide what to fake; it does not remove the need to disclose that substitution in a reproducibility claim.
05 — Practical decisionCompare the same outcome, not necessarily identical prose #
Before running the comparison, choose whether success means identical bytes, the same structured values or passing the same behavioral checks. These are different standards. A rewritten explanation can preserve the answer while a visually identical report contains a changed number.
Same prompt and same model alias are insufficient evidence of identical execution conditions. Record a snapshot identifier where the provider exposes one, together with generation settings and routing. A fixed random seed, where supported, is another controlled input; it is not a substitute for documenting tools and environment or proof of a deterministic service.
ACM SIGIR’s artifact policy distinguishes accessible artifacts and their evaluation from evidence validating results. Apply that distinction to your report: publishing a trace makes evidence inspectable, while a successful fresh rerun is a separate finding. This worksheet confers no academic badge or certification.
06 — Practical decisionReport a rerun that someone else can interpret #
A useful report names the original run, the new run, the replay mode, the controlled inputs and every consequential difference. State how many attempts you actually made and retain failures. Do not choose the most similar output after many unreported attempts and call it the reproduced result.
When a required artifact is unavailable, narrow the claim. You may be able to validate arithmetic in the final answer while being unable to recreate the search path. You may be able to rerun tests on a delivered patch while lacking the model snapshot used to generate it. These partial checks still have value when their boundaries are visible.
The benchmark evidence guide covers another limitation of interpreting scores. For AI transformation work, agree the reproduction target before purchasing evaluation time; otherwise two teams can both run carefully and answer different questions.
Download the blank worksheet. It contains the same sixteen reference rows. Observed status, evidence, unit or value, checked date and notes are blank so you can record actual findings. Use pass, fail, unknown or not applicable only after the relevant check.
Evidence and scope
- As-of date
- September 12, 2026: sources retrieved and reviewed. September 12 is the editorial allocation. Verified event dates are stated separately.
- Sources and method
- Sixteen original inventory fields, grouped four each into inputs, execution, outside world and assessment. Sources: LangGraph time travel and persistence, plus ACM SIGIR artifact policy.
- Limits
- Chart counts derive from the Group column. No execution comparison, deterministic guarantee or provider coverage census.
07 — Next stepState exactly what your replay established #
State exactly what your replay established
A replay is useful when its evidence matches its claim. Preserve the starting package, distinguish saved responses from live work, and compare against an agreed outcome. If an important condition changed, disclose it and narrow the conclusion rather than claiming an exact reproduction.