An AI evaluation improves after a prompt edit. A week later, a teammate cannot reproduce the comparison. The prompt file is available, but the retrieved note changed, a model alias may point somewhere else, and nobody recorded the adapter revision. There is an answer on disk without enough context to explain its origin.
Create a run manifest that identifies the actual request inputs, configuration, source snapshot, and evaluator revision. Then assign that manifest a stable digest. The digest helps detect changes to the recorded configuration; it does not guarantee identical model output.
A run manifest is a structured record describing an evaluation attempt. Its useful purpose is traceability: someone should be able to tell which conditions were held constant and which changed before interpreting an apparent improvement.
A template filename does not identify the messages actually sent. Variables may have been substituted, earlier messages retained, retrieved passages reordered, or a tool schema changed. Those differences can exist while the filename remains unchanged.
Record the resolved model identifier where the provider exposes one, the provider or endpoint configuration, generation settings, rendered messages, ordered context, tool definitions, adapter revision, and evaluation rubric revision. Use immutable references or protected artifacts for large inputs.
Keep event metadata separate from configuration identity. Two attempts can use the same configuration but have different run IDs, timestamps, outputs, and durations. If a timestamp enters the configuration digest, every retry appears to be a configuration change.
The fixture below uses synthetic model names. They are labels for the demonstration, not identifiers for a real model or claims about a provider's versioning behavior.
Python's json module supports sorted object keys and explicit separators. hashlib provides SHA-256. Combining these can give the same accepted data structure a stable byte representation under a stated convention.
This example permits strings, integers, booleans, nulls, lists, and dictionaries with string keys. Decimal settings are represented as strings. It deliberately rejects floating-point values, tuples, and other objects, keeping the accepted schema small and its representation easier to explain.
Save the following as ai_run_manifest.py and run it with Python 3.12:
import copy
import hashlib
import json
def validate_json_value(value):
if value is None or type(value) in (str, int, bool):
return
if type(value) is list:
for item in value:
validate_json_value(item)
return
if type(value) is dict and all(type(key) is str for key in value):
for item in value.values():
validate_json_value(item)
return
raise TypeError("use string keys and JSON values; encode decimals as strings")
def stable_bytes(value):
validate_json_value(value)
return json.dumps(
value, sort_keys=True, ensure_ascii=False, separators=(",", ":")
).encode("utf-8")
def digest_bytes(value):
return hashlib.sha256(value).hexdigest()
def manifest_id(manifest):
return digest_bytes(stable_bytes(manifest))
if __name__ == "__main__":
rendered_messages = [
{"role": "system", "content": "Summarize supplied notes. Preserve limits."},
{"role": "user", "content": "Summarize the evaluation note."},
]
source_bytes = b"Routine prompts passed; unusual prompts were not tested."
base = {
"schema": "ai-run-manifest-v1",
"adapter_revision": "demo-adapter-v1",
"provider": "synthetic",
"model": "demo-model-v1",
"settings": {"temperature": "0", "max_output_tokens": 300},
"messages_sha256": digest_bytes(stable_bytes(rendered_messages)),
"tools_sha256": digest_bytes(stable_bytes([])),
"context": [
{
"id": "note-1",
"revision": "v1",
"sha256": digest_bytes(source_bytes),
}
],
"evaluator_revision": "human-rubric-v1",
}
reordered = dict(reversed(list(base.items())))
changed = copy.deepcopy(base)
changed["context"][0]["revision"] = "v2"
print(
"same fields, different key order:",
manifest_id(base) == manifest_id(reordered),
)
print(
"changed source revision:",
manifest_id(base) == manifest_id(changed),
)
print("configuration ID:", manifest_id(base))
messages_sha256 fingerprints the rendered synthetic messages. The context list retains source order, ID, revision, and a digest of the source bytes. tools_sha256 identifies the empty tool list used in this fixture. A real adapter must supply the actual artifacts used by the request.
The Ranknod example uses invented configuration values so the identity rules can be inspected without exposing private prompts or implying a deployed integration.
Run:
python3 ai_run_manifest.py
The observed output begins:
same fields, different key order: True
changed source revision: False
The full configuration ID produced by this exact fixture was:
58e0a9173837863e61accca952edb3e883dc203b292f9ac1e07f231566972370
Reordering dictionary keys leaves the ID unchanged. Updating the recorded source revision changes it. Local checks also confirmed that changing list order changes the ID and that unsupported values, including floats and non-string dictionary keys, are rejected.
The source revision change in this demonstration leaves the content bytes unchanged. That is intentional: the convention treats a provenance revision as a configuration change even when the payload happens to be identical. Choose and document that behavior before comparing IDs across runs.
A matching digest means the recorded manifest serializes to the same identified configuration, subject to the properties of the hash. It cannot prove that the adapter accurately recorded every input. It cannot recover a source that was deleted, reveal a provider-side change hidden behind an alias, or explain nondeterministic behavior inside a model service.
Keep the actual request artifacts in an approved store when retention is permitted. A hash without recoverable context gives you a comparison signal, not a replay. Store returned provider metadata with the individual run where available; do not invent a backend version that the service never supplied.
This format is also not a general cross-language canonical JSON standard. Another runtime may serialize accepted values differently. Unicode normalization is not performed, and list order remains significant. If multiple languages produce manifests, adopt a documented common serialization scheme and test shared fixtures.
No. A digest is not encryption, anonymization, or a permission boundary. Someone who can guess a short input may be able to hash candidates and compare them. Treat manifests and their source artifacts according to the sensitivity of the workflow.
Never place API keys, bearer tokens, or credentials in the manifest. If endpoint or account information is needed for traceability, use an approved identifier. Apply retention and access controls to outputs as well as inputs; generated text can reproduce sensitive source material.
When an evaluation score changes, compare the manifests before explaining the change. If messages, context, model configuration, and evaluator all changed, describe the result as a comparison of workflows. It does not isolate the effect of one prompt sentence.
For a controlled prompt experiment, hold the other recorded conditions constant where possible and keep multiple output attempts when variation matters. Save the result alongside its run ID, configuration ID, and evaluator result. A failed run belongs in the record too.
Traceability does not make an AI result correct. It makes an investigation possible. The next time someone asks why an answer changed, the team can begin with the conditions that actually changed instead of reconstructing a test from a filename and a memory.