{"slug": "track-ai-prompt-changes-with-a-run-manifest", "title": "Track AI Prompt Changes with a Run Manifest", "summary": "A developer published a Python recipe for creating a run manifest that records the actual inputs, configuration, source snapshot, and evaluator revision behind an AI evaluation, then hashes it with SHA-256 for a stable digest. The approach uses Python's json module with sorted keys and explicit separators to produce a deterministic byte representation, rejecting floats, tuples, and non-string keys, and keeps event metadata like timestamps and run IDs out of the configuration digest so retries aren't mistaken for config changes.", "body_md": "An AI evaluation improves after a prompt edit. A week later, a teammate cannot reproduce the comparison. The prompt file is available, but the retrieved note changed, a model alias may point somewhere else, and nobody recorded the adapter revision. There is an answer on disk without enough context to explain its origin.\n\n**Create a run manifest that identifies the actual request inputs, configuration, source snapshot, and evaluator revision.** Then assign that manifest a stable digest. The digest helps detect changes to the recorded configuration; it does not guarantee identical model output.\n\nA run manifest is a structured record describing an evaluation attempt. Its useful purpose is traceability: someone should be able to tell which conditions were held constant and which changed before interpreting an apparent improvement.\n\nA template filename does not identify the messages actually sent. Variables may have been substituted, earlier messages retained, retrieved passages reordered, or a tool schema changed. Those differences can exist while the filename remains unchanged.\n\nRecord the resolved model identifier where the provider exposes one, the provider or endpoint configuration, generation settings, rendered messages, ordered context, tool definitions, adapter revision, and evaluation rubric revision. Use immutable references or protected artifacts for large inputs.\n\nKeep event metadata separate from configuration identity. Two attempts can use the same configuration but have different run IDs, timestamps, outputs, and durations. If a timestamp enters the configuration digest, every retry appears to be a configuration change.\n\nThe fixture below uses synthetic model names. They are labels for the demonstration, not identifiers for a real model or claims about a provider's versioning behavior.\n\nPython's [`json` module](https://docs.python.org/3.12/library/json.html) supports sorted object keys and explicit separators. [`hashlib`](https://docs.python.org/3.12/library/hashlib.html) provides SHA-256. Combining these can give the same accepted data structure a stable byte representation under a stated convention.\n\nThis example permits strings, integers, booleans, nulls, lists, and dictionaries with string keys. Decimal settings are represented as strings. It deliberately rejects floating-point values, tuples, and other objects, keeping the accepted schema small and its representation easier to explain.\n\nSave the following as `ai_run_manifest.py` and run it with Python 3.12:\n\n``` python\nimport copy\nimport hashlib\nimport json\n\ndef validate_json_value(value):\n    if value is None or type(value) in (str, int, bool):\n        return\n    if type(value) is list:\n        for item in value:\n            validate_json_value(item)\n        return\n    if type(value) is dict and all(type(key) is str for key in value):\n        for item in value.values():\n            validate_json_value(item)\n        return\n    raise TypeError(\"use string keys and JSON values; encode decimals as strings\")\n\ndef stable_bytes(value):\n    validate_json_value(value)\n    return json.dumps(\n        value, sort_keys=True, ensure_ascii=False, separators=(\",\", \":\")\n    ).encode(\"utf-8\")\n\ndef digest_bytes(value):\n    return hashlib.sha256(value).hexdigest()\n\ndef manifest_id(manifest):\n    return digest_bytes(stable_bytes(manifest))\n\nif __name__ == \"__main__\":\n    rendered_messages = [\n        {\"role\": \"system\", \"content\": \"Summarize supplied notes. Preserve limits.\"},\n        {\"role\": \"user\", \"content\": \"Summarize the evaluation note.\"},\n    ]\n    source_bytes = b\"Routine prompts passed; unusual prompts were not tested.\"\n    base = {\n        \"schema\": \"ai-run-manifest-v1\",\n        \"adapter_revision\": \"demo-adapter-v1\",\n        \"provider\": \"synthetic\",\n        \"model\": \"demo-model-v1\",\n        \"settings\": {\"temperature\": \"0\", \"max_output_tokens\": 300},\n        \"messages_sha256\": digest_bytes(stable_bytes(rendered_messages)),\n        \"tools_sha256\": digest_bytes(stable_bytes([])),\n        \"context\": [\n            {\n                \"id\": \"note-1\",\n                \"revision\": \"v1\",\n                \"sha256\": digest_bytes(source_bytes),\n            }\n        ],\n        \"evaluator_revision\": \"human-rubric-v1\",\n    }\n    reordered = dict(reversed(list(base.items())))\n    changed = copy.deepcopy(base)\n    changed[\"context\"][0][\"revision\"] = \"v2\"\n    print(\n        \"same fields, different key order:\",\n        manifest_id(base) == manifest_id(reordered),\n    )\n    print(\n        \"changed source revision:\",\n        manifest_id(base) == manifest_id(changed),\n    )\n    print(\"configuration ID:\", manifest_id(base))\n```\n\n`messages_sha256` fingerprints the rendered synthetic messages. The context list retains source order, ID, revision, and a digest of the source bytes. `tools_sha256` identifies the empty tool list used in this fixture. A real adapter must supply the actual artifacts used by the request.\n\nThe Ranknod example uses invented configuration values so the identity rules can be inspected without exposing private prompts or implying a deployed integration.\n\nRun:\n\n```\npython3 ai_run_manifest.py\n```\n\nThe observed output begins:\n\n```\nsame fields, different key order: True\nchanged source revision: False\n```\n\nThe full configuration ID produced by this exact fixture was:\n\n```\n58e0a9173837863e61accca952edb3e883dc203b292f9ac1e07f231566972370\n```\n\nReordering dictionary keys leaves the ID unchanged. Updating the recorded source revision changes it. Local checks also confirmed that changing list order changes the ID and that unsupported values, including floats and non-string dictionary keys, are rejected.\n\nThe source revision change in this demonstration leaves the content bytes unchanged. That is intentional: the convention treats a provenance revision as a configuration change even when the payload happens to be identical. Choose and document that behavior before comparing IDs across runs.\n\nA matching digest means the recorded manifest serializes to the same identified configuration, subject to the properties of the hash. It cannot prove that the adapter accurately recorded every input. It cannot recover a source that was deleted, reveal a provider-side change hidden behind an alias, or explain nondeterministic behavior inside a model service.\n\nKeep the actual request artifacts in an approved store when retention is permitted. A hash without recoverable context gives you a comparison signal, not a replay. Store returned provider metadata with the individual run where available; do not invent a backend version that the service never supplied.\n\nThis format is also not a general cross-language canonical JSON standard. Another runtime may serialize accepted values differently. Unicode normalization is not performed, and list order remains significant. If multiple languages produce manifests, adopt a documented common serialization scheme and test shared fixtures.\n\nNo. A digest is not encryption, anonymization, or a permission boundary. Someone who can guess a short input may be able to hash candidates and compare them. Treat manifests and their source artifacts according to the sensitivity of the workflow.\n\nNever place API keys, bearer tokens, or credentials in the manifest. If endpoint or account information is needed for traceability, use an approved identifier. Apply retention and access controls to outputs as well as inputs; generated text can reproduce sensitive source material.\n\nWhen an evaluation score changes, compare the manifests before explaining the change. If messages, context, model configuration, and evaluator all changed, describe the result as a comparison of workflows. It does not isolate the effect of one prompt sentence.\n\nFor a controlled prompt experiment, hold the other recorded conditions constant where possible and keep multiple output attempts when variation matters. Save the result alongside its run ID, configuration ID, and evaluator result. A failed run belongs in the record too.\n\nTraceability does not make an AI result correct. It makes an investigation possible. The next time someone asks why an answer changed, the team can begin with the conditions that actually changed instead of reconstructing a test from a filename and a memory.", "url": "https://wpnews.pro/news/track-ai-prompt-changes-with-a-run-manifest", "canonical_source": "https://dev.to/ranknod/track-ai-prompt-changes-with-a-run-manifest-4jgc", "published_at": "2026-09-25 15:41:14+00:00", "updated_at": "2026-09-25 16:01:12.067462+00:00", "lang": "en", "topics": ["mlops", "ai-tools", "developer-tools", "large-language-models"], "entities": ["Python", "hashlib", "json"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/track-ai-prompt-changes-with-a-run-manifest", "markdown": "https://wpnews.pro/news/track-ai-prompt-changes-with-a-run-manifest.md", "text": "https://wpnews.pro/news/track-ai-prompt-changes-with-a-run-manifest.txt", "jsonld": "https://wpnews.pro/news/track-ai-prompt-changes-with-a-run-manifest.jsonld"}}