{"slug": "before-you-trust-that-ai-diff-replay-real-requests-against-it", "title": "Before You Trust That AI Diff, Replay Real Requests Against It", "summary": "A developer shared a technique for validating AI-suggested code changes by replaying real traffic against a shadow service and comparing responses, arguing this catches runtime regressions that unit tests and model reviews miss. The approach uses a JSONL request log to send identical requests to baseline and candidate services, then reports mismatches, with a provided Python harness that skips write methods and normalizes unstable fields.", "body_md": "The hardest regressions from AI-suggested changes are the ones every unit test green-lighted. A patch can pass compile, pass the function suite, and still break on the sixth request in a burst because it assumes the `items`\n\narray is never empty. Static review catches style and obvious logic errors. It does not see the request pattern your service actually receives on a Wednesday morning.\n\nA common next step is to ask another model to review the diff. That gives you an opinion, usually with confident wording. What it doesn't give you is evidence about runtime behavior. This article describes a different step: deploy the candidate change to a shadow service, replay a day of real traffic against it, and compare what comes back.\n\nWhen two implementations answer the same request, you can compare status codes and bodies directly. That comparison is reproducible. Two runs over the same log should produce the same mismatch list, except for non-deterministic features you deliberately normalize. A model-generated review is harder to reproduce because temperature, prompt wording, and recent context change the output.\n\nA shadow service fits a disposable server well. You do not need it to be production-grade; you need it to run the candidate code with enough of the same dependencies to respond realistically. The cheaper that server is to throw away, the more often you can run the comparison. That is where a free server option, if you have one, changes the frequency of verification rather than the final verdict.\n\nThe harness below reads a JSONL request log, sends each request to both a baseline service and a candidate service, then prints mismatches. It skips write methods by default because replaying writes safely requires a shadow database and careful cleanup.\n\n``` python\nimport json\nimport sys\nfrom urllib.request import Request, urlopen\nfrom urllib.error import HTTPError\n\nWRITE_METHODS = {\"POST\", \"PUT\", \"PATCH\", \"DELETE\"}\nSENSITIVE_HEADERS = {\"authorization\", \"cookie\", \"x-api-key\"}\n\ndef load_requests(path):\n    with open(path, encoding=\"utf-8\") as f:\n        for line in f:\n            if line.strip():\n                yield json.loads(line)\n\ndef safe_headers(headers):\n    return {\n        k: v for k, v in headers.items()\n        if k.lower() not in SENSITIVE_HEADERS\n    }\n\ndef call(base_url, req, timeout=5):\n    url = base_url + req[\"path\"]\n    body = req.get(\"body\")\n    data = None if body is None else json.dumps(body).encode()\n    request = Request(\n        url,\n        data=data,\n        headers=safe_headers(req.get(\"headers\", {})),\n        method=req.get(\"method\", \"GET\"),\n    )\n    try:\n        with urlopen(request, timeout=timeout) as resp:\n            payload = resp.read().decode()\n            try:\n                payload = json.loads(payload)\n            except json.JSONDecodeError:\n                pass\n            return resp.status, payload\n    except HTTPError as e:\n        payload = e.read().decode()\n        try:\n            payload = json.loads(payload)\n        except json.JSONDecodeError:\n            pass\n        return e.code, payload\n\ndef normalize(value, unstable_keys):\n    \"\"\"Replace values at given dotted paths with a sentinel.\"\"\"\n    # A production version should use JSONPath or a recursive walker.\n    if isinstance(value, dict):\n        return {k: normalize(v, unstable_keys) for k, v in value.items()}\n    if isinstance(value, list):\n        return [normalize(v, unstable_keys) for v in value]\n    return value\n\ndef main():\n    baseline = sys.argv[1]\n    candidate = sys.argv[2]\n    request_log = sys.argv[3]\n    mismatches = []\n\n    for req in load_requests(request_log):\n        if req.get(\"method\") in WRITE_METHODS:\n            continue\n        status_a, body_a = call(baseline, req)\n        status_b, body_b = call(candidate, req)\n\n        # Shallow compare for the article; add canonicalization in practice.\n        if status_a != status_b or body_a != body_b:\n            mismatches.append({\n                \"path\": req[\"path\"],\n                \"method\": req.get(\"method\", \"GET\"),\n                \"baseline_status\": status_a,\n                \"candidate_status\": status_b,\n                \"baseline_body\": body_a,\n                \"candidate_body\": body_b,\n            })\n\n    print(f\"{len(mismatches)} mismatches\")\n    for item in mismatches[:20]:\n        print(json.dumps(item, indent=2))\n\nif __name__ == \"__main__\":\n    main()\n```\n\nThis is intentionally simple. Before using it, replace the shallow body comparison with a canonicalizer that strips volatile fields such as IDs, timestamps, trace IDs, and generated URLs. Otherwise, the mismatch list will be noisy enough to ignore.\n\nNot every difference deserves the same response. A useful decision table looks like this:\n\n| Difference type | Action |\n|---|---|\n| Status code differs | Investigate immediately |\n| Body differs only in normalized volatile fields | Treat as a match |\n| Body differs in a known date field format | Fix the candidate serializer |\n| Same status, unexpected body difference | Review the specific request and response |\n| Timeout or connection error on candidate | Profile the changed code path |\n\nAfter the deterministic compare runs, a model can help group the mismatches into a shorter report—for example, \"12 mismatches are timestamp-only, 3 are empty-array handling, and 1 is a 500 on missing locale.\" That reduces reading time without making the model the approver. If MonkeyCode's free model access and free server option are available to you, they map to those two resource slots: the server hosts the shadow instance, and the model summarizes the deterministic mismatch list. Disclosure: This article was prepared as part of MonkeyCode's product outreach.\n\nReplay only covers the requests in your log. It misses cold-start latency, enormous payloads you have never seen, and the first request after a cache flush. If your candidate introduces a new dependency, the shadow server may fail differently from production because the dependency environment is not identical. That does not make the test useless; it means you should treat a clean replay as a reduction in risk, not proof of correctness.\n\nWrite operations are the biggest gap. Running a candidate against the same database as production is dangerous. If you need to verify writes, start with a restored read replica or a dedicated shadow database, and make sure the request log cannot contain credentials or personal data. Redact before storing the log, not after.\n\nPick an endpoint that receives real traffic and has a stable response shape. Capture one day of requests, strip sensitive headers, and run the harness on the smallest disposable server you can find. If the mismatch list is zero after canonicalization, you have evidence the candidate behaves like today's service for that traffic. If it is not zero, you have a list of concrete requests to debug—far better than arguing with a model about whether a diff is safe.\n\nDo not skip the baseline. Comparing candidate against what you think should happen is weaker than comparing it against what currently happens. Real traffic is messy, and the baseline already encodes that mess. Use it.", "url": "https://wpnews.pro/news/before-you-trust-that-ai-diff-replay-real-requests-against-it", "canonical_source": "https://dev.to/codepy_1473/before-you-trust-that-ai-diff-replay-real-requests-against-it-2b27", "published_at": "2026-08-14 01:05:03+00:00", "updated_at": "2026-08-14 01:45:20.017143+00:00", "lang": "en", "topics": ["developer-tools", "ai-tools", "mlops"], "entities": [], "alternates": {"html": "https://wpnews.pro/news/before-you-trust-that-ai-diff-replay-real-requests-against-it", "markdown": "https://wpnews.pro/news/before-you-trust-that-ai-diff-replay-real-requests-against-it.md", "text": "https://wpnews.pro/news/before-you-trust-that-ai-diff-replay-real-requests-against-it.txt", "jsonld": "https://wpnews.pro/news/before-you-trust-that-ai-diff-replay-real-requests-against-it.jsonld"}}