Before You Trust That AI Diff, Replay Real Requests Against It

A developer shared a technique for validating AI-suggested code changes by replaying real traffic against a shadow service and comparing responses, arguing this catches runtime regressions that unit tests and model reviews miss. The approach uses a JSONL request log to send identical requests to baseline and candidate services, then reports mismatches, with a provided Python harness that skips write methods and normalizes unstable fields.

The hardest regressions from AI-suggested changes are the ones every unit test green-lighted. A patch can pass compile, pass the function suite, and still break on the sixth request in a burst because it assumes the items array is never empty. Static review catches style and obvious logic errors. It does not see the request pattern your service actually receives on a Wednesday morning. A common next step is to ask another model to review the diff. That gives you an opinion, usually with confident wording. What it doesn't give you is evidence about runtime behavior. This article describes a different step: deploy the candidate change to a shadow service, replay a day of real traffic against it, and compare what comes back. When two implementations answer the same request, you can compare status codes and bodies directly. That comparison is reproducible. Two runs over the same log should produce the same mismatch list, except for non-deterministic features you deliberately normalize. A model-generated review is harder to reproduce because temperature, prompt wording, and recent context change the output. A shadow service fits a disposable server well. You do not need it to be production-grade; you need it to run the candidate code with enough of the same dependencies to respond realistically. The cheaper that server is to throw away, the more often you can run the comparison. That is where a free server option, if you have one, changes the frequency of verification rather than the final verdict. The harness below reads a JSONL request log, sends each request to both a baseline service and a candidate service, then prints mismatches. It skips write methods by default because replaying writes safely requires a shadow database and careful cleanup. python import json import sys from urllib.request import Request, urlopen from urllib.error import HTTPError WRITE METHODS = {"POST", "PUT", "PATCH", "DELETE"} SENSITIVE HEADERS = {"authorization", "cookie", "x-api-key"} def load requests path : with open path, encoding="utf-8" as f: for line in f: if line.strip : yield json.loads line def safe headers headers : return { k: v for k, v in headers.items if k.lower not in SENSITIVE HEADERS } def call base url, req, timeout=5 : url = base url + req "path" body = req.get "body" data = None if body is None else json.dumps body .encode request = Request url, data=data, headers=safe headers req.get "headers", {} , method=req.get "method", "GET" , try: with urlopen request, timeout=timeout as resp: payload = resp.read .decode try: payload = json.loads payload except json.JSONDecodeError: pass return resp.status, payload except HTTPError as e: payload = e.read .decode try: payload = json.loads payload except json.JSONDecodeError: pass return e.code, payload def normalize value, unstable keys : """Replace values at given dotted paths with a sentinel.""" A production version should use JSONPath or a recursive walker. if isinstance value, dict : return {k: normalize v, unstable keys for k, v in value.items } if isinstance value, list : return normalize v, unstable keys for v in value return value def main : baseline = sys.argv 1 candidate = sys.argv 2 request log = sys.argv 3 mismatches = for req in load requests request log : if req.get "method" in WRITE METHODS: continue status a, body a = call baseline, req status b, body b = call candidate, req Shallow compare for the article; add canonicalization in practice. if status a = status b or body a = body b: mismatches.append { "path": req "path" , "method": req.get "method", "GET" , "baseline status": status a, "candidate status": status b, "baseline body": body a, "candidate body": body b, } print f"{len mismatches } mismatches" for item in mismatches :20 : print json.dumps item, indent=2 if name == " main ": main This is intentionally simple. Before using it, replace the shallow body comparison with a canonicalizer that strips volatile fields such as IDs, timestamps, trace IDs, and generated URLs. Otherwise, the mismatch list will be noisy enough to ignore. Not every difference deserves the same response. A useful decision table looks like this: | Difference type | Action | |---|---| | Status code differs | Investigate immediately | | Body differs only in normalized volatile fields | Treat as a match | | Body differs in a known date field format | Fix the candidate serializer | | Same status, unexpected body difference | Review the specific request and response | | Timeout or connection error on candidate | Profile the changed code path | After the deterministic compare runs, a model can help group the mismatches into a shorter report—for example, "12 mismatches are timestamp-only, 3 are empty-array handling, and 1 is a 500 on missing locale." That reduces reading time without making the model the approver. If MonkeyCode's free model access and free server option are available to you, they map to those two resource slots: the server hosts the shadow instance, and the model summarizes the deterministic mismatch list. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Replay only covers the requests in your log. It misses cold-start latency, enormous payloads you have never seen, and the first request after a cache flush. If your candidate introduces a new dependency, the shadow server may fail differently from production because the dependency environment is not identical. That does not make the test useless; it means you should treat a clean replay as a reduction in risk, not proof of correctness. Write operations are the biggest gap. Running a candidate against the same database as production is dangerous. If you need to verify writes, start with a restored read replica or a dedicated shadow database, and make sure the request log cannot contain credentials or personal data. Redact before storing the log, not after. Pick an endpoint that receives real traffic and has a stable response shape. Capture one day of requests, strip sensitive headers, and run the harness on the smallest disposable server you can find. If the mismatch list is zero after canonicalization, you have evidence the candidate behaves like today's service for that traffic. If it is not zero, you have a list of concrete requests to debug—far better than arguing with a model about whether a diff is safe. Do not skip the baseline. Comparing candidate against what you think should happen is weaker than comparing it against what currently happens. Real traffic is messy, and the baseline already encodes that mess. Use it.