{"slug": "why-code-diffs-are-not-enough-for-ai-agent-changes", "title": "Why Code Diffs Are Not Enough for AI Agent Changes", "summary": "AgentInspect, an open-source TypeScript toolkit for inspecting agent runs, introduces a run-diff workflow that compares execution traces to source diffs. The toolkit's CLI can identify behavioral changes such as added or removed steps, duration shifts, and first divergence points, providing a more accurate risk assessment for AI agent modifications.", "body_md": "A pull request changes three lines in a prompt. The source diff is tiny. The resulting agent run adds a tool call, skips a planning step, changes its recovery path, and takes twice as long.\n\nWhich diff describes the risk more accurately?\n\nBoth do—but they describe different things.\n\nI maintain [AgentInspect](https://github.com/rajudandigam/agent-inspect), an open-source TypeScript toolkit for inspecting agent runs locally. I designed its run-diff workflow around a simple idea: code review tells us what the developer changed; execution evidence tells us what the agent did differently. The examples below use synthetic fixtures verified against [`agent-inspect@6.17.4`](https://github.com/rajudandigam/agent-inspect/tree/agent-inspect%406.17.4).\n\nIn ordinary deterministic code, a source diff is often a strong predictor of runtime change. Agent systems add several moving parts:\n\nA large refactor can preserve the same trajectory. A one-word prompt change can alter tool selection. Identical code can behave differently when a model or external result changes.\n\nThat does not make code review obsolete. It means the review unit needs another layer:\n\n```\nsource diff                    behavior diff\n-----------                    -------------\nwhat was edited?               what path changed?\nwhat code owns the change?     where did runs diverge?\nis the implementation sound?   which steps/errors/outputs differ?\n```\n\nA useful behavioral comparison needs a deliberate baseline and candidate:\n\n```\nbaseline\n  same synthetic input\n  pinned or recorded configuration\n  known-good trace\n\ncandidate\n  same synthetic input\n  intended code/prompt/model change\n  newly captured trace\n```\n\nControl what you can. Record what you cannot. If the input, retrieval corpus, model, and tool fixtures all change at once, the diff may be accurate but difficult to interpret.\n\nWith two local run IDs, compare them from the CLI:\n\n```\nnpx agent-inspect diff minimal-success minimal-error \\\n  --dir .agent-inspect\n```\n\nThe synthetic fixture output begins with the summary and first divergence:\n\n```\nRun diff\nLeft:  minimal-success\nRight: minimal-error\n\nSummary:\n  Differences: 4\n  Errors: 0\n  Warnings: 3\n  Info: 1\n\nFirst divergence:\n  run-status at (run)\n    left: success\n    right: error\n```\n\nThat “first divergence” is often more actionable than a long list of event differences. It gives the reviewer a starting point for the causal investigation.\n\nThe same fixture reports:\n\n```\nDifferences:\n  [warning] run-status\n    Run completion status differs\n    left: success\n    right: error\n  [info] duration\n    Run duration differs\n    left: 120\n    right: 70\n  [warning] step-removed plan\n    Step only in left run: plan\n    left: step_root\n    right: (undefined)\n  [warning] step-added failing-step\n    Step only in right run: failing-step\n    left: (undefined)\n    right: step_fail\n```\n\nThe evidence says that `plan` appears only in the left run and `failing-step` only in the right. It does not automatically say why.\n\nPossible explanations include:\n\nThis is why a behavioral diff is an input to review, not an automatic verdict. Pair it with the source diff and inspect the execution tree around the divergence.\n\nThe CLI can limit the comparison to a specific check dimension. To inspect only structure:\n\n```\nnpx agent-inspect diff minimal-success minimal-error \\\n  --dir .agent-inspect \\\n  --check structure\n```\n\nThat removes status and duration noise and leaves the added/removed steps. For a performance-oriented review:\n\n```\nnpx agent-inspect diff minimal-success minimal-error \\\n  --dir .agent-inspect \\\n  --check timing \\\n  --duration-threshold 20ms\n```\n\nThe command also supports JSON output for automation, `--ignore-duration`, focus modes, and verbose output. A practical rule is to begin with the broad human-readable diff, then narrow the view when you know which hypothesis you are testing.\n\nThe fixture reports `120` versus `70` milliseconds. It would be a mistake to generalize that single synthetic delta into a performance claim.\n\nAgent latency can vary with network conditions, cache state, provider load, token volume, and concurrency. A timing diff is most useful when:\n\nUse `--duration-threshold` to suppress insignificant changes, but derive the threshold from your environment. Do not choose a number merely because it makes a current test pass.\n\nAgentInspect’s diff is a read-only comparison of persisted traces. It does not replay either agent, invoke a model, or prove that the difference will recur.\n\nThat property is useful for review: the comparison is deterministic for the two stored artifacts. It is also a limitation: representative capture remains your responsibility.\n\nI think of the workflow as three separate actions:\n\n``` php\nexecute -> capture evidence\ncompare -> describe observed differences\njudge   -> decide whether the change is acceptable\n```\n\nOnly the middle action is the run-diff engine.\n\nFor changes that can materially affect an agent path, a compact pull-request section can make review faster:\n\n```\n## Agent behavior evidence\n\n- Fixture: `refund-eligible-order`\n- Baseline run: `refund-before`\n- Candidate run: `refund-after`\n- Expected change: prefer cached policy when current\n- First divergence: `retrieve-policy` replaced by `load-policy-cache`\n- Contract result: pass\n- Evidence artifact: attached CI bundle\n- Reviewer note: no production or customer data used\n```\n\nThis is intentionally concise. The full trace should remain an artifact, not be pasted into the pull-request description.\n\nThe most important field is “expected change.” It tells reviewers whether an observed divergence is intentional. Without that statement, the diff is merely a list of facts.\n\nA diff answers “What changed?” A contract answers “Did a declared invariant still hold?” Use both.\n\nSuppose a candidate run replaces remote retrieval with a cache hit. The structural diff should show the path change. A contract might still require:\n\nThe candidate can therefore differ from the baseline and still pass the invariant gate. This is healthier than either extreme:\n\nChoose fixtures around decisions, not around random traffic. High-value comparisons include:\n\nDid tool selection, ordering, retry behavior, or token usage change?\n\nDoes the new model reach the same goal with a different trajectory? Does it invoke a fallback more often in controlled cases?\n\nDid a renamed or re-described tool disappear, get replaced, or start failing?\n\nDid parent-child structure, concurrency, or error propagation change even if final answers stayed stable?\n\nDid the run add or skip retrieval, or generate before the required evidence step?\n\nFor each case, pair the behavioral observation with a domain-specific quality check. A shorter path is not necessarily a better answer.\n\nCode diffs remain the foundation of software review. Agent systems need an additional artifact because runtime behavior depends on more than source text.\n\nA disciplined workflow is straightforward:\n\nThis does not eliminate nondeterminism. It gives reviewers something more precise than “I tried the prompt and it looked better.”\n\nThe tagged release and fixtures used for this article are available on [GitHub](https://github.com/rajudandigam/agent-inspect/tree/agent-inspect%406.17.4). If you adopt behavior diffs in code review, begin with one high-risk fixture and learn which differences your team actually finds actionable.", "url": "https://wpnews.pro/news/why-code-diffs-are-not-enough-for-ai-agent-changes", "canonical_source": "https://dev.to/raju_dandigam/why-code-diffs-are-not-enough-for-ai-agent-changes-3fhn", "published_at": "2026-09-07 17:28:52+00:00", "updated_at": "2026-09-07 18:02:09.541902+00:00", "lang": "en", "topics": ["developer-tools", "ai-agents"], "entities": ["AgentInspect", "rajudandigam"], "alternates": {"html": "https://wpnews.pro/news/why-code-diffs-are-not-enough-for-ai-agent-changes", "markdown": "https://wpnews.pro/news/why-code-diffs-are-not-enough-for-ai-agent-changes.md", "text": "https://wpnews.pro/news/why-code-diffs-are-not-enough-for-ai-agent-changes.txt", "jsonld": "https://wpnews.pro/news/why-code-diffs-are-not-enough-for-ai-agent-changes.jsonld"}}