{"slug": "can-you-trust-code-you-didn-t-write", "title": "Can You Trust Code You Didn't Write?", "summary": "Developer Tony describes a verification workflow for code produced by AI coding agents such as Claude Code, arguing that an agent reporting \"implementation complete\" or \"all tests passed\" does not establish that the requested behavior actually works. His approach pairs a pre-implementation acceptance contract of observable behaviors with focused authorization tests and a recorded evidence trail, illustrated on a multi-tenant TypeScript API with a group-based permission bug.", "body_md": "I'm Tony. I build and run web products, and Claude Code is my main development tool. A coding agent finishes implementing a feature, reports that everything is complete, and presents a summary of the changes. The code looks reasonable, the tests appear green, and you have another task waiting. But what exactly did the agent prove?\n\nGenerating working code and verifying correct behavior are different engineering activities. I use a verification workflow that starts with observable acceptance criteria, checks the implementation at several levels, and records evidence before accepting a change. I'll walk through it using a TypeScript application with an admin interface and a multi-tenant API, including an authorization bug that ordinary tests can miss. You'll leave with a reusable verification checklist and a completed example showing what an acceptable evidence trail looks like.\n\nA coding agent can produce a convincing implementation without establishing that it solves the original problem. It might modify the right files, follow the existing patterns, and run the available tests successfully. None of those results automatically establishes that the requested behavior works. They answer narrower questions.\n\nConsider four statements an agent might make:\n\n| Statement | What it actually establishes | \n|---|---|\n| \"Implementation complete\" | The agent produced code it considers finished. | \n| \"TypeScript compilation passed\" | The compiler found no errors covered by that check. | \n| \"All tests passed\" | The executed tests passed under their configured conditions. | \n| \"The requested behavior works\" | Requires evidence that the implementation satisfies the acceptance criteria. | \n\nThe difference matters because tests only exercise the scenarios someone anticipated. A passing authorization test might establish that a resource owner can access a document, while saying nothing about another authorized member or an unrelated tenant. A successful browser interaction might show text appearing in an editor without proving that the correct value was saved. Even a valid download can contain files with incorrect permissions.\n\nI think of this as a verification gap. The agent's final message belongs on one side; evidence that supports the product requirements belongs on the other. My responsibility is to close that gap, not merely decide whether the generated code looks plausible.\n\nThe same distinction applies to human-written code, but agents change the amount of implementation a developer can produce without personally tracing every decision. That makes a disciplined acceptance process more important.\n\nBefore implementation, I want a short contract describing what the system must do and how we will know it works. This is not a complete specification or an elaborate test plan. It is a set of observable behaviors that can be checked independently of the agent's explanation.\n\nImagine a pnpm TypeScript monorepo with the following structure:\n\n```\napps/\n  admin/              # React admin application\n  api/                # Multi-tenant API\npackages/\n  ui/                 # Shared UI components\n  shared/             # Shared contracts and utilities\n```\n\nSuppose the API serves multiple customer organizations. A resource can be associated with several groups, and users receive access through group membership. The requested change is to correct a permission check that incorrectly rejects some authorized users.\n\nHere is the acceptance contract I would give the agent:\n\n```\nTask:\nCorrect group-based access to shared resources.\n\nExpected behavior:\n1. A member of any group attached to a resource\n   can access that resource within the same tenant.\n2. A user outside all attached groups is denied.\n3. A user from another tenant is denied, even when\n   group identifiers overlap.\n4. Existing resource-owner access remains unchanged.\n\nEvidence required:\n- Focused authorization tests for all four cases.\n- A regression test where the authorized user\n  belongs to the second attached group.\n- Confirmation that the regression test fails\n  without the fix and passes with it.\n- A summary of changed files and remaining risks.\n\nConstraints:\n- Preserve tenant isolation.\n- Do not change unrelated authorization behavior.\n- Do not modify tests merely to make them pass.\n```\n\nThis contract specifies results rather than implementation details. I have not instructed the agent to use a particular SQL query, ORM method, or function structure. It can choose an approach that fits the existing code, but it cannot redefine successful completion as \"the code compiles.\"\n\nThe edge cases carry much of the value. Testing an owner and an unauthorized stranger is insufficient when access depends on several relationships. The important case is a user who belongs to a valid group that the previous implementation failed to consider.\n\nFor a bug fix, I also provide the reproduction command, relevant stack trace when one exists, and steps needed to reproduce the behavior. I state whether the failure is intermittent. These details reduce guesswork and give the agent a concrete starting point.\n\nThe principle is simple: define success before evaluating the implementation. Otherwise, the agent's solution can influence what everyone considers an acceptable test.\n\nI separate implementation instructions from acceptance decisions. The agent can inspect the repository, propose a solution, edit code, and execute tests. It can also identify missing cases and explain technical tradeoffs. But I do not treat its final assessment as independent evidence.\n\nFor changes with unclear scope, I start Claude Code in plan mode:\n\n```\nclaude --permission-mode plan\n```\n\nPlan mode lets Claude read the project and propose an approach without editing files until approval. I use that boundary to examine assumptions, identify impacted modules, and check whether the proposed tests cover the acceptance contract. The plan is useful because it exposes decisions before implementation makes those decisions harder to notice.\n\nFor the authorization example, I might use this prompt:\n\n```\nInspect the shared-resource authorization flow.\n\nIdentify:\n- How resources are associated with groups.\n- Where tenant isolation is enforced.\n- Which tests cover users with multiple group paths.\n- Why membership in the second attached group fails.\n\nPropose the smallest safe change and the tests\nneeded to verify it. Do not edit files yet.\n```\n\nAfter reviewing the proposal, I authorize implementation and ask for a report containing the actual commands and outcomes. I also ask the agent to identify missed edge cases. This makes the agent useful as an implementation partner without making it the final authority on correctness.\n\nAn instruction in CLAUDE.md can require the agent to include test evidence in its response. That is a useful working convention, not an enforcement mechanism. The presence of a well-formatted report does not prove that the underlying evidence is valid.\n\nIf the change moves in the wrong direction, Claude Code's `/rewind` can restore the conversation and code to a checkpoint. That helps manage the implementation process, but it does not replace version control or acceptance testing.\n\nVerification works better when each check has a specific responsibility. Compilation, automated tests, browser interactions, and human review observe different properties of the system. Running more checks is not automatically better; choosing checks that cover different failure modes is what matters.\n\nI start with the checks already defined by the repository. For a TypeScript project, these typically include type checking, linting, and build validation. They reveal broken imports, incompatible types, invalid code patterns, and some integration mistakes.\n\nFor example:\n\n```\npnpm -F api exec tsc --noEmit\n```\n\nThis command only makes sense when the workspace and TypeScript configuration support it. A successful result tells me that the relevant TypeScript checks passed. It does not establish that group membership is evaluated correctly or that the API returns the expected authorization decisions.\n\nI record which command ran, its exit status, and whether the check covers the changed code. \"Static checks passed\" is less informative than the exact command and result.\n\nConsider this incorrect permission check:\n\n``` js\nconst group = await db.resourceGroup.findFirst({\n  where: {\n    resourceId,\n  },\n});\n\nconst allowed = group\n  ? await isGroupMember(userId, group.groupId)\n  : false;\n```\n\nThe implementation checks membership against the first group returned by the database. If a resource belongs to groups A and B, a user belonging only to B can be denied access. The code compiles, and a test using a member of A might pass.\n\nThe correction should ask whether there is at least one qualifying membership, not whether the user belongs to one arbitrarily selected group. One possible approach is a relational query that keeps the tenant condition within the authorization predicate:\n\n``` js\nconst allowed = await db.resourceGroup.count({\n  where: {\n    resourceId,\n    resource: {\n      tenantId,\n    },\n    group: {\n      tenantId,\n      memberships: {\n        some: {\n          userId,\n        },\n      },\n    },\n  },\n}) > 0;\n```\n\nThe exact query depends on the application's data model and ORM schema. Its essential property is that it evaluates all eligible relationships while preserving tenant boundaries. The surrounding authorization flow must still validate the user's tenant context and any additional access rules.\n\nThe regression test is more valuable than a generic \"authorized user succeeds\" test:\n\n``` js\nit(\"allows a member of the second attached group\", async () => {\n  const resource = await createResource({\n    tenantId: tenantA.id,\n    groupIds: [groupA.id, groupB.id],\n  });\n\n  const user = await createUser({\n    tenantId: tenantA.id,\n    groupIds: [groupB.id],\n  });\n\n  const response = await requestResource({\n    resourceId: resource.id,\n    asUser: user,\n  });\n\n  expect(response.status).toBe(200);\n});\n```\n\nHere, `createResource`, `createUser`, and `requestResource` represent application-specific test helpers. A real implementation should use the repository's existing fixtures and request utilities, with deterministic setup that makes the relevant group ordering explicit.\n\nA passing test is useful. A test that fails against the broken implementation and passes against the correction provides stronger evidence that it detects the intended defect.\n\nOne way to perform this check in a controlled working tree is:\n\n```\ngit stash push src/auth/permissions.ts\nnpm test -- permissions\ngit stash pop\n```\n\nThis sequence assumes the working-tree state, stash contents, and test command have been inspected first. The stash must actually restore the known broken implementation for the test to mean anything. It can also stash other uncommitted edits in the named file, so I would use a clean, isolated working tree rather than run it casually against active work.\n\nNotice another operational detail: combining the test and restore operations with `;` allows the restore attempt to run even if the test fails. With three separate commands, I must explicitly execute the restoration after the expected failure and inspect the resulting working-tree state.\n\nThe evidence I want is specific: the regression test fails for the expected authorization reason without the fix, then passes with the correction restored. A failure caused by missing fixtures, invalid imports, or environment setup is not proof that the regression test detects the original bug.\n\nThe repository might use pnpm instead:\n\n```\npnpm -F api test -- permissions\n```\n\nI use the command that matches the actual project rather than changing package managers simply to match an example.\n\nAPI tests do not establish that the browser handles state correctly. A common case is automation entering text into a rich-text editor built with DraftJS-style or React-managed state. The text becomes visible, but the application saves a default value because the submitted content lives in parent component state.\n\nA screenshot of the editor would look correct while the saved record would be wrong. The appropriate fix is to update the value through the component's supported `onChange` mechanism, then save and read back the persisted content. The verification target is the reopened or fetched record, not merely the visible text before submission.\n\nFor the authorization example, a browser check can establish that an authorized member can open the shared resource and an unauthorized member receives the intended denial. That user-facing result complements the API tests without replacing them.\n\nI cover browser automation patterns in more detail in the article on browser testing with Playwright in this series.\n\nAutomated checks can establish consistency with written expectations. They cannot decide whether those expectations fully reflect the product's intended policy. A developer still needs to evaluate whether the change introduces unintended access, violates an architectural boundary, or relies on an assumption the tests never challenge.\n\nFor authorization code, I examine query scoping, user identity, tenant ownership, and any caching that might retain stale permissions. I also look for inconsistent behavior between endpoints using the same permission model. Those concerns are part of engineering review, not defects a green test suite automatically rules out.\n\nAfter executing the checks, I return to the original acceptance contract and inspect the actual diff. The question is not simply whether I understand the implementation. I want to know whether every change serves the requested behavior.\n\nFor the authorization fix, the intended scope is narrow: correct membership evaluation and add meaningful coverage. A change that also modifies authentication middleware, restructures tenant resolution, or removes failing assertions deserves separate justification. Scope expansion increases the number of assumptions I need to verify.\n\nMy diff review focuses on five questions:\n\nAnother useful example is generating a downloadable ZIP package. The download can succeed, contain the correct files, use the expected version, and pass a secrets check. Yet its setup script might lose the executable bit because the ZIP library does not preserve Unix file modes.\n\nAfter extraction, this result is wrong:\n\n```\n-rw-r--r-- scripts/setup.sh\n```\n\nRunning the script directly produces:\n\n```\n./scripts/setup.sh: Permission denied\n```\n\nFor a package where scripts are expected to run directly on Unix, a packaging rule can assign mode `0755` to files beginning with `#!` and `0644` to ordinary files. Verification should then inspect the extracted archive, not just the source directory.\n\nThe expected extracted permission is:\n\n```\n-rwxr-xr-x scripts/setup.sh\n```\n\nThis example demonstrates why a passing pipeline can still miss an important delivery requirement. The implementation artifact must be checked in the form and environment where someone actually uses it.\n\nI prefer a compact verification report over a long transcript of everything the agent attempted. The report should let another developer understand the requested behavior, identify the executed checks, reproduce important results, and see what remains uncertain. It should also distinguish the agent's statements from recorded observations.\n\nThe following is a completed example for our fictional authorization fix. The paths and outcomes illustrate what a report should contain; they are not claims that these commands were executed against a real repository.\n\n`agent-change-verification.md`\n\n```\n# Agent Change Verification\n\n## Task\nCorrect group-based access to shared resources.\n\n## Acceptance criteria\n- [x] Member of any attached group is allowed.\n- [x] Nonmember is denied.\n- [x] User from another tenant is denied.\n- [x] Existing owner access remains unchanged.\n\n## Files changed\n- src/auth/permissions.ts\n  Replaced first-group lookup with membership query.\n- src/auth/permissions.test.ts\n  Added second-group and tenant-isolation cases.\n\n## Checks executed\n\n| Check | Result | Evidence |\n|---|---|---|\n| TypeScript check | PASS | typecheck.log |\n| API permission tests | PASS | permissions-test.log |\n| Regression without fix | EXPECTED FAIL | regression-red.log |\n| Regression with fix | PASS | regression-green.log |\n| Focused diff review | PASS | review-notes.md |\n\n## Expected vs observed\nExpected:\nA user in group B can access a resource shared\nwith groups A and B within the same tenant.\n\nObserved:\nThe API returns 200 for the group B member.\nThe regression test fails with the original\nfirst-group logic and passes with the fix.\n\n## Unverified scenarios\n- Concurrent group membership changes.\n- Cache invalidation after permission updates.\n- Browser-level permission error presentation.\n\n## Known limitations\n- Tests use a local test database.\n- Production data distribution was not reproduced.\n- No performance evaluation of the new query.\n\n## Human review decision\nAPPROVED FOR REVIEW, NOT DEPLOYMENT.\n\nReason:\nAcceptance tests pass and regression behavior\nis demonstrated. Query execution characteristics\nand cache behavior require separate review.\n```\n\nThe report does not need to claim absolute confidence. Its value comes from making the evidence inspectable and showing where the evidence stops. In a real repository, the referenced logs and review notes should exist, correspond to the tested revision, and be accessible to the reviewer.\n\nI find it useful to rank evidence by what it establishes:\n\nThis is an ordering of evidence strength, not a replacement for evaluating test quality or requirement coverage. A read-back confirms a particular observed result; it does not establish that every untested condition is safe.\n\nFor write operations, I like tools that return both `verified` and `evidence`. The first indicates whether post-action verification succeeded. The second explains what the system returned when the tool read the result back.\n\n```\n{\n  \"operation\": \"deleteResource\",\n  \"verified\": true,\n  \"evidence\": {\n    \"method\": \"GET\",\n    \"path\": \"/resources/example-id\",\n    \"status\": 404\n  }\n}\n```\n\nIn this example, the write tool deleted a resource and then performed a separate read. A successful delete request alone would not provide equivalent evidence. The observed 404 supports the claim that the resource is no longer retrievable at that endpoint, assuming the read uses the appropriate identity and tenant context.\n\nFor pull requests, I can use `gh pr create` to connect a Claude Code session with a PR and reopen that session using `claude --from-pr <number>`. The PR then becomes a practical place to review the diff, attach test evidence, and ask Claude to flag risks. The review decision still belongs to the people responsible for accepting the change.\n\nA verification checklist helps make acceptance consistent, but it cannot compensate for missing requirements or an environment that fails to represent production behavior. Authorization, billing, data migration, destructive operations, and sensitive-data handling deserve closer attention because incorrect behavior can have consequences beyond a single feature.\n\nWeak test coverage also changes how much confidence I assign to a passing suite. If the application has no realistic integration environment, no tenant-isolation fixtures, or no way to inspect persisted changes, the verification report should say so. The correct response is to identify the missing evidence, not lower the acceptance criteria until everything appears green.\n\nI also separate new failures from existing failures. If an unrelated test was already failing, I do not edit it, skip it, or change its expectations just to produce a green result. I document the failure in the PR, explain why it appears unrelated, and leave the decision visible to reviewers.\n\nSome decisions remain fundamentally human. Whether two groups should share access to a sensitive resource is a product and security policy question before it is a query implementation question. An agent can implement the stated rule and test that implementation, but the team must decide whether the rule is correct.\n\nThe workflow I use is therefore not \"make the agent pass every check.\" It is to define observable expectations, collect independent evidence, review the change against its intent, and explicitly record what has not been established. That process makes code written by an agent easier to understand, verify, and maintain.\n\nThis is the foundation of *Making Coding Agents Reliable*, a six-article series about practical workflows for working with AI coding agents. The next article covers CLAUDE.md and Skills, including how to make project instructions useful without confusing instructions with enforcement.\n\n**What's the one check you never skip before accepting an AI-generated change?**", "url": "https://wpnews.pro/news/can-you-trust-code-you-didn-t-write", "canonical_source": "https://dev.to/thetonynotes/can-you-trust-code-you-didnt-write-2e97", "published_at": "2026-10-24 23:34:34+00:00", "updated_at": "2026-10-24 23:41:36.412244+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools", "ai-safety"], "entities": ["Claude Code", "Anthropic", "TypeScript", "pnpm", "React"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/can-you-trust-code-you-didn-t-write", "markdown": "https://wpnews.pro/news/can-you-trust-code-you-didn-t-write.md", "text": "https://wpnews.pro/news/can-you-trust-code-you-didn-t-write.txt", "jsonld": "https://wpnews.pro/news/can-you-trust-code-you-didn-t-write.jsonld"}}