{"slug": "ai-agent-capability-diffs-review-new-powers-before-they-reach-production", "title": "AI Agent Capability Diffs: Review New Powers Before They Reach Production", "summary": "A developer outlined a vendor-neutral pattern for \"AI agent capability diffs,\" which snapshot the operational powers available to an agent—effect, resource, sensitivity, identity boundary, and environment—and compare each release against the last approved one to surface new data-exfiltration paths that line-level code review misses. The approach generates a deterministic manifest from source-controlled tool definitions and deployment configuration rather than having an LLM infer permissions from tool descriptions, and emits \"unknown\" when a tool's side effects can't be determined. The author positions the diff as release-time evidence that complements IAM, runtime policy, and human approval rather than replacing them.", "body_md": "An agent can gain a dangerous new power in a change that looks harmless in a code review. A new MCP server, a broader OAuth scope, or one extra tool field may let it read a customer record and send it somewhere else. The diff says “add connector.” It does not say “new data-exfiltration path.”\n\nThat is the gap an **AI agent capability diff** closes. Instead of reviewing only source files, record the operational powers available to each agent, compare the new snapshot with the approved one, and require a deliberate decision for meaningful additions. This is not a replacement for IAM, runtime policy, or human approval. It is the release-time evidence that tells those controls where the risk changed.\n\nThis guide builds a small, vendor-neutral pattern you can use with MCP servers, function tools, SDK-based agents, and internal APIs.\n\nTraditional reviews are good at showing changed lines. They are weaker at showing the effect of a change spread across a tool manifest, an OAuth configuration, a prompt package, and a deployment environment.\n\nConsider this change:\n\n```\n{\n  \"name\": \"search_customer_notes\",\n  \"endpoint\": \"/notes/search\",\n  \"scopes\": [\"notes.read\"]\n}\n```\n\nIt looks read-only. But if the same agent already has `post_webhook`, an untrusted document can steer it to copy retrieved notes into an external request. Neither tool alone explains the risky path. A capability diff gives reviewers a compact statement: **this release adds access to sensitive notes and the agent already has external egress**.\n\nThe idea works because it asks a stable question on every release:\n\nWhat can this agent do now that it could not do in the last approved release?\n\nThat question is more useful than “did the tool list change?” A renamed read tool may not change authority. A new parameter that broadens a filesystem path can. So the unit of review is a normalized capability, not a tool’s friendly description.\n\nStart with a manifest generated from source-controlled tool definitions and deployment configuration. Do not ask an LLM to infer permissions from descriptions at review time. Descriptions are helpful context, but enforcement evidence should be deterministic.\n\nHere is a deliberately small TypeScript shape:\n\n```\ntype Effect = \"read\" | \"write\" | \"execute\" | \"network\" | \"unknown\";\ntype Sensitivity = \"public\" | \"internal\" | \"customer\" | \"secret\" | \"unknown\";\n\ntype Capability = {\n  agent: string;\n  id: string;                 // stable: \"crm.customer.write\"\n  effect: Effect;\n  resource: string;           // \"crm:customer\" or \"filesystem:/work/*\"\n  sensitivity: Sensitivity;\n  environment: \"dev\" | \"staging\" | \"production\";\n  boundary: \"service-token\" | \"user-delegated\" | \"human-approved\";\n  source: string;             // tool/server/config location\n};\n\ntype Snapshot = {\n  version: 1;\n  generatedAt: string;\n  capabilities: Capability[];\n};\n```\n\nUseful fields answer five review questions:\n\n| Field | Reviewer question | \n|---|---|\n| `effect` | Can it observe, change, run, or send? | \n| `resource` | Which records, paths, endpoints, or infrastructure? | \n| `sensitivity` | How harmful is unintended access? | \n| `boundary` | Which identity or approval boundary protects it? | \n| `environment` | Is the power present where real users and data exist? | \n\nAvoid a single “risk score” as the source of truth. Scores can prioritize attention, but a reviewer needs the underlying facts. `filesystem.write` on `/tmp/agent` and `filesystem.write` on a production deployment manifest are not equivalent.\n\nEvery adapter should emit explicit facts it knows. If a tool’s side effect or resource scope cannot be determined, emit `unknown`; never silently label it read-only.\n\n```\nfunction classifyTool(tool: {\n  name: string;\n  server: string;\n  annotation?: { readOnly?: boolean; destructive?: boolean };\n  scopes?: string[];\n}): Capability[] {\n  const effect: Effect = tool.annotation?.destructive\n    ? \"write\"\n    : tool.annotation?.readOnly\n      ? \"read\"\n      : \"unknown\";\n\n  return [{\n    agent: \"support-triage\",\n    id: `${tool.server}.${tool.name}`,\n    effect,\n    resource: tool.scopes?.join(\",\") || \"unknown\",\n    sensitivity: \"unknown\",\n    environment: \"production\",\n    boundary: \"service-token\",\n    source: `mcp:${tool.server}/${tool.name}`\n  }];\n}\n```\n\nYour adapters may read MCP tool metadata, OpenAPI operations, cloud IAM policy, filesystem allowlists, or custom function definitions. Keep their output separate from policy. The collector says what it observed or declared; policy decides whether that state may ship.\n\nAlso save the raw discovery artifact and a hash of the normalized snapshot. When an adapter changes, you need to distinguish a real authority change from a classifier upgrade.\n\nCompare capability identity plus its meaningful fields. A tool with the same name but a broader resource selector is a modification, not a no-op.\n\n``` js\nconst fingerprint = (c: Capability) =>\n  [c.id, c.effect, c.resource, c.sensitivity, c.environment, c.boundary].join(\"|\");\n\nfunction addedCapabilities(before: Snapshot, after: Snapshot) {\n  const old = new Set(before.capabilities.map(fingerprint));\n  return after.capabilities.filter(c => !old.has(fingerprint(c)));\n}\n\nfunction classifyChange(c: Capability) {\n  if (c.effect === \"unknown\" || c.resource === \"unknown\") return \"review\";\n  if (c.environment !== \"production\") return \"notice\";\n  if ([\"write\", \"execute\", \"network\"].includes(c.effect)) return \"block\";\n  if (c.sensitivity === \"secret\" || c.sensitivity === \"customer\") return \"review\";\n  return \"notice\";\n}\n```\n\nThe policy above is intentionally strict about unknowns and production mutations. Tune it to your delivery process, but do not solve noisy findings by converting everything to a notice. First improve the classifier, add an expiry-bound exception, or narrow the resource field.\n\nThe most useful findings involve two capabilities that are individually expected. Add simple relationship rules before reaching for complex graph analysis.\n\n```\nfunction has(caps: Capability[], effect: Effect, sensitivity?: Sensitivity) {\n  return caps.some(c => c.effect === effect &&\n    (!sensitivity || c.sensitivity === sensitivity));\n}\n\nfunction crossCapabilityFindings(caps: Capability[]) {\n  const findings = [];\n  if (has(caps, \"read\", \"customer\") && has(caps, \"network\")) {\n    findings.push(\"Customer-data read access can combine with external network access\");\n  }\n  if (has(caps, \"execute\") && has(caps, \"read\", \"secret\")) {\n    findings.push(\"Command execution can combine with secret access\");\n  }\n  return findings;\n}\n```\n\nThis does not prove exfiltration or compromise. It identifies a path that needs an owner, a resource boundary, and a runtime control. Make that distinction explicit in the CI output; security findings should be evidence, not drama.\n\nA useful check gives reviewers three things: additions, meaningful scope changes, and the decision that allowed them.\n\n```\n- run: npm run capabilities:generate -- --out current.json\n- run: npm run capabilities:diff -- approved.json current.json --format markdown > capability-diff.md\n- run: npm run capabilities:policy -- approved.json current.json\n```\n\nFor every `block` or `review` change, require a compact approval record:\n\n```\nchange: crm.customer.write\nowner: customer-platform\nreason: \"Agent creates a draft case after a verified support request\"\ncontrols:\n  - user-delegated token\n  - schema validation\n  - approval for production writes\nrollback: disable tool registration\nexpires: 2027-02-01\n```\n\nThe expiry matters. Permanent exceptions quietly become undocumented authority. When the date arrives, the same team must renew, narrow, or remove the capability.\n\nOne snapshot is not enough in a mature system. Keep three views:\n\nThese views uncover different failures. A declared-but-unallowed tool may mean a bad rollout. An observed-but-undeclared tool may mean shadow configuration. An allowed-but-never-used power is an opportunity to reduce reach.\n\nDo not turn the release diff into a runtime authorization system. Keep runtime policy as the final decision point, because tool schemas and configuration can drift after a build. The diff makes that drift visible; it does not eliminate the need to deny unsafe calls when they occur.\n\nThe hardest bugs are often not new tool names. They are small expansions hidden inside an existing capability. A filesystem tool may change from one repository to a parent directory. A CRM query may add a `tenant=*` selector. A network client may accept arbitrary URLs instead of an allowlist. If the snapshot stores those selectors as a single opaque string, reviewers must mentally parse a noisy diff.\n\nNormalize scopes where practical, then render a sentence beside the machine comparison:\n\n``` php\nMODIFIED  github.pull_request.write\nresource  repo:acme/help-center  ->  repo:acme/*\ndecision  BLOCK: production write scope broadened\nowner     developer-experience\n```\n\nThis output does not need to be clever. It needs to be precise enough for someone unfamiliar with the implementation to decide whether the new reach is intentional. Keep stable IDs for correlation, but show the human meaning of a selector, environment, and boundary in the pull-request comment.\n\nResource scope also needs a vocabulary. Define how your system represents a single record, a tenant, a repository, a directory tree, an API host, and a wildcard. Without it, one adapter might emit `all` while another emits `*`, making semantically identical states appear different—or worse, making a broadening look harmless. Add contract fixtures for that vocabulary before you start enforcing diffs.\n\nBe especially careful with credentials. A tool’s input schema might look restricted while its service token has account-wide access. The snapshot should record the effective credential boundary when it is known, and produce `unknown` when it is not. Reviewers should never confuse an agent’s intended scope with the access the downstream service will actually honor.\n\nStart in report-only mode for two or three releases. Measure unknown classifications, repeated additions, and reviewers’ response time. Then block only new production `write`, `execute`, and broad `network` capabilities. Add sensitive read paths next. This sequence creates space to repair adapters without normalizing exceptions.\n\nTest the workflow like production code:\n\n`/workspace/project/*` to `/**` must be a blocking scope expansion.`unknown` and require review.\nFinally, make ownership visible. Every production capability needs a system owner, not just the person who added a tool. Reviewers can approve a change; owners maintain the answer when the tool changes six months later.\n\nBefore you ship an agent-tool change, confirm:\n\nThe payoff is modest but important: reviewers stop guessing whether an agent “has more access.” They can see what changed, why it exists, who owns it, and which control keeps it bounded.\n\nNo. IAM is a crucial input, but an agent capability diff combines identity scope with tool effects, resource selectors, environment, and agent-to-tool wiring. It answers the operational question of what the agent can do through its complete tool surface.\n\nUsually not. Read-only tools can still expose sensitive data, so classify their data scope and use review for customer or secret access. Block only where your risk policy calls for it; noisy checks are ignored checks.\n\nEmit `unknown` for effect or scope and require review. Add an adapter override with an owner and tests rather than assuming the description is accurate. Unknown is a data-quality task, not proof of safety.\n\nNo. It runs before release and detects changed authority. Runtime authorization evaluates the actual call, user, resource, and context after deployment. Use both layers.\n\nInventory production tools, normalize their effect and resource scope, store one approved JSON snapshot, and fail CI on newly added write, execute, or external-network powers. Add sensitive-data and relationship checks after the basic signal is trustworthy.\n\nGenerate them on every pull request and release candidate. Also run scheduled discovery against deployed environments so you can find drift introduced by configuration, dependencies, or manually connected servers.", "url": "https://wpnews.pro/news/ai-agent-capability-diffs-review-new-powers-before-they-reach-production", "canonical_source": "https://dev.to/jackm-singularity/ai-agent-capability-diffs-review-new-powers-before-they-reach-production-o57", "published_at": "2026-10-29 03:39:30+00:00", "updated_at": "2026-10-29 03:43:29.437020+00:00", "lang": "en", "topics": ["ai-agents", "ai-safety", "agent-protocols", "developer-tools", "mlops"], "entities": ["MCP"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/ai-agent-capability-diffs-review-new-powers-before-they-reach-production", "markdown": "https://wpnews.pro/news/ai-agent-capability-diffs-review-new-powers-before-they-reach-production.md", "text": "https://wpnews.pro/news/ai-agent-capability-diffs-review-new-powers-before-they-reach-production.txt", "jsonld": "https://wpnews.pro/news/ai-agent-capability-diffs-review-new-powers-before-they-reach-production.jsonld"}}