We Gave Two AI Agents the Same API Spec Drift Problem. The Difference Was Context.

An experiment comparing two AI agents—Postman AI Engineer and Claude Code with Postman plugin—on the same API spec drift problem found they produced opposite fixes: one updated the spec to match the code, the other updated the code to match the spec. The difference stemmed from the agents' access to context, with Postman AI Engineer leveraging a broader organizational understanding of API meaning.

We Gave Two AI Agents the Same API Spec Drift Problem. The Difference Was Context. API specification drift is one of those failure modes that’s invisible until it isn’t. The OpenAPI https://www.openapis.org/ document says one thing, the running service does another, and somewhere downstream a client breaks because a field your team renamed two sprints ago never made it back into the contract. The hardest part isn’t finding the difference. The hardest part is answering: which side is actually correct? Should the specification change to match the implementation, or should the implementation change to honor the published contract? And, more importantly: what else breaks if we change it? We ran an experiment to see how AI agents handle this. Two agents, one prompt: Identify all API spec drift. Ensure the API contract is valid. One was Claude Code with the Postman plugin https://blog.postman.com/announcing-the-postman-plugin-for-claude-code/ installed, bringing Postman capabilities directly into the developer workflow. The other was the Postman AI Engineer https://blog.postman.com/introducing-the-ai-engineer/ , running natively with the Postman Context Graph. Same repository, same request, two different approaches — and the result reveals something important about AI-assisted development: the quality of an agent’s answer depends on the context it can access. What API specification drift means API drift https://www.wiz.io/academy/api-security/api-drift happens when the behavior of an API no longer matches its specification. A response adds a required field but the schema is never updated. A status code changes from 200 to 201 . A parameter that was optional quietly becomes required. Individually, these changes might seem harmless. Together, they break the contract that consumers build against. Contract testing https://learning.postman.com/docs/design-apis/contract-testing/ helps teams catch these gaps, but detecting drift is only the first step. The harder question is whether the contract should move or the code should move — and that decision needs context. The experiment We pointed both agents at the same API project and ran the same prompt: Identify all API specification drift. Ensure the API contract is valid. Here’s what happened. | Postman AI Engineer | Claude Code with Postman plugin | | |---|---|---| | PR | Rewrote the specification to match the implementation | Rewrote the implementation to match the specification | | Treats as authoritative | The running code | The API specification | | Wall-clock time | 4 min | 5 min | | Cost | $1.69 | $1.93 | Both PRs removed the drift and both were technically valid. They also solved two different problems. Why the agents disagreed Claude Code with the Postman plugin uses Model Context Protocol https://modelcontextprotocol.io/ tools to bring Postman capabilities into the IDE. That bridge gives Claude direct access to API artifacts and workflows while it reasons across the codebase, files, and implementation context in the repository. The Postman AI Engineer approaches the same problem from another dimension. Built on the Postman Context Graph — a continuously updated map of APIs, collections, specifications, environments, workspaces, governance rules, and the relationships between them — it can reason about more than syntax. Instead of only asking “Does this code match this specification?” , it can ask: What does this API mean to this organization? That distinction matters because APIs are not implementation details. They’re formal contracts that encode how systems, teams, and business capabilities interact. Moving from “does this match the spec” to “what does this API mean to the organization” expands the evaluation context. A single endpoint may represent payments, provisioning, identity, or data consistency, so a change can have downstream effects far beyond a code diff. This broader lens catches issues that static validation misses — breaking consumer assumptions, violating SLAs, or altering business-critical behavior. The result is fewer production regressions and less rework because decisions are made with system-wide context, not line-level correctness. A real example: a code change that looks correct but breaks the API ecosystem Imagine a developer adds pagination to an endpoint. The implementation changes: GET /customers Now returns: { "customers": { "id": "123", "name": "Acme" } , "next cursor": "abc123" } The developer updates the OpenAPI file. Everything looks good. A code-focused agent can validate that the endpoint works, the schema matches the implementation, and the tests pass. But the Context Graph reveals something else. The endpoint is referenced by: - a customer onboarding collection - an internal billing workflow - a partner-facing collection - a mock server used by frontend developers - contract tests owned by another team One consumer is still expecting the old response shape. The implementation change is correct. The specification update is correct. The API ecosystem is still broken. The Context Graph surfaces the relationships that are invisible from the repository alone. This is the difference between changing an API and understanding an API. Why context is the bottleneck for AI coding agents AI models keep getting more capable. The limiting factor is usually not reasoning ability — it’s access to the right context. A repository tells an agent what code exists, how services are implemented, and what files changed. It usually doesn’t tell an agent which APIs are consumed by other teams, which collections represent real workflows, which specifications are published, which environments validate production behavior, or which changes violate governance rules https://learning.postman.com/docs/api-governance/api-definition/api-definition-warnings/ . That context lives outside the codebase. It lives across the API lifecycle, and the Postman Context Graph connects those signals. Claude Code + Postman: different strengths, stronger together This isn’t a comparison where one agent replaces the other. Claude Code brings powerful reasoning directly into the developer environment, and the Postman plugin lets developers work with Postman resources and API workflows without leaving their IDE. The Postman AI Engineer adds API lifecycle intelligence through the Context Graph. Together, Claude helps answer “How should I change the code?” while the Postman AI Engineer helps answer “Should this change happen, and what else does it affect?” The future of AI engineering isn’t one universal agent — it’s specialized agents with the context they understand best. What this experiment shows The experiment started as a simple comparison and turned into a lesson about context. Models aren’t the only factor. Code context shows implementation; API context shows impact. The Postman Context Graph provides the missing layer: how APIs connect across teams, consumers, and systems. That’s what moves AI from generating changes to making informed engineering decisions. Try it yourself If you have the integration for Slack set up, you can ask the Postman AI Engineer to analyze drift directly: @Postman Identify all API specification drift. To reason about the same drift from inside Claude Code https://docs.claude.com/en/docs/claude-code/overview , install the Postman plugin https://blog.postman.com/announcing-the-postman-plugin-for-claude-code/ and run setup: claude plugin install postman claude /postman:setup Compare the OpenAPI specification in api/openapi.yaml against the implementation in services/. Identify any drift and show both sides before suggesting a fix. Then compare what each agent surfaces. If the answers diverge from what your team would have done, that’s a signal worth investigating — usually it means a governance rule isn’t encoded yet, or the specification is missing a constraint that everyone has internalized but nobody wrote down. The goal isn’t to find differences. The goal is to understand why they exist — and make changes that preserve the systems built around your APIs. Resources Announcing the Postman plugin for Claude Code https://blog.postman.com/announcing-the-postman-plugin-for-claude-code/ Introducing the Postman AI Engineer https://blog.postman.com/introducing-the-ai-engineer/ Postman Spec Hub documentation https://learning.postman.com/docs/design-apis/specifications/overview/ Postman MCP server overview https://learning.postman.com/docs/reference/postman-api/postman-mcp-server/overview Contract testing in Postman https://learning.postman.com/docs/design-apis/contract-testing/ Postman API Governance https://learning.postman.com/docs/api-governance/api-definition/api-definition-warnings/ OpenAPI specification https://www.openapis.org/ Model Context Protocol specification https://modelcontextprotocol.io/ Claude Code documentation https://docs.claude.com/en/docs/claude-code/overview