Most drift tooling checks two things at once. The OpenAPI spec against the server. The server against the request client. Two-way comparisons catch obvious contract breaks, but they miss the more interesting failure mode: the spec, the requests your teams actually send, and the responses your running service actually returns can all quietly disagree with each other at the same time.
I hit this recently on a fintech-style API where the spec said one thing, our Postman Collection was updated in a hurry to send another, and the server returned a third shape entirely. Everything worked. Nothing was correct.
This post walks through how I use the Postman AI Engineer and the Context Graph to find that kind of three-way drift, with the exact prompt I sent and the sort of output it produced.
Why two-way drift checks miss the interesting bugs #
The typical spec-to-code check works like this: parse the OpenAPI file, hit the running server, compare responses against the schema. That workflow catches the case where an engineer changed the response body and forgot to update the spec.
But APIs live inside an organization, not on a whiteboard. There are three sources of truth in play at any moment:
- The specification, which describes the contract you promised. - The request client, meaning the Postman Collections your team actually runs (in CI, in demos, in QA). These are documented consumers of the API. - The running server, meaning the responses production or staging really returns.
Two-way tooling reduces this to pairs. Spec vs code. Spec vs tests. Tests vs code. Any pair on its own can look clean while the third side is quietly wrong. Some concrete examples I have run into:
- The spec says
amount
is an integer. The collection sends it as an integer. The server, since a recent refactor, coerces the input to a string internally and echoes it back as a string in the response. Spec-vs-collection looks fine. Spec-vs-server fails. Collection-vs-server passes on request and fails on response. - The spec says a field is required. The server accepts requests without it. The collection has stopped sending it. Everything “works” until the next consumer reads the spec and assumes the field will always come back.
- The spec says the endpoint returns problem+json errors per RFC 9457. The server still returns the older ad-hoc error shape. The collection asserts the older shape, so tests are green, and the spec has been misleading readers for months.
None of these get caught by a single pairwise diff. You need something that can hold all three views at once.
What the Context Graph actually knows #
The Context Graph is the piece that makes the three-way check practical. It is a continuously updated map of the APIs, Postman Collections, specifications, environments, workspaces, monitors, and governance rules across your Postman organization, plus the relationships between them.
The important word is relationships. The graph knows that this OpenAPI spec is the canonical contract for that service, that these Postman Collections are documented consumers of that spec, that this Postman Monitor is a live validation surface hitting the staging URL, and that this workspace owns all of the above. When you ask a question, the AI Engineer walks the graph instead of guessing.
The nodes and edges that matter for drift
Any graph is only as useful as the things it chooses to model. The Context Graph models the artifacts a drift question actually needs:
Specifications are nodes. Every version of every OpenAPI (orAsyncAPI, Smithy, protobuf, GraphQL) spec is retained, so the graph knows what the contract said last month as well as what it says today. That is what lets the AI Engineer answer “when did this drift start,” not only “is there drift right now.”Postman Collections are nodes, and every saved request inside them is a child node. The graph does not treat a collection as an opaque JSON blob. It sees the endpoint, the method, the headers, the body, and the test script assertions as individually addressable facts.Live surfaces are nodes. A Postman Monitor is one; a manual collection run against a running URL is another. Both of them capture real request/response pairs, and those pairs land in the graph with timestamps.Environments are nodes, which is how the graph knows which base URL and which auth material were in play when a particular request went out. Without that, “the collection sends X” would be ambiguous across dev, staging, and prod.Workspaces, teams, and governance rules are nodes too. That is what turns “the spec is wrong” into “the spec owned by the Payments team is wrong, and the rule that flags missingrequired
fields did not fire because it is scoped to a different workspace.”
The edges carry as much information as the nodes. A consumes
edge from a Postman Collection to a spec is what lets the AI Engineer answer “who breaks if I change this endpoint.” A validates
edge from a Postman Monitor to a spec is what lets it distinguish “the server used to match the spec and no longer does” from “the server never matched the spec.” A derives_from
edge between two spec versions is what makes historical drift questions cheap to answer.
Why the graph beats hand-rolled diffing for three-way drift
You could, in principle, do three-way drift detection by hand. Export the OpenAPI file, export the Postman Collection JSON, tail your server logs for a day, write a script that reconciles the three, and try to remember what changed since last quarter. I have done this. It works exactly once, on the day you build it, for the one API you built it for.
The Context Graph does that reconciliation continuously, for every API in your organization, without a script. Two properties in particular are what make it fit for three-way drift specifically:
All three sides are first-class citizens. The spec is not treated as ground truth against which the other two are measured. Each side is a set of nodes the graph can compare pairwise or all together, and it can tell you which two of the three agree when the third dissents. That is exactly the “which side is wrong” question a two-way diff cannot answer.The graph is timestamped end-to-end. Every node has a version and every edge has acreated_at
, so the AI Engineer can reason about “the response shape changed on this deploy” or “the collection assertion has always been wider than the spec.” Point-in-time diffing is what turns drift from a mystery into a git blame you can act on.
There is a third quieter benefit. Because governance rules and workspace ownership are in the graph, the AI Engineer can attach a name to every drift row. “The refunded
enum value shows up in production and is not in the spec. The spec is owned by @payments-platform
, and their governance rule requires enum changes to bump a spec version.” That is the difference between a drift report you file in a ticket and one you can route directly.
That is why the AI Engineer can compare the three sides without you stitching anything together. The spec is a node. The collection’s saved requests are nodes. The monitor’s recent runs (and the responses they captured) are nodes. Drift is a diff across all of those, not a pair of them, and the graph has already done the joins.
The prompt #
Here is the actual prompt I sent to the AI Engineer. I tried to be specific about which three sides I wanted compared, because the default temptation is to fall back to spec-vs-code:
Compare the spec to what the collection actually sends and what the running server returns. Show me all three way drift, not just spec vs code.
That is the whole thing. No JSON schema to attach, no scripts to paste. The Context Graph resolves the spec, the consuming Postman Collection, and the running server for that API on its own.
You can send this prompt from the Postman interface, from Slack, or via the API. I usually run it in Slack when I want the report to land next to the PR discussion.
What the report looks like #
The output arrives as a per-endpoint breakdown. Each row lists what the spec says, what the collection actually sends, and what the server actually returns, so you can see which side is wrong at a glance.
A simplified excerpt from a real run:
POST /payments
spec: body.amount is integer, required
collection: body.amount is integer, sent
server: response.amount is string ("120.00")
verdict: server-side drift (integer → string response)
GET /payments/{id}
spec: response.status is string enum ["pending","captured","failed"]
collection: asserts response.status in ["pending","captured","failed","refunded"]
server: returns "refunded" in ~4% of recent responses
verdict: three-way drift (spec, collection, and server all disagree)
DELETE /payments/{id}
spec: 204 No Content
collection: asserts 200
server: returns 200 with empty body
verdict: spec-vs-implementation drift; collection is aligned with server, not spec
The GET /payments/{id}
row is the interesting one. Nothing in a spec-vs-code check would flag it. The spec and the server disagree on the enum. The collection has its own enum that is wider than the spec and narrower than reality. All three are wrong in different directions.
That is the class of bug the Context Graph is good at surfacing, because it can see the collection’s assertions as a first-class artifact instead of ignoring them.
Following up in the same conversation #
The AI Engineer is agentic, so once you have the initial report you can keep asking. The prompts I use most often after the first pass:
For each three-way drift row, tell me which side I should trust and why.
Which Postman Collections consume /payments and would break if I make the
spec match the server response for `amount`?
Draft the OpenAPI patch to align the spec with the server for the enum on
GET /payments/{id}, and list the collections that need updated assertions.
That last one is where the “context” part earns its name. The AI Engineer walks the graph, finds every Postman Collection whose requests touch the endpoint, and reports the ones whose test scripts assert against the old shape. You get a list of Postman Collection names, not a hopeful “you may want to search your repo.” I have used this to keep partner-facing collections from silently regressing after an internal cleanup.
You can read more about that downstream-dependency workflow in Managing downstream dependencies with the AI Engineer.
How to run this in your workspace #
You need three things wired up before the AI Engineer can do a three-way check for real:
A published OpenAPI spec in a Postman workspace. TheSpec Hub docswalk through creating and versioning one. Without a spec, the AI Engineer only has two sides to compare.A Postman Collection that actually calls the API, ideally the one your team uses in CI or demos. This is the “what the client sends” side. See thecollections documentationif you are starting from scratch.A recent live surface. APostman Monitorhitting staging or production works well because it gives the Context Graph fresh responses to reason over. In a pinch, the AI Engineer can run the collection itself against a running URL in its sandbox.
With those in place, the prompt above works as written. If you want to run it from CI, the Postman CLI exposes the AI Engineer through the API, so you can wire the drift check into your pull request pipeline the same way you would run tests.
I also keep an environment variable for the base URL and the auth token, because I do not want either hardcoded in the collection:
{
"id": "payments-staging",
"name": "Payments Staging",
"values": [
{ "key": "base_url", "value": "https://staging.api.example.com", "type": "default" },
{ "key": "auth_token", "value": "{{secret_staging_token}}", "type": "secret" }
]
}
Import that as a new environment, mark the token as secret, and the AI Engineer will use it when running the collection against the live surface.
Things to watch for #
A few gotchas I have hit that are worth flagging up front.
Recent traffic matters. The Context Graph reasons over the responses it has actually observed. If your monitor has not run for a week and the server changed yesterday, ask the AI Engineer to run the collection first and then rerun the drift analysis. Otherwise you are comparing the spec and the collection to a stale snapshot of production.
Collection assertions are part of the “consumer” side. If your test scripts silently pin the shape of a response, that pin is itself a promise your team is making. When the drift report says the collection disagrees with the spec, that is often a sign that someone patched around a spec bug months ago and forgot to fix the spec.
Not every drift row is a bug. Sometimes the server is deliberately returning fields that are not in the spec yet because a feature is behind a flag. The useful move is to ask the AI Engineer to classify each row as “spec is wrong,” “server is wrong,” or “collection is wrong,” and to trace the last change on each side from the graph. That usually points at the guilty commit or Postman Collection revision without any archaeology.
Auth is often the loudest offender. In my last run, the noisiest drift was not on business fields at all. The spec described a bearer token in the Authorization
header, half the collection requests were sending it as a query parameter left over from an older auth model, and the server accepted both. Nothing broke. Everything was wrong. That is exactly the kind of quiet drift that three-way comparison catches and pairwise checks miss. If you want a sanity check on your own auth flow while you are in there, the Postman API Security rules are worth pointing the AI Engineer at as a follow-up.
Add it to your review workflow #
The workflow that has worked best for me is not one-off drift hunts. It is running the three-way prompt on every non-trivial API pull request. The AI Engineer posts the report as a Slack thread, the reviewer scans it before looking at the diff, and any row that is not “no drift” gets addressed before merge.
Two things I like about this setup: the spec, the collection, and the server all get reviewed as a single system instead of three separate artifacts, and the report is written in terms the whole team already knows (endpoints, fields, status codes) rather than an internal governance dialect.
Try it against your own API. Point the AI Engineer at a spec, a Postman Collection, and a running server, and send the prompt above verbatim. Then run it a second time after your next merge and see which of the three sides moved. The interesting drift almost always shows up in the difference between those two runs.
Resources #
Introducing the AI EngineerWe gave two AI agents the same API spec drift problem. The difference was context.Managing downstream dependencies with the AI EngineerPostman Spec Hub documentationPostman Collections documentationPostman Monitors documentationThe Postman CLI documentationOpenAPI InitiativeRFC 9457: Problem Details for HTTP APIs