{"slug": "the-data-was-public-the-agent-path-wasn-t-so-his-mock-became-my-documentation", "title": "The Data Was Public. The Agent Path Wasn't. So His Mock Became My Documentation.", "summary": "A developer documented how a contributor named Pouya cloned the ask-the-record project and opened its first outside pull request, a fix to the agent's GROQ schema routing, only for the maintainer to find the patch's worked examples and embedded schema did not match the live Sanity dataset. Testing the PR's own queries against the public dataset returned HTTP 400 and null results, and the patch described fields such as claim.subject, finding.finders and patch.findingRef that do not exist in the database. The maintainer traced the underlying gap to the agent's authenticated Context MCP endpoint, which requires an organization API token rather than a project token, so cloning the repo gave the contributor no way to authenticate the path the agent actually uses.", "body_md": "Two commands against the same data. Run them yourself:\n\n``` bash\n$ curl -sS --get 'https://u58x3mt0.api.sanity.io/v2025-08-15/data/query/production' \\\n       --data-urlencode 'query=count(*)'\n\n{\"query\":\"count(*)\",\"result\":120,\"syncTags\":[\"s1:dmd/mg\"],\"ms\":11}\n```\n\n120 documents, no key, no account. Now the path my agent actually uses:\n\n``` bash\n$ curl -sS -o /dev/null -w '%{http_code}\\n' \\\n    'https://api.sanity.io/v1/context/organizations/<ORG>/mcp/self-correcting-systems'\n\n401\n```\n\nSame underlying dataset. One route is publicly queryable. The route my agent takes goes through\n\nan authenticated Context MCP endpoint, which needs an organization API token — not a project\n\ntoken, which Sanity explicitly does not accept there. I had not provisioned Pouya any access to my\n\norganization, so cloning the repo gave him no way to authenticate the path the agent actually uses.\n\nI am being careful about \"same\" here: an MCP endpoint can be configured with its own sources and a\n\n`groqFilter`, so it is not guaranteed to expose the same 120 documents the public route does. Same\n\nsource, different access path, different interface contract. That distinction turns out to be the\n\nbug.\n\nThat gap is the whole story, and I did not know it was there until the contributor who fell into\n\nit told me.\n\nA developer named Pouya read a post of mine, decided my diagnosis was wrong, cloned the repo and\n\nopened a pull request. First outside contribution the project has ever had.\n\n```\nPR #1   https://github.com/keniel13-ui/ask-the-record/pull/1\n        PouyaZX4:fix/groq-schema-routing\nopened  2026-09-22T12:33:52Z\nmerged  2026-09-24T00:53:58Z   ->  36.3 hours\n        3 commits, 1 file, +25 / -12, merge 4a2940f\n```\n\nIt is merged, so it will not appear in the default pull request list, which shows open ones only.\n\nA reviewer told me the repo had no PRs at all for exactly that reason, which is a small instance of\n\nthe same mistake this whole post is about.\n\nHis diagnosis was reasonable. My agent queries a Sanity dataset with GROQ, and he thought the\n\nfailures came from the model guessing at document types and inventing query syntax. His fix: put\n\nthe real schema and worked examples into the tool description, so the model is told the shape\n\ninstead of guessing it.\n\nSound idea. I would have tried the same thing.\n\nThen I ran his examples.\n\nBoth from the pull request description. Here is the one for a claim, against the live dataset:\n\n```\n*[_type == \"claim\" && _id == \"claim-ledger-population\"][0].{status, expiryStatus}\n\n-> HTTP 400  \"attribute expected\"\n```\n\nA stray dot between `[0]` and `{`. Remove it and it works:\n\n```\n*[_type == \"claim\" && _id == \"claim-ledger-population\"][0]{status, expiryStatus}\n\n-> {\"status\": \"standing\", \"expiryStatus\": \"no_expiry_set\"}\n```\n\nAnd the one for a patch record:\n\n``` php\n*[_type == \"patch\" && findingRef._ref == \"finding-b1\"][0]\n\n-> null\n```\n\nNull for two separate reasons. There is no `findingRef` field on my patch documents — it is\n\n`findings`, and it is an array. And the ids are case sensitive: `finding-B1`, not `finding-b1`.\n\nWritten correctly:\n\n``` php\n*[_type == \"patch\" && \"finding-B1\" in findings[]._ref][0]{sha, inMain}\n\n-> {\"sha\": \"dd1a654\", \"inMain\": false}\n```\n\nThe patch did not just carry two broken examples. It wrote a schema into the tool description,\n\nand the schema did not match my data:\n\n| the patch told the model | what the database actually has | \n|---|---|\n| `claim.subject` | nothing | \n| `claim.statement` | `text` | \n| `finding.finders` | `foundBy` | \n| `finding.verifiedReceipts` | `commentId` or`commentIds` , depending on the record | \n| `patch.findingRef` | `findings[]` | \n| `patch.commitHash` | `sha` | \n\nCheck it yourself, no key required:\n\n```\ncurl -sS --get 'https://u58x3mt0.api.sanity.io/v2025-08-15/data/query/production' \\\n  --data-urlencode 'query=*[_id==\"finding-B1\"][0]'\n```\n\nThat is the record his example referenced. An earlier draft of this post ran the same command\n\nagainst `finding-B8` instead, because B8 carries `commentIds` and B1 carries the singular\n\n`commentId`, which made my table read cleaner. That is picking the sample that proves the point,\n\nin a post about not doing that, and a reviewer caught it before it shipped.\n\nSo a patch written to stop a model from inventing schema would have installed a mismatched schema\n\ninto the exact place the model reads it from. Worse than the bug it fixed, because a guess made at\n\nruntime is a guess, while a guess sitting in the tool description arrives with the authority of\n\ndocumentation.\n\nI wrote a draft of this post that said I did not know how the patch was produced and was not going\n\nto guess. That was the right call, because when I asked, the answer was nothing I would have\n\nguessed. In his words, published with his permission:\n\nThe dataset itself is public of course, but I wasn't able to replicate the full MCP setup on my\n\nend at the time I tested it so I was hitting that 401 on the Context MCP endpoint. So I just\n\ntested the query generation logic against an offline mock schema on LM Studio instead, which is\n\nwhere those field names came from.\n\nNothing was hallucinated. He hit the 401 at the top of this post.\n\nHe could read every one of my 120 documents in a browser. He could not run my agent, because the\n\nagent does not take the public route — it goes through an org-scoped MCP endpoint needing an\n\norganization credential I had not provided and was not going to publish with the repository.\n\nSo he did the careful thing. He built an offline mock of the schema and tested the query\n\ngeneration logic against that. And when he built the mock he chose better names than mine:\n\nWhen creating that schema, I used clean, self-describing semantic names (commitHash instead of\n\nsha, verifiedReceipts instead of commentIds, finders instead of foundBy) because explicit naming\n\nmakes it way easier for the SLMs to understand what fields represent and prevents confusion.\n\nHe is right about that, by the way. `commitHash` is a better name than `sha`. `foundBy` is worse\n\nthan `finders`. Every wrong name in that table is the name a competent person would pick if they\n\nwere designing the schema rather than reading it.\n\nOne of them is worse than a naming difference. `verifiedReceipts` does not just rename\n\n`commentIds` — it asserts something the data cannot support. A comment id says a comment exists at\n\nthat address. It says nothing about whether anyone verified it. That distinction is the entire\n\nsubject of the record it was describing.\n\nThe mock is not the bug. The mock was a reasonable response to a 401.\n\nThe bug is that **my repository advertised a reproduction path that is not the one the system uses.** The README says \"Public dataset\" and gives the URL at the top of this post. That is true,\n\n`curl`, no key, check it\nA contributor following my README lands in a place where every record is visible and nothing is\n\nrunnable. The only way forward is to build a stand-in. And a stand-in built at that boundary does\n\nnot stay at the boundary — his mock's field names travelled from a local LM Studio test into the\n\ntool description, which is where my harness explicitly tells the model what the schema is.\n\nThat is the whole difference in one line. A runtime guess is visibly a guess. Put the same guess in\n\nthe tool description and I have promoted it into authoritative guidance.\n\nI have shipped the same class of error. I described my own validator to someone as checking\n\nwhether a cited record matched what was retrieved. It does not. I was describing the system I\n\nmeant to build, from memory, instead of opening the file.\n\nNot an argument. Two commands.\n\nI pulled his branch, ran both examples against the live dataset, and sent him exactly what came\n\nback: the 400, the null, and the corrected queries with their real output. No opinion about his\n\napproach, no debate about the diagnosis.\n\nHe pushed a revision in about two hours. Every field name corrected, both queries rewritten:\n\n``` php\nclaim  claim-ledger-population -> standing / no_expiry_set\npatch  for finding-B1         -> dd1a654 / false\nfinding finding-B8            -> commentIds ['3ee98'], status unbuilt\n```\n\nAll three still return exactly that today. Note the third is **B8**, not the B1 I told you to curl\n\nabove — B1 carries the singular `commentId` `\"3eanf\"` and status `implemented`. Those are two\n\ndifferent records and I have mixed them up once already while writing this.\n\nStill one stale reference in a routing rule, so I sent that too, with the null it produced. He\n\nfixed it that morning. I re-ran the three patterns, checked the file still parsed and still had no\n\nthird-party imports, and merged it.\n\nThree commits, thirty six hours, between two people who have never met. And one detail I like:\n\nthe pull request body still shows `[0].{status, expiryStatus}`, the broken stray-dot form, after\n\nthe merged code had moved to the corrected one. Documentation can keep a false schema alive after\n\nthe executable stops using it, which is the same failure as this whole post, one layer up.\n\nIf you maintain anything an outsider might contribute to, run this check:\n\n**Can someone who clones your repo actually execute the path your system takes, or only the path your README documents?**\n\nFor me those were different, and the gap was invisible from the inside because I hold the token.\n\nEverything worked on my machine for a reason I never had to think about.\n\nWhen they differ, a contributor's only option is a mock. They will build a good one — Pouya's\n\nnames were better than mine. And then the mock's assumptions become your documentation, because\n\nthe tool description is documentation, and the model does not know it was written against a\n\nfixture.\n\nThere is a third option, and it is better than either of the two I first wrote down. I said the\n\nfix was to make the real path reachable, or to say in the README that it is not. Both are weak,\n\nbecause both still leave the contributor inventing the contract.\n\n**If outsiders cannot execute a privileged dependency, ship them a reproducible contract for it.**\n\nA fixture generated from the real schema, checked into the repo, lets someone test against the same\n\ninterface without ever receiving my organization credential. The pipeline becomes\n\n``` php\nproduction schema -> generated contract fixture -> contributor harness\n```\n\ninstead of\n\n``` php\nREADME -> unreachable MCP -> contributor invents a substitute schema\n```\n\nThe problem was never that Pouya mocked the boundary. It is that my repository gave him no\n\ncanonical boundary to mock, so he had to design one — and a well-designed guess is still a guess.\n\nThat generalises past Sanity and past agents. Anything an outsider cannot run — a private API, a\n\npayment sandbox, an internal queue, an OAuth service — has this shape. If you do not own the\n\nstand-in, your contributors will build one, and theirs will encode their assumptions instead of\n\nyours.\n\nThanks to Pouya for the patch, for taking two rounds of corrections without once arguing the\n\ndiagnosis, and for answering the question about where those names came from when he could easily\n\nhave let me publish a guess instead. The merge is `4a2940f`. Every query above can be run by\n\nanyone against the public dataset — which, as it turns out, is exactly the point.", "url": "https://wpnews.pro/news/the-data-was-public-the-agent-path-wasn-t-so-his-mock-became-my-documentation", "canonical_source": "https://dev.to/kenielzep97/the-data-was-public-the-agent-path-wasnt-so-his-mock-became-my-documentation-413a", "published_at": "2026-09-30 04:36:02+00:00", "updated_at": "2026-09-30 04:46:45.912665+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools", "agent-protocols"], "entities": ["Pouya", "Sanity", "ask-the-record", "GROQ", "GitHub", "Context MCP"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/the-data-was-public-the-agent-path-wasn-t-so-his-mock-became-my-documentation", "markdown": "https://wpnews.pro/news/the-data-was-public-the-agent-path-wasn-t-so-his-mock-became-my-documentation.md", "text": "https://wpnews.pro/news/the-data-was-public-the-agent-path-wasn-t-so-his-mock-became-my-documentation.txt", "jsonld": "https://wpnews.pro/news/the-data-was-public-the-agent-path-wasn-t-so-his-mock-became-my-documentation.jsonld"}}