This is a submission for the Sanity Challenge, Path One: Ship an Agent That Queries Real Content
pygame and pygame-ce are two separate distributions of the same library. They install under the same pygame namespace, cannot coexist in one environment, and share most of their API. They also maintain two separate documentation sites, and they have drifted: each has functions, classes and modules the other does not.
A tutorial, a Stack Overflow answer or an AI-generated snippet almost never says which one it was written for. When the call fails, the error is a bare AttributeError that gives no hint the distribution is the problem.
So I built a documentation agent that answers for both at once and never blends them. Every claim names the distribution it belongs to, with the version the feature landed in, and the entry it came from.
Asked whether Surface.premul_alpha_ip() works in upstream pygame, it does not just say no. It says the method is pygame-ce only, that upstream has premul_alpha() which returns a copy instead of working in place, and it cites the entry. A keyword search for premul_alpha_ip returns the pygame-ce documentation page and nothing else — which is exactly the trap, because the page looks authoritative and says nothing about the distribution you actually have installed.
The agent ends every answer with machine-readable verdict lines:
VERDICT: premul_alpha = pygame:yes, pygame-ce:yes
VERDICT: premul_alpha_ip = pygame:no, pygame-ce:yes
Those lines exist because of what happened when I tried to measure the thing. More on that below.
Two questions and the test suite, thirty seconds:
https://asciinema.org/a/83DymqBcTnFnhVlT
https://github.com/anaalkmim/pygame-ce-agent
I pointed Sanity Context at the API reference of both documentation sites:
| Source | Pages indexed |
|---|---|
https://www.pygame.org/docs/ref/ |
42 |
https://pyga.me/docs/ref |
42 |
Only /ref, deliberately. Knowledge Bases index up to 150 documents in beta, and the full crawl of both sites came to 196. Rather than trim arbitrarily, I dropped the tutorials, the C API and the index pages. The tutorials are largely inherited from the fork and near-identical across both, so indexing them would have spent the budget on duplicates while adding no points of disagreement. The API reference is where signatures, arguments and availability actually diverge.
The Purpose field did more work than I expected. It is read at every stage of the build and decides which sources survive, so my first draft — which led with "surface where the two documentations disagree" — risked producing a knowledge base full of conflicts sitting on top of no reference material. The version I shipped names the full API reference as leading content and the disagreements as a layer on top of it.
The MCP endpoint carries its own instructions, which is what keeps the distributions apart at the source rather than only in my client:
Always state which distribution a claim applies to. When the two documentations differ on the same function or behavior, present both accounts side by side with their version numbers and source, rather than picking one. Never blend them into a single answer.
The endpoint exposes two tools: initial_context, which returns the knowledge base outline, and knowledge_base_read, which reads full entries by path. The Anthropic API talks to the endpoint directly, so the model decides which entries it needs rather than my code guessing in advance. There is no retrieval logic in the repository at all — the agent is a system prompt, an endpoint and a scorer.
The agent was the easy part. Measuring it was not.
I wrote seven evaluation questions, verified each answer against both documentation sites and the pygame-ce release notes, and ran each one three times — because early on, the same question produced different claims on different runs.
First scorer, 4/7. It matched required and forbidden phrases in the prose. All seven failures turned out to be correct answers scored wrong, for two reasons. Negation: the pattern available in pygame matched "premul_alpha_ip is not available in pygame". Scope: the pattern pygame-ce only matched an answer about premul_alpha because it closed with a true aside about premul_alpha_ip. Substring matching cannot tell which claim a phrase belongs to.
Second scorer, 3/7. Worse. I stopped scoring prose and had the agent emit verdict lines instead. Negation disappears when there is nothing to negate; scope disappears when each claim carries its own name. But the model wrote Surface.premul_alpha where my case file expected premul_alpha, and my parser only stripped a pygame. prefix. Every failing verdict was correct on the facts.
Same answers, 6/7. Stripping any dotted prefix and rescoring the saved responses — no new API calls — took it to 6/7. A fresh run with the corrected prompt scored 7/7, stable across three runs each.
| Run | Score | What it actually measured |
|---|---|---|
| v1, substring | 4/7 | negation and scope handling in my regexes |
| v2, verdict lines | 3/7 | dotted-prefix handling in my parser |
| v2, parser fixed | 7/7 | the agent |
Twenty of the twenty-one answers were right before any scorer was fixed. Two rounds of evaluation measured my scorer, not the model. Both earlier runs are committed in outputs/ rather than deleted.
Seven cases is a small set, and I picked the cases knowing the answers. 7/7 means it did not fail what I tested. It does not mean it is reliable on questions nobody thought to ask.
Project ID: pdm0xx31
Knowledge base: kbJl4b52PrF6
Organization: obnospbbu