This is a submission for the Sanity Challenge, Path One: Ship an Agent That Queries Real Content.
You copy a render-setup script from a Blender 4.2 tutorial, run it in Blender 5.0, and get:
TypeError: bpy_struct: item.attr = val: enum "BLENDER_EEVEE_NEXT" not found in ('BLENDER_EEVEE', ...)
Ask a plain LLM to write that script for 5.0 and it can hand you the very same line. That is exactly what happened in my eval (case 10 below).
The bpy API breaks in almost every major release. scene.objects.link died in 2.80, the context-override dict for operators died in 4.0, the EEVEE engine identifier changed in 4.2 and changed back in 5.0, the fast boolean solver was renamed in 5.0. Old tutorials keep ranking, and a plain LLM answers from a memory where every version is blended together.
bpy-compass answers Blender Python scripting questions for the Blender version you pick, and tells you which popular pattern broke, in which version, and what replaced it. It reads a Sanity Context Knowledge Base built from the official release notes, a curated dataset of API changes and, on purpose, four stale tutorials. Every answer has three blocks:
If the Knowledge Base has no entry for part of a question, the agent marks those lines # NOT in Knowledge Base: and the UI highlights them, instead of inventing a citation.
Ask "Set the render engine to EEVEE and the boolean solver to the fast one" on 4.2 and on 5.0 and you get two different scripts (BLENDER_EEVEE_NEXT / FAST vs BLENDER_EEVEE / FLOAT), each explaining why. A keyword search over the release notes returns both identifiers with no way to tell which one applies to you.
To keep myself honest, every generated script is executed in headless Blender against an assert, for bpy-compass and for the same model without the Knowledge Base. The results, failures included, are on the live /eval page and in the table below.
Live app, no login: https://bpy-compass.vercel.app
BLENDER_EEVEE_NEXT and FAST. Switch to BLENDER_EEVEE and FLOAT.# NOT in Knowledge Base:./ chat with the version rail. Try the same question on 4.2 and 5.0./eval the headless-Blender results table, filled from Sanity evalRun documents./about how it works, plus the Knowledge Base Issues screenshots before and after resolution.
Rate limit is 10 questions per minute per IP, so the OpenRouter key survives a public demo.
https://github.com/Rustam335/bpy-compass (MIT)
app/ chat + version rail · /eval · /about · /studio · api/chat/route.ts
lib/ model.ts (pinned model/provider/temperature) · prompt.ts · context-mcp.ts · sanity.ts · rate-limit.ts
sanity/ schemaTypes/ (apiChange, blenderVersion, testCase, evalRun) · seed/*.json
scripts/ eval.ts (baseline vs bpy-compass → blender -b → evalRun docs) · seed-api-changes.ts
kb-sources/ the four stale tutorials uploaded as Knowledge Base file sources + ATTRIBUTION.md
Stack: Next.js App Router on Vercel, Vercel AI SDK with the MCP client, OpenRouter (z-ai/glm-5.3-flash, provider pinned to novita, temperature 0), Sanity Context MCP in knowledge_base mode, Sanity Studio embedded at /studio.
Three kinds of sources, all in one Knowledge Base (bpy-compass):
apiChange documents, each with symbol, kind (removed / renamed / behavior), replacement, before and after code, a reference to the blenderVersion document it changed in, and the release-note URL it was verified against. This is the part a keyword search cannot reconstruct: the scene.objects.link, obj.select = True, override dicts, calc_normals). They are there to make the build Adding the stale tutorials next to the release notes made Sanity Context raise six Critical conflicts on the Issues tab:
Two of them were plain wrong-vs-right and got a pick (for example, Mesh.calc_normals() was removed in 4.0, not "available before 4.1"). Four of them were not conflicts at all: both sides were true for different Blender versions (BLENDER_EEVEE_NEXT in 4.2 vs BLENDER_EEVEE in 5.0; solver FAST up to 4.4 vs FLOAT from 5.0). "Pick a winner" is the wrong tool for that, so I resolved those with version-scoped picks and wrote a standing Instruction:
Every bpy API fact in this knowledge base is scoped to a Blender version. Two sources that give different values for different Blender versions are NOT in conflict: record both, each labeled with its version range. When two sources disagree about the SAME version, the official Blender release notes and the apiChange dataset are ground truth over tutorials. Always keep the old form in the entry, labeled with the version it stopped working in and its replacement.
Saving the Instruction flagged a contradiction in the "Modifiers & Boolean Solvers" entry and rebuilt it. The entry now carries a solver table by version range and a version-safe selection snippet:
Final state: 14 resolved issues and 4 manual Instructions. Two of the Instructions came from the eval harness catching the Knowledge Base being wrong (more on that below).
One lesson that cost me a rebuild: stale tutorials that carry an "intentionally outdated" banner produce zero conflicts, because the build reads the banner and files them as history. Real stale tutorials have no banner, so mine do not either.
The Context MCP endpoint bpy-compass exposes the Knowledge Base only. Per request:
initial_context) and inlines it into the system prompt together with the requested Blender version, so the model knows which entry paths exist.knowledge_base_read with the paths it needs, up to five tool steps. The UI shows the paths as they are read.
The baseline used for the eval gets the exact same model, provider, temperature, system prompt, output contract and user prompt. The only difference is the Knowledge Base: the baseline has no tools, no outline and no KB rules, and is told so.
Each script was run in headless Blender, one exact build per target version (4.2.23 LTS, 4.5.14 LTS, 5.0.1) with --factory-startup, then the test case's assert script. A row passes only if the script ran and the answer kept its contract: WATCH OUT names every API change the test case expects (old symbol and replacement), and SOURCES lists only Knowledge Base entries the agent actually read. Runs are stored as evalRun documents in Sanity; /eval shows one complete run (both contenders, every case), never a mix of runs.
| # | Question (short) | Target | Plain LLM | bpy-compass |
|---|---|---|---|---|
| 1 | Create a triangle mesh object and link it to the scene | 4.5 | ✅ | ✅ |
| 2 | Deselect all, select 'Cube', make it active | 4.5 | ✅ | ✅ |
| 3 | Print world-space vertex coordinates | 4.5 | ✅ | ✅ |
| 4 | Boolean UNION with a sphere, then apply the modifier | 4.5 | ❌ script ran, but WATCH OUT never mentions the override dict → temp_override change |
✅ |
| 5 | Material 'Glow', Principled BSDF emitting orange at strength 5 | 4.5 | ✅ | ✅ |
| 6 | Geometry Nodes group with Geometry in/out and a Float 'Scale' socket | 4.5 | ✅ | ✅ |
| 7 | Export selected objects to OBJ | 4.5 | ✅ | ✅ |
| 8 | Shade smooth by angle (30°) | 4.5 | ❌ 'Mesh' object has no attribute 'use_auto_smooth' (wrote it anyway, with a comment saying it was removed) |
✅ |
| 9 | EEVEE engine + 640×480 resolution | 4.2 | ✅ | ✅ |
| 10 | EEVEE engine + 640×480 resolution | 5.0 | ❌ enum "BLENDER_EEVEE_NEXT" not found |
✅ |
| 11 | Boolean DIFFERENCE with the fast solver | 4.5 | ✅ | ✅ |
| 12 | Boolean DIFFERENCE with the fast solver | 5.0 | ❌ enum "FAST" not found in ('FLOAT', 'EXACT', 'MANIFOLD') |
❌ knew FAST →FLOAT , but created the cutter withobjects.new without linking it, thenselect_set |
| Pass rate | 8/12 | 11/12 |
What the numbers say, honestly:
@ matrix multiply, temp_override, node-group sockets, the new OBJ exporter: the plain model's scripts get them right (its case 4 script runs; it only failed to modifiers.new(..., 'SMOOTH_BY_ANGLE'), a modifier type that does not exist. The agent's own WATCH OUT had flagged that two entries disagreed. The fix was an Instruction (the real replacement for use_auto_smooth is bpy.ops.object.shade_smooth_by_angle), a rebuild of two entries, and a rerun. The failed run is still in Sanity.blender --version before the first LLM call. The rest covered the eval (WATCH OUT/SOURCES were never checked, baseline and compass got different prompts, --only could silently run everything), the API route (message validation, the model-config guard only ran in eval, a rate limiter that never forgot an IP) and an inconsistent grounding policy. Every issue is fixed and referenced in the commits; the numbers above are from the rerun after those fixes.useCdn: false).
The apiChange dataset source is what makes "which version" answerable. A release-notes page says a symbol was removed; it does not say what the replacement is for a user on 4.2 versus 5.0, and it does not know that a 2.7x tutorial still ranks on Google. The dataset ties symbol, version and replacement together, the Knowledge Base build merges it with the release notes and detects where the stale tutorials disagree, and the Instructions encode the one rule a human had to decide: different versions are not a conflict, same version is, and release notes win.
uuc8lnyk · dataset production (public read)bpy-compass (kbMlSqMSn4Q9) · testCase (12), Agent session transcript: bpy-compass build session: Sanity Context and Knowledge Base builds
It's the main Claude Code build session, cut to the Sanity part (382 of 669 messages): enabling Context, the three Knowledge Base builds (0 → 2 → 6 conflicts), resolving the Issues and writing the Instructions, and wiring the Context MCP endpoint into the chat route. I scanned it for keys and tokens before up; the up also redacted home-directory paths.
Disclosure, set in the DEV editor: AI-Assisted. I used Claude Code to build the app and draft this post, and I'm publishing it under my name.