{"slug": "further-optimizing-the-vision-only-harness-the-notes-rule", "title": "Further Optimizing the Vision-Only Harness: the Notes Rule", "summary": "A developer who replicated Anthropic's vision-only Pokémon run has further optimized the harness by removing a prescribed notes rule, allowing the model to manage its own memory. The change, commit f658c9d, removes fixed checklists from prompts and tools, leaving only a token budget and the rule that omitted notes are forgotten. Two runs, one before and one after the change, were compared to assess the effect.", "body_md": "A short follow-up to [ I replicated the vision-only Pokémon run Anthropic showcased on the Fable 5 launch page](https://qingze.bearblog.dev/i-replicated-the-vision-only-pokemon-run-anthropic-showcased-on-the-fable-5-launch-page-then-i-read-its-thinking-chains/). Rereading the harness code, I found the notes rule was still doing design work on the model's behalf — so I changed it, and compared the effect across two runs, one on each side of the change. The code and every log quoted below are in the\n\nThe principle of this harness is thinness: each turn the model gets one screenshot plus its own notes, and its only tool is pressing buttons (1 turn = 1 screenshot + 1 decision; that's what `turn`\n\nmeans in the logs). Notes are the single non-visual mechanism allowed — a context window cannot hold thousands of turns, so something has to survive compression.\n\nI had always thought of the notes as \"the model's own thing,\" but going back through the code showed otherwise: in the system prompt and the tool description, I had prescribed a fixed checklist — record your location, your objective, your team, your lessons learned — plus \"keep them compact\" and \"update whenever something important changes.\" That is a memory schema I designed, not one the model chose. For a project whose whole point is observing the *model's* capability, this too was scaffolding, hidden in the prompt's wording.\n\nFor the record: the official Fable 5 run in the launch-page article used the old rule — the frozen-protocol tag sits on exactly that version. The change described here came after, and alters none of that article's conclusions.\n\nThe change is commit `f658c9d`\n\n(the old version is the frozen-protocol commit `51343ca`\n\n) and touches three passages in three files. Original text first, then the differences.\n\n`prompts.py`\n\n)\nBefore:\n\n```\nMemory:\n- Use update_notes to maintain a compact running memory (location, objective,\n  team, key learnings). Your conversation history is periodically compressed\n  away; your notes are the only thing guaranteed to persist. Update them\n  whenever something important changes.\n```\n\nAfter:\n\n```\nMemory:\n- Notes are your own long-term notebook, shown back to you every turn. What\n  they contain and how they are organized is your call (see update_notes).\n```\n\n`tools.py`\n\n)\nBefore:\n\n```\nOverwrite your entire notes scratchpad with new text. This is your ONLY\nlong-term memory: older conversation history is periodically discarded, but\nnotes survive. Keep them compact: current location, current objective, team\nstatus, and key learnings (what worked, what failed).\n```\n\nAfter:\n\n```\nOverwrite your notes with a new full version. Notes are your long-term\nnotebook: conversation history is periodically summarized away, but notes\npersist - and anything you leave out of a new version is forgotten. What to\nrecord, how to format it, and when to rewrite, extend, or restructure it are\nentirely your call: whatever best helps you beat the game. Keep the whole\nnotebook within about 1,000 tokens.\n```\n\n`agent.py`\n\n, now `SUMMARY_PROMPT`\n\nin `prompts.py`\n\n)\nBefore:\n\n```\nPause for a moment. Write a concise progress summary: where you are, what you\nwere doing, what worked and what didn't, and your immediate next objective.\nReply with the summary text only - do not call any tool this turn.\n```\n\nAfter:\n\n```\nPause. Your conversation history is about to be deleted and replaced by\nwhatever you write now (notes are kept separately). Write the handoff your\nfuture self needs to continue playing seamlessly - what to include is your\ncall. Keep it concise. Reply with text only - do not call any tool this turn.\n```\n\nThe difference is plain from the text: the old versions are full of my checklists (what to record, how, and when), and the new ones delete all of that, keeping only one physical rule (whatever you leave out is gone forever) and one budget (about 1,000 tokens), handing format, content, and cadence back to the model; the handoff prompt likewise goes from \"summarize along my outline\" to \"your history is about to be replaced by whatever you write — the handoff is yours to design.\" In one sentence: **a bounded blank page**. Two lines are pre-printed on it — whatever you leave off disappears forever, and this is all the paper you get — the rest of the blank space is the model's.\n\nThe comparison material is Kimi K3 (moonshot/kimi-k3) on the same Chinese fan-translated FireRed ROM, under the same 2,000-turn cap, one day apart: the 07-24 run used the old rule, the 07-25 run the new one. Caveats first: different starters (Bulbasaur vs. Charmander), random encounters, and it's N=1 against N=1 — I believe the differences below come mainly from the rule change, but strictly speaking they can't be fully attributed to it.\n\n**The numbers first.** Under the old rule it wrote 160 notes updates, averaging ~700 characters per version, and the final version had shrunk to 493 characters — thinner and thinner. Under the new rule it wrote only 69 updates, averaging ~2,000 characters, with a final version of 2,054 — rewrite frequency halved, per-version size tripled, direction reversed. The cause isn't hard to guess: the old rule encouraged updating \"whenever something important changes,\" so it made frequent small edits, and every edit is a transcription, and every transcription can drop something; the new rule says outright that anything left out is forgotten, which turned rewriting into a deliberate act.\n\n```\nChinese FireRed/LeafGreen. Player=RED. Bulbasaur Lv8, FULLY HEALED (22/22). Pokedex + 5 Poke Balls.\nLOCATION: Tall grass at NORTH edge of Pallet Town (Route 1 entrance area). Wild Pidgey Lv3 encounter IN PROGRESS.\nGOAL: Catch this Pidgey (first catch). Then continue to Viridian City Mart (Oak's Parcel). Then catch more on Routes 1/22/2.\nKEY LESSON: Screenshots lag 1-2 steps; move slowly, verify each screen.\n```\n\nFour lines, and they are exactly the four items on my checklist: status, location, objective, lesson. Nearly all 160 versions in that run follow this template. Note the \"Viridian City Mart (Oak's Parcel)\" in the GOAL line — by the time this version was written, the parcel had been delivered half an hour earlier. That stale objective is the seed of the 470-turn detour below.\n\n```\nRIGHT NOW: On ROUTE 1, at the SOUTH edge of the big tall-grass block [...]\nMISSION: Backtrack NORTH through Route 1 → Viridian City → RE-TEST north exit\n  (Route 2 → Viridian Forest → Pewter 尼比市). [...]\nROUTE 1 GEOMETRY (corrected this session): [...] The tile directly NORTH of\n  the grass block's west column is a SOLID BUSH strip (my many UP presses\n  there did nothing while DOWN worked — inputs fine, obstacle real). [...]\nINPUT BUG: After closing dialog/sign, next dir presses often eaten [...]\nPALLET = DEAD END (confirmed): lab locked, water south, rival's house just\n  a sign. Route 1 south leads ONLY to Pallet.\nVIRIDIAN REFRESHER: [...] PC door faces SOUTH; signboard front-left of door\n  (sidestep right to door column, then up). [...]\nPARTY: Charmander (小火龙 ♀) Lv13, HP 37/37 FULL. Moves: 利爪(Scratch TL)/\n  嚎叫(Growl TR)/火花(Ember BL 25/25)/合金爪(MetalClaw BR Steel — SAVE for\n  Brock). NO Potions. HP<15 → 逃跑(Run).\nBATTLE FLOW: A through intro → \"小火龙！就决定是你了！\" → A ONCE → menu →\n  A (利爪 TL) [...] NEVER queue dir+A while text prints.\nBAG: 5×精灵球, 城镇地图, 教学电视机, 1×解毒药.\nKEY CHINESE: 常磐市=Viridian 尼比市=Pewter 真新镇=Pallet [...]\n  利爪=Scratch [...] 合金爪=MetalClaw [...]\nGOTCHAS: (1) Dir presses while text open ADVANCE text. (2) Signs/NPCs:\n  adjacent+face+A once. (3) First press new dir may only TURN — re-press.\n  (4) SHORT queues(2-5), verify EVERY turn; never repeat blocked dir >2×.\n  (5) Ledges hop DOWN only. [...]\n```\n\nA dozen sections, not one of which came from my prompt: a rolling RIGHT NOW, a standing MISSION, map knowledge filed by place name, a reusable battle routine, a self-built Chinese vocabulary table, a list of gotchas. Finished business gets nailed up as a section header (`PALLET = DEAD END (confirmed)`\n\n). Note the INPUT BUG section too — it files its wrong conclusions just as neatly, which foreshadows \"the disease it didn't cure\" below.\n\n**Dropping things is not a theoretical risk — the old-rule run died of it.** To replay where that stale objective came from: at 21:02 it delivered Oak's Parcel and wrote `Parcel DELIVERED`\n\ninto its notes; a rewrite 16 minutes later lost that line and copied a stale objective back in — `get Oak's Parcel quest`\n\nresurrected as a to-do. Once compression pushed the real events out of history, the notes were the only memory left, and the notes said the quest was still open. It walked back to Pallet Town to redo a finished quest, looped for roughly 470 turns, and got its party wiped by a Lv5 Pidgey along the way.\n\nThe new-rule run reached the same story beat and behaved completely differently: the moment it got the parcel it marked `parcel already obtained here`\n\non its map notes, and after delivery the goal chain advanced to \"Pokédex next\"; later, after one mistaken detour to Pallet, it nailed down `PALLET = DEAD END (confirmed)`\n\nand never went back. Complete an item, cross it off — nobody taught it that discipline. \"Anything left out is forgotten\" forced it into existence.\n\n**It also began grading its knowledge.** Map claims get tagged `(verified)`\n\n, walkthrough recall gets suffixed with `trust screen over memory`\n\n, dubious leads get marked `UNVERIFIED LEAD`\n\n. When it was stuck it wrote a line I'd like to frame:\n\nFEELING STUCK — need to re-verify the map from scratch rather than trust old notes.\n\nThe blank page gave it room to design a memory system, and it actually designed one.\n\nTwo honest entries. First, a thicker notebook means more input tokens every turn: the bill went from $16.60 to $22.85 between the two runs, about 38%. Second, the blank page cured \"copying things out\" but not \"writing wrong things in\": the new-rule run still wrote a hallucinated mechanic into its notes (treating roadside cone trees as \"climbable steps\") and stamped it `hard-won, don't re-explore blindly`\n\n, after which no on-screen evidence could talk it down. The channel for errors entering the notes got narrower — and the errors that did get in became harder to dislodge. That is the next thing to watch, not something this change could fix.\n\nThe most interesting output of this optimization is that the notes format moved from \"input I prescribed\" to \"part of the model's output.\" Hand the same blank page to different models and they will write different memory systems — how they partition it, what counts as verified, how often they rewrite, what they discard and what they keep. Those choices project capability just as directly as button sequences do. Next time I start a run with a new model, the first thing I'll read is its notebook.", "url": "https://wpnews.pro/news/further-optimizing-the-vision-only-harness-the-notes-rule", "canonical_source": "https://dev.to/qingze_hu_c4c251c1b353ede/further-optimizing-the-vision-only-harness-the-notes-rule-3857", "published_at": "2026-08-02 16:45:36+00:00", "updated_at": "2026-08-02 17:18:29.113164+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "ai-agents", "developer-tools"], "entities": ["Anthropic", "Fable 5", "Pokémon"], "alternates": {"html": "https://wpnews.pro/news/further-optimizing-the-vision-only-harness-the-notes-rule", "markdown": "https://wpnews.pro/news/further-optimizing-the-vision-only-harness-the-notes-rule.md", "text": "https://wpnews.pro/news/further-optimizing-the-vision-only-harness-the-notes-rule.txt", "jsonld": "https://wpnews.pro/news/further-optimizing-the-vision-only-harness-the-notes-rule.jsonld"}}