{"slug": "mcp-server-for-mario-paint-it-draws-and-composes-inside-the-real-snes-game", "title": "MCP server for Mario Paint. It draws and composes inside the real SNES game", "summary": "A developer released MCPaint, an MCP server that drives the 1992 SNES game Mario Paint headlessly through a libretro core and injects content directly into the console's WRAM to draw pictures and compose music, capturing output as PNG, WAV and MP4. The project requires Go 1.24+, ffmpeg on PATH, a user-supplied Mario Paint ROM and a libretro SNES core such as snes9x_libretro.dylib, and exposes tools including draw_image, draw, compose, import_midi, play and record_session. Image quantization uses 16 fixed colors matched in Oklab, with optional dithering, a -vivid hue-first mode and -map pinning for colors with no palette neighbor.", "body_md": "An MCP server that lets an AI create pictures and music with **Mario Paint**\n(SNES, 1992) — the real game, running headless — and capture the result as PNG,\nWAV and MP4.\n\nA libretro core runs the game with no window at all. Content is injected straight into the console's WRAM at the addresses the game itself uses, and the game renders it exactly as if it had been drawn by hand.\n\nDrawing is played back step by step with the in-game cursor following the\nstroke, so a recording shows Mario Paint *drawing the picture* rather than a\npicture pasted into Mario Paint. It comes with sound: the canvas background\ntrack plays over the timelapse, and a recording opens on the tune's first\nnote.\n\nAll audio is the SPC700's own output, captured from the same run that produced the frames, so picture and sound need no resynchronising.\n\nEvery example is rebuilt from its source by [`examples/make.sh`](https://github.com/TheLiux/MCPaint/blob/master/examples/make.sh).\n\n| Source | In Mario Paint | Drawn | \n|---|---|---|\n\nThe fractal is a photograph as far as the palette is concerned: dithered,\nmatched on lightness. The robot is flat pixel art, drawn with dithering off and\nhue matched first, so every block stays a single colour.\n[`testdata/scene.json`](https://github.com/TheLiux/MCPaint/blob/master/testdata/scene.json) draws a scene out of the game's own\ntools instead:\n\n[Ode to Joy](https://github.com/TheLiux/MCPaint/blob/master/examples/ode/ode.mid), written in D major with a bass line, comes\nout on the staff in C across two pages: [watch it play](https://github.com/TheLiux/MCPaint/blob/master/examples/ode/ode.mp4)\nor [just listen](https://github.com/TheLiux/MCPaint/blob/master/examples/ode/ode.mp3).\n\nThe [guide](https://github.com/TheLiux/MCPaint/blob/master/docs/GUIDE.md) covers doing the same with your own pictures and\nsongs: framing, colour flags, choosing a key, tempo and instruments.\n\n- A Mario Paint ROM. **Not included** — supply your own copy.\n- A libretro SNES core, e.g. `snes9x_libretro.dylib` .\n- `ffmpeg` on PATH, for video and audio output.\n- Go 1.24+.\n\n```\nexport MCPAINT_CORE=\"$HOME/Library/Application Support/RetroArch/cores/snes9x_libretro.dylib\"\nexport MCPAINT_ROM=\"/path/to/Mario Paint (JU).smc\"\nexport MCPAINT_OUT=\"$HOME/mcpaint\"   # optional, this is the default\n\ngo build -o build/ ./cmd/...   # mcpaint, mcpaint-cli, mcpaint-midi, mcpaint-song\n```\n\nRegister it with an MCP client:\n\n```\n{\n  \"mcpServers\": {\n    \"mcpaint\": {\n      \"command\": \"/path/to/build/mcpaint\",\n      \"env\": {\n        \"MCPAINT_CORE\": \"/path/to/snes9x_libretro.dylib\",\n        \"MCPAINT_ROM\": \"/path/to/Mario Paint (JU).smc\"\n      }\n    }\n  }\n}\n```\n\n| Tool | What it does | \n|---|---|\n| `reference` | The palette, instruments, note range and hard limits | \n| `draw_image` | Redraw an existing image, quantized to 16 colours | \n| `draw` | Draw with the game's own tools, optionally recording a timelapse with sound | \n| `screenshot` | Capture the canvas or the whole screen | \n| `compose` | Write notes into the music composer | \n| `import_midi` | Load a MIDI file, with a report of what had to give | \n| `play` | Play the song and record audio, optionally with video | \n| `record_session` | One clip: the picture being drawn, then the song playing | \n\nSixteen fixed colours with no blending, so how a source image is matched to them matters more than usual.\n\n- Matching is done in **Oklab** . Luminance-weighted distance collapsed\nsaturated colours onto grey, because the weighting leans so heavily on the\ngreen channel.\n- **Dithering** makes a photograph read as one and makes flat artwork read as\nnoise. Leave it on for photos, off for logos.\n- **Vivid** (`-vivid` ) weighs hue ahead of lightness, which suits flat art.\n- Some colours have no honest neighbour at all. Google's amber yellow sits\nbetween the palette's lemon and its olive, so every weighting lands on peach\nor olive. **Pinning** (`-map \"#F4B400=yellow\"` ) names the substitution\ninstead, and pinned pixels take no dither error so flat areas stay flat.\n\nDark photographs are worth lifting before they are quantized -- the palette\nhas no dark greys, only black -- for example with\n`ffmpeg -i in.png -vf \"eq=brightness=0.18:contrast=1.6:saturation=1.8\" out.png`.\n\nThe canvas has its own background track, and `draw` and `record_session` take a\n`music` option: `theme-1` (default), `theme-2`, `your-song` — whatever is\ncurrently in the composer — or `off`.\n\nA recording always starts on the tune's first note. Getting there is fiddlier than it looks: re-picking a track does not rewind it, because the sequencer keeps running underneath. Switching to silence stops it and switching back starts the tune again, about 36 frames later, while leaving the SELECT MUSIC screen takes roughly 32 — so the canvas is up just before the music begins. The prepared state is then nudged forward to the last silent frame, past the screen-transition sound effect, and cached.\n\nThe limits are tight and shape everything:\n\n- **Canvas** 248×168, 16 fixed colours, no blending.\n- **Songs** up to 96 columns holding at most three notes each.\n- **Pitches** are 13 diatonic staff positions, B3 to G5. There are no sharps\nor flats, so imported music gets snapped.\n\n`cmd/mcpaint-cli` drives the game directly, without the MCP server.\n\n```\n# Redraw an image and screenshot the result\nbuild/mcpaint-cli -image picture.png -out out/canvas.png\n\n# Record it being drawn stroke by stroke, from boot, with music\nbuild/mcpaint-cli -image picture.png -strokes -title 3 \\\n    -video out/picture.mp4 -out out/picture.png -seconds 20\n\n# Flat artwork: no dithering, hue-first matching, brand colours pinned\nbuild/mcpaint-cli -image logo.png -fit contain -dither=false -vivid \\\n    -map \"#4285F4=blue,#F4B400=yellow\" -out out/logo.png\n\n# Play a set of drawing operations\nbuild/mcpaint-cli -ops testdata/scene.json -video out/scene.mp4 -full\n```\n\n`cmd/mcpaint-song` turns a chord chart into a composition and records it. A staff\nholds 96 columns and a single tempo, so the chart is laid out across pages that\nbreak wherever the tempo changes -- which is how a chorus can move faster than\nthe verse around it.\n\n`cmd/mcpaint-midi` plays a MIDI file. The staff holds 96 columns, so a longer piece\nis split across pages: each is loaded in turn, recorded, and the recordings are\nstitched back together. Every page is cut to its exact length -- when the first\ncolumn sounds and how long a column lasts are measured from the game at the\nchosen tempo -- so a rest that falls across a page break survives the seam.\n\n```\n# See how a piece fits before starting the emulator\nbuild/mcpaint-midi -midi song.mid -dry -auto-key\n\n# Play it, however many staves it takes\nbuild/mcpaint-midi -midi song.mid -auto-key -tempo 24 \\\n    -instruments \"0=star,1=gameboy,2=mario\" -out out/song.mp4\n```\n\n`-auto-key` is worth reaching for. The staff is strictly diatonic C major, so a\ntune in another key has every accidental pulled to a neighbour; shifting the\nwhole piece can put it in a key the staff actually has. On a Sicilian folk tune\nit took the accidentals from 86 down to 19.\n\nInstruments are chosen by register: channels are ranked by median pitch and\nhanded voices spanning the palette from darkest to brightest, so a bass line\ndoes not land on the same bright voice as the melody. That ordering was\nmeasured, not guessed from the icons -- each instrument was played and the\nspectral centroid of its attack taken, running from about 320 Hz to 2250 Hz.\n`internal/session` has the test that measures it.\n\nThree limits bite, and each has a different answer.\n\n**Range.** Thirteen diatonic positions, B3 to G5, is under two octaves, so a\npiece spanning more has to fold. A note the staff can hold stays exactly where\nit was written. One that cannot goes to the octave nearest where its own part\nalready is, rather than wherever the arithmetic lands, so a rising line does\nnot jump down mid-phrase and a bass does not end up above the melody. Applying\nthat nearness to every note is a trap: it folds any leap wider than half an\noctave back into a step, and on one march it moved 162 of 413 notes when only\n3 were out of range.\n\n**Three voices per column.** Most of what looks like a shortage is not one:\narrangements double notes at the octave, and once the staff has folded them\nthey collapse onto the same position. Playing that twice wastes a voice on a\nunison. Folding the doublings away recovered nearly everything on the test\npiece -- dropped voices went from 50 to 12. What is still too thick keeps the\nbass and the melody, which carry the outline, plus a note from the middle where\nthe chord's character lives; the rest spreads onto the next column, the way a\nplayer would roll a chord by hand. One note in 394 was actually lost.\n\n**Percussion.** General MIDI reserves channel 10 for drums, where the key\nnumber picks a kit piece rather than a pitch. Reading those as notes puts\nnonsense on the staff and poisons key detection along with it: on one\narrangement dropping the drums took the accidentals from 2157 to 755 and\nchanged which key `-auto-key` chose. `-drums auto` keeps the pattern instead,\nlaid on three staff positions standing for low, middle and high drums.\n\n**Too many parts.** Three voices cannot hold a seven-part arrangement.\n`-channels \"5,1,9\"` keeps the ones that matter -- typically a melody, a bass\nand the drums -- which beats letting an arbitrary rule decide.\n\n**No accidentals.** Nothing to be done: the staff has no black keys, so sharps\nand flats are pulled to a neighbour. `-auto-key` minimises how often that\nhappens, but it cannot reach zero for a tune that really does change key.\n\n```\ngo test ./...\n```\n\nThe unit tests in `internal/mp` need neither a ROM nor a core, which is where\nmost of the logic lives. The end-to-end test in `cmd/mcpaint` drives the\nreal server over stdio and skips unless `MCPAINT_CORE` and `MCPAINT_ROM`\nare set.\n\nMario Paint's title screen has no clickable \"start\": a full grid sweep found\nonly the letter gags. The attract demo walks into the canvas by itself, so the\nboot rides it in and then clears the demo flag at `$7E:04E2` to take over. That\ncosts ~2400 emulated frames, so the resulting state is cached and reused.\n\nTwo things that look like they should work but do not:\n\n- `$7E:0C24` is named \"scroll end\" and is not a song length. It sits at 784\nwhatever is on the staff, and writing a column count there cuts playback off\nafter the first note.\n- Pressing PLAY twice in a row plays nothing the second time. The playhead stays at the end of the previous run, so STOP has to rewind it first.\n- Video has to be encoded at the console's own frame rate, 60.0988, not a round 30 or 60. Anything else and the picture drifts away from the sound.\n\nMemory addresses come from the labelled RAM map in\n[Yoshifanatic1/Mario-Paint-Disassembly](https://github.com/Yoshifanatic1/Mario-Paint-Disassembly)\nand were each confirmed against the running game. The palette is decoded from\nthe game's own `Canvas.bin` rather than sampled from screenshots.", "url": "https://wpnews.pro/news/mcp-server-for-mario-paint-it-draws-and-composes-inside-the-real-snes-game", "canonical_source": "https://github.com/TheLiux/MCPaint", "published_at": "2026-09-24 08:18:47+00:00", "updated_at": "2026-09-24 09:01:57.687359+00:00", "lang": "en", "topics": ["ai-agents", "agent-protocols", "ai-tools", "generative-ai", "developer-tools"], "entities": ["MCPaint", "Mario Paint", "SNES", "libretro", "snes9x_libretro.dylib", "ffmpeg", "Go", "Oklab"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/mcp-server-for-mario-paint-it-draws-and-composes-inside-the-real-snes-game", "markdown": "https://wpnews.pro/news/mcp-server-for-mario-paint-it-draws-and-composes-inside-the-real-snes-game.md", "text": "https://wpnews.pro/news/mcp-server-for-mario-paint-it-draws-and-composes-inside-the-real-snes-game.txt", "jsonld": "https://wpnews.pro/news/mcp-server-for-mario-paint-it-draws-and-composes-inside-the-real-snes-game.jsonld"}}