{"slug": "show-hn-pigame-pi-llm-plays-black-box-browser-games-via-observe-move-tools", "title": "Show HN: Pigame – pi+LLM plays black-box browser games via observe/move tools", "summary": "Developer ifoxhz released Pigame, an open-source game agent that combines Pi.dev's agent platform with an LLM such as DeepSeek to play black-box browser games through observe and move tools, with the first demo being Neon Snake. The project, codenamed GameMind, provides scaffolding where the LLM acts as the brain and the tools serve as eyes, hands, and feet, and it includes a patched version of the snake game that pauses between moves to prevent stale decisions. Pigame is available on GitHub and can be installed via Pi CLI, with smoke tests for CI and support for visible Chromium via WSLg.", "body_md": "**English** | [中文](/ifoxhz/pigame/blob/main/README.zh-CN.md)\n\nGeneral **game agent** on [Pi](https://pi.dev) + LLM (e.g. DeepSeek).\n\nDesign codename: **GameMind**. First demo game: **Neon Snake**.\n\nRemote: `git@github.com:ifoxhz/pigame.git`\n\nWelcome.Contributions of any kind and any ideas are welcome — let’s make the Pi agent more fun.\n\n```\nPi + LLM  = brain (strategy / when to stop)\npigame    = scaffolding only\n  tools   = eyes / hands / feet\n```\n\n| Layer | Role |\n|---|---|\nPi + LLM |\nDecide what to do; judge when the goal is done |\n`game_observe` |\nEyes — see the screen → AgentState |\n`game_move` / `game_wait` |\nHands & feet — act / wait |\n`npm run smoke:*` / `/game eat` |\nTest scaffolding only — hardcoded loops for CI; not the agent path |\n\nThere is **no** `game_eat_food`\n\n/ `game_survive`\n\ntool. The model loops observe → move itself.\n\n| Tool | Notes |\n|---|---|\nNode.js |\n≥ 20 recommended (`--experimental-strip-types` ) |\nnpm |\nComes with Node |\nPi CLI |\nRequired for the agent path (`pi` on PATH) |\nLLM in Pi |\ne.g. DeepSeek configured in your Pi settings |\nWSLg / display |\nOnly for visible Chromium (`HEADED=1` ) |\n\n```\ngit clone git@github.com:ifoxhz/pigame.git\ncd pigame   # or your local folder name\nnpm install --prefix pigame\nnpx --prefix pigame playwright install chromium\n```\n\n**Default game (no extra clone):** `pigame/games/snake.html`\n\n— a vendored Neon Snake with **agent mode** (see below).\n\nAnyone who clones this repo gets the patched game automatically; Adapter loads it on `/game connect`\n\n.\n\nOnly if you want the original continuous-loop game for comparison:\n\n```\ngit clone git@github.com:Digiman/mini-game-studio-ai-gen.git\n# HTTPS: https://github.com/Digiman/mini-game-studio-ai-gen.git\nSNAKE_HTML=./mini-game-studio-ai-gen/games/snake.html HEADED=1 pi -e ./.pi/extensions/pigame.ts\n```\n\nLive site (reference): [https://digiman.github.io/mini-game-studio-ai-gen/](https://digiman.github.io/mini-game-studio-ai-gen/)\n\nNo separate compile step for smokes: Node strip-types runs TypeScript directly.\n\nRoot `package.json`\n\nonly **proxies** scripts into `pigame/`\n\n.\n\n```\ngit clone git@github.com:ifoxhz/pigame.git\ncd pigame   # or your local folder name\n\nnpm install --prefix pigame\nnpx --prefix pigame playwright install chromium\ncd /path/to/pigame\nHEADED=1 pi -e ./.pi/extensions/pigame.ts\n```\n\nIn the Pi TUI (in order):\n\n```\n/game connect\n/skill:eat-food\n```\n\nOr paste this prompt instead of the skill:\n\n```\nUse only game_observe, game_move, and optionally game_wait.\nPrefer groundTruth grid coords. Eat one food: observe → move → repeat.\nWhen score increases, stop and report before/after score.\nDo not reverse 180°.\n```\n\nWatch the **Playwright** Chromium window (snake freezes between moves).\n\nWhen done:\n\n```\n/game disconnect\n```\n\nAfter code changes in a live session: `/reload`\n\n, then `/game disconnect`\n\n→ `/game connect`\n\nagain.\n\n```\n# Adapter + perception + I/O only (no Pi / no LLM)\nnpm run smoke:eat\nnpm run smoke:eat:headed\nnpm run pack:pi\n```\n\nProduces:\n\n`dist/pi-pigame/`\n\n— installable directory`dist/pigame-0.1.0.tgz`\n\n— npm tarball\n\nOthers install into Pi with:\n\n```\npi install /path/to/pigame/dist/pi-pigame\n# or\npi install /path/to/pigame/dist/pigame-0.1.0.tgz\n\n# after you publish to npm:\n# npm publish ./dist/pigame-0.1.0.tgz\n# pi install npm:pigame\n```\n\nFrom a git clone (without packing), they can also:\n\n```\npi install ./pigame\n```\n\nThen restart Pi or `/reload`\n\n. First time: `npx playwright install chromium`\n\n(from the installed package dir if needed).\n\n**Problem:** Upstream Snake ticks on its own. While the LLM is in a long observe → think → move chain, the snake has already moved, so decisions are stale and play feels clumsy.\n\n**What we ship:** `pigame/games/snake.html`\n\n(see [pigame/games/README.md](/ifoxhz/pigame/blob/main/pigame/games/README.md))\n\n| Behavior | Meaning |\n|---|---|\nAuto `gameLoop` |\nPaused after connect (`setAgentMode(true)` ) |\n`game_observe` |\nWorld stays frozen (no time pass) |\n`game_move` |\n`setDir` + exactly 1 grid step (`MOVE_STEPS` to change) |\n\nNo special startup flag is required for agent mode — connect uses the vendored file by default.\n\n| Variable | Default | Meaning |\n|---|---|---|\n`SNAKE_HTML` |\n`pigame/games/snake.html` |\nOverride HTML path |\n`MOVE_STEPS` |\n`1` |\nGrid steps per `game_move` |\n`HEADED` |\nunset | `1` → show Chromium |\n\nThis is the intended product flow: **you open Pi; the LLM uses tools**.\n\n```\ncd /path/to/pigame\nHEADED=1 pi -e ./.pi/extensions/pigame.ts\n# or: pi   (auto-loads .pi/extensions/pigame.ts if the project is trusted)\n```\n\nAfter code changes: `/reload`\n\n.\n\nYou should see something like: `GameMind ready — LLM tools: game_observe, game_move, game_wait`\n\n.\n\nIn Pi:\n\n```\n/game connect\n```\n\nPlaywright opens Chromium and loads ** pigame/games/snake.html** (agent-mode), then Start.\n\nWorld is frozen until each\n\n`game_move`\n\n.**Do not** open\n\n`snake.html`\n\nin your own browser — only the Playwright window is controlled.Optional: `/game observe`\n\nto sanity-check AgentState.\n\nExample (DeepSeek / any model in Pi):\n\nUse only\n\n`game_observe`\n\n,`game_move`\n\n, and optionally`game_wait`\n\n.\n\nEat one food: observe → choose a direction → move → repeat.\n\nWhen`score`\n\nincreases, stop and report before/after score.\n\nDo not reverse 180° from the previous move.\n\nOr load the skill: `/skill:eat-food`\n\n.\n\n```\ngame_observe  →  read player / food / score\ngame_move     →  one step toward food\ngame_observe  →  …\n… until the model decides the goal is done …\n```\n\n| Command | Action |\n|---|---|\n`/game` |\nStatus + help |\n`/game connect` |\nStart Playwright Neon Snake |\n`/game observe` |\nCompact AgentState (manual check) |\n`/game state` |\nFull AgentState JSON |\n`/game reset` |\nRestart round |\n`/game disconnect` |\nClose browser |\n`/game mock` |\nSwitch to mock adapter |\n`/game eat` |\nTest scaffolding — scripted auto-eat |\n`/game survive` |\nTest scaffolding — scripted play until death |\n\n| Tool | Role |\n|---|---|\n`game_observe` |\nEyes |\n`game_move` |\nHands / feet (`up` / `down` / `left` / `right` ) |\n`game_wait` |\nWait |\n\nThese commands **do not** use Pi or an LLM. They run fixed policies (`eatFoodLoop`\n\n/ `surviveUntilEnd`\n\n) to verify Adapter + Perception + I/O. Use them for CI and debugging — **not** as the agent design.\n\n```\n# Eat one food (score 0 → 10), headless\nnpm run smoke:eat\n\n# Same with visible Chromium (WSLg)\nnpm run smoke:eat:headed\n\n# Long run until death (or stop at a score)\nnpm run smoke:survive:headed\nMAX_SCORE=50 npm run smoke:survive:headed\n```\n\n| Command | What it checks (scaffolding) |\n|---|---|\n`npm run smoke` |\nMock adapter / AgentState pipeline |\n`npm run smoke:snake` |\nPlaywright open + Start + key + screenshot |\n`npm run smoke:perception` |\nCV fixture + live perception |\n`npm run smoke:eat` |\nScripted steer until score↑ |\n`npm run smoke:survive` |\nScripted loop until game over (or `MAX_SCORE` ) |\n`npm run smoke:*:headed` |\nSame with `HEADED=1` |\n\n| Variable | Default | Meaning |\n|---|---|---|\n`HEADED` |\nunset | `1` / `true` → show Chromium |\n`SNAKE_HTML` |\nvendored agent snake | Path override |\n`MOVE_STEPS` |\n`1` |\nSteps per move (agent mode) |\n`MAX_SCORE` |\nnone | Survive stops when score ≥ value |\n`MAX_STEPS` |\n`50000` |\nSurvive step cap |\n`STEP_MS` |\n`200` |\nVisual delay between smoke moves |\n`HOLD_MS` |\n`5000` –`8000` |\nKeep window open after headed runs |\n\nNeon Snake has **no clear** — only death; `MAX_SCORE`\n\nis an artificial stop for tests.\n\n```\n.\n├── README.md / README.zh-CN.md\n├── package.json                 # script proxies\n├── docs/                        # design & schema\n├── pigame/                      # main package\n│   ├── games/snake.html         # DEFAULT agent-mode Neon Snake (vendored)\n│   ├── games/README.md\n│   ├── extensions/pigame.ts\n│   ├── skills/\n│   ├── scripts/                 # smoke:* (test scaffolding)\n│   └── src/\n├── .pi/extensions/pigame.ts     # Pi entry\n└── mini-game-studio-ai-gen/     # optional upstream clone (SNAKE_HTML=…)\n```\n\nDesign docs: [docs/README.md](/ifoxhz/pigame/blob/main/docs/README.md).\n\n| Issue | Fix |\n|---|---|\nMissing `pigame/games/snake.html` |\nPull latest pigame — file is in-repo |\n| Want upstream continuous loop | `SNAKE_HTML=./mini-game-studio-ai-gen/games/snake.html` (clone Digiman first) |\n| Snake runs by itself while LLM thinks | You are not on agent-mode HTML; check default path / `SNAKE_HTML` |\n`ENOENT package.json` at root |\nUse root scripts or `cd pigame` |\n| No Chromium window | `HEADED=1` or `*:headed` ; check WSLg / `echo $DISPLAY` |\n| Opened Snake yourself, nothing happens | Only Playwright’s window is controlled |\n| Pi extension not loaded | `pi -e ./.pi/extensions/pigame.ts` then `/reload` |\n\nThanks to ** Mini Game Studio** (\n\n`Digiman/mini-game-studio-ai-gen`\n\n) for the original Neon Snake and HTML game collection.We vendor a\n\n**patched** copy under\n\n`pigame/games/`\n\nfor Pi + LLM agent-mode testing (paused auto loop + single-step moves).Work in progress (MVP: Pi + LLM drives Snake via observe / move).\n\nPhase notes: [docs/implementation.md](/ifoxhz/pigame/blob/main/docs/implementation.md).", "url": "https://wpnews.pro/news/show-hn-pigame-pi-llm-plays-black-box-browser-games-via-observe-move-tools", "canonical_source": "https://github.com/ifoxhz/pigame", "published_at": "2026-08-21 01:20:26+00:00", "updated_at": "2026-08-21 01:43:35.025946+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-agents", "ai-tools", "ai-research"], "entities": ["ifoxhz", "Pigame", "Pi.dev", "DeepSeek", "Neon Snake", "GameMind", "GitHub", "Playwright"], "alternates": {"html": "https://wpnews.pro/news/show-hn-pigame-pi-llm-plays-black-box-browser-games-via-observe-move-tools", "markdown": "https://wpnews.pro/news/show-hn-pigame-pi-llm-plays-black-box-browser-games-via-observe-move-tools.md", "text": "https://wpnews.pro/news/show-hn-pigame-pi-llm-plays-black-box-browser-games-via-observe-move-tools.txt", "jsonld": "https://wpnews.pro/news/show-hn-pigame-pi-llm-plays-black-box-browser-games-via-observe-move-tools.jsonld"}}