Know the best move when playing chess with Meta glasses

A project called chesshacker lets a person wearing Meta glasses play over-the-board chess by video-calling a Mac over WhatsApp and having the Mac read the board and speak Stockfish's best move back into the call. The vision model only reports the squares it sees, code matches that to a legal move, and Stockfish — never the language model — picks the reply; the default vision model gpt-6.1-sol read 3 of 3 test moves at 1.4¢ and 4–4.6 seconds per frame, versus 3 of 3 for claude-opus-5-5 at 4¢ and 3.6–6.3 seconds. The system was tested end to end on glasses recordings but not yet on a live WhatsApp call, and spending pauses at $10 by default via --max-cost.

Overview chesshacker lets the person wearing Meta glasses play over-the-board chess with the best move in their ear. The glasses video-call a Mac on WhatsApp and share their view. The Mac works out when the opponent has moved, asks Stockfish for the reply, and speaks it back into the call. Two rules keep it honest: - The vision model only reports what it sees . Code matches that to a legal move. - Stockfish picks the move, never the language model. Tested end to end on glasses recordings. Not yet tried on a live WhatsApp call. Setup You need a Mac with Node.js 22 or newer brew install node . Clone this project Make this project your own and edit it easily with Reverse. Clone https://www.reverse.dev/signup?from=chess Setup downloads Stockfish, asks for your keys, and checks everything. You need an OpenAI API key; a Jev key from TypeSafe is optional and makes games cheaper. Keys are kept in your login Keychain. Also install WhatsApp from the Mac App Store and BlackHole 2ch brew install blackhole-2ch , and give your terminal app Screen Recording permission. Then check a real call once with npx chesshacker test-call . Playing a game 1. From the glasses, video-call this Mac’s WhatsApp and answer it on the Mac. 2. Double-tap the glasses’ capture button to share their view. In the call’s ⋯ menu, set the microphone to BlackHole 2ch. 3. Start chesshacker with your color: npx chesshacker run --color white npx chesshacker run --color black --fen "<position " Look at the board. You’ll hear “Ready”, then the best move after each of your opponent’s moves. The command center opens in your browser at 127.0.0.1:8000, with the live frame, the board and its arrow, and controls to pause, undo, enter a move, or say it again. Spending pauses at $10 by default --max-cost . How a move is read About once a second, a frame gets two looks: 1. A quick check. Jev skips it if the board isn’t readable under 0.8 , a hand is over it over 0.5 , or nothing changed under 0.1 . 2. A full read. The vision model lists the squares that look different from the last confirmed frame. Code compares that with the board after every legal move, and after every move plus a reply. A move is committed once it’s the best fit on 2 frames in a row, by 3 or more squares in total. | Vision model | Moves | Per frame | |---|---|---| | gpt-6.1-sol default | 3 of 3 | 1.4¢ · 4–4.6 s | | claude-opus-5-5 | 3 of 3 | 4¢ · 3.6–6.3 s | About 10 seconds pass from a move to the spoken reply, mostly the two model reads. When something’s wrong Not sure what’s missing Run npx chesshacker doctor . It checks Stockfish, your keys, WhatsApp, BlackHole and Screen Recording, and prints the exact fix for each problem. It watches the wrong window npx chesshacker windows lists WhatsApp’s windows. Pick one with --window . “Lost track of the board” Fix it from the command center: undo, enter the move yourself, or start a new game from a FEN. No speech in the glasses Set the call’s microphone to BlackHole 2ch in WhatsApp’s ⋯ menu. test-call checks this for you.