This is a submission for the MLH x DEV Writing Challenge Throw Hands is a 3D fighting game that runs in a browser tab. Your webcam tracks your body, and a fighter in a UFC-style octagon copies your arms, torso and head. Throw a real jab and it jabs; raise your forearms and it blocks; step toward the camera and it walks forward. You can fight a CPU, a friend on the same camera, or friends on other laptops and phones by room code. It's live at https://throwhands.tech, with no download, account or controller.
AI shows up in three places. A pose model in the browser (MediaPipe Pose Landmarker) is the controller. Fighter Forge turns a sentence like "lava golem with a crown" into a fighter with a name, backstory and card art, using Meta's Muse Spark and Muse Image. Gemini commentates the fight (details below).
I built it solo at HackTX 2026 in about 24 hours, working with two Claude Code agents.
My first jab detector measured arm extension, the distance from shoulder to wrist. It missed punches thrown straight at the lens. One webcam measures depth poorly: on a labeled clip, a full jab only changed the measured arm length from 0.24 m to 0.34 m.
The camera measures angles well. The elbow snaps open from about 75° to 120° in a tenth of a second, whatever the depth. Detecting peaks in elbow angle took recall on the labeled jab clip from 4 of 5 to 6 of 6.
The second detector used velocity over a 100 ms window. At 30 fps it worked. At 15 fps, which a phone or a busy laptop delivers, its recall fell to 20%, because a window holding one frame can't give a velocity.
Strikes are now peaks: a local maximum of elbow angle (jab), wrist height (uppercut) or sideways sweep (hook) that rose far enough within 350 ms. That costs one frame of latency and behaves the same at 10, 15 and 30 fps. An evaluation tool replays recorded body points at any frame rate, so every change is scored at both 30 and 15 fps.
I couldn't stand in front of a camera all night, so the end-to-end tests use downloaded video of real people as the webcam. Chromium's --use-file-for-fake-video-capture plays an MJPEG clip as a camera. For WebKit, the test injects a getUserMedia that paints a clip into a canvas stream.
The best test opens two browsers driven by two pro fighters' shadowboxing footage. They create and join a room through the real UI and play a full best-of-3 while each client logs every state snapshot it receives. Comparing the two logs: 2,160 shared ticks, 0 mismatches in health, round or timer.
The tests found real bugs. Safari never started pose tracking, for three separate reasons:
<video> sat inside a display:none element, and WebKit won't load a stream into it.
Shadowboxing clips got the gestures working, but my first session at a laptop broke movement: leaning back walked my fighter forward. I added a drill mode that shows one prompt at a time ("stand still", "big hook", "step back") and records the body points, with the prompt as the label. Replaying a recording through the same recognizer reproduced each bug before I fixed it.
One deterministic TypeScript simulation handles health, guard, supers, grappling, cage collisions and KOs. It runs in the browser for local play and on the server for online rooms. Every message carries a fighter slot ID, and bots send the same inputs humans do, so a 1v1 duel, an 8-player Brawl and winner-stays-on are all configurations of one room.
Claude Code made this video too. The implementer agent recorded the gameplay in a headless browser, rendered the motion graphics, generated the voiceover with ElevenLabs and cut it together with ffmpeg. I reviewed every cut and sent back fixes with timestamps.
Gemini API. Every 3 to 4 seconds, the server sends a batch of match events (hits, guard breaks, supers, cage slams, KOs, health) to a Gemini Flash-Lite model, which returns one roast or hype line about the fight. In online rooms the server writes one line per room, so every player and spectator sees the same call and the free tier lasts longer. Gemini TTS was supposed to voice the lines, but its free quota ran out during the hackathon. Live lines now appear as captions while a matching pre-recorded announcer roast plays. Next time I'd build the caption fallback first and add the voice on top.
Meta Muse API. Fighter Forge sends the player's one-line prompt to Muse Spark (muse-spark-1.3), which returns JSON with a name, backstory, three move names and two colors in about 2 seconds. Muse Image (muse-image-1.0) then paints the fighter card; setting reasoning_strength: "low" cut that from about 17 seconds to about 10. The fighter's suit takes the card's dominant color. After a two-player online match, the server makes one rivalry card per match (a trash-talk line from Muse Spark and a fight poster from Muse Image) and sends it to both players. Three things I learned:
model_not_found while 1.2 kept working, so text calls fall back to 1.2.
Tiger Data. Hits, blocks and supers go into a match_events hypertable, and continuous aggregates compute the Elo leaderboard. Players can opt in to share their body-point coordinates (never video) for a motion-data program, and those streams go into a pose_streams hypertable. My first schema stored each frame as an array, and compression only reached 1.7×. Storing one smallint column per coordinate let TimescaleDB's columnar compression work: 19.7 MB became 5.9 MB per player-hour, 3.4×.
Vultr. One Vultr VM in Dallas, the smallest plan (1 vCPU, 1 GB RAM), serves the game over HTTPS through Caddy. It also runs the WebSocket server and the simulation for online rooms, and it makes the Gemini, Meta and Tiger Data calls so no API key reaches the browser. An 8-browser Brawl test ran against it with 0 mismatches between clients.
.Tech domain. "Throw hands" is slang for fighting, so the game lives at throwhands.tech.
HackTX 2026 ran October 24 to 25 at the University of Texas at Austin, with about 695 hackers. I entered solo and split the work between two Claude Code agents:
They talked through files in the public repo: SPEC.md for what to build, REVIEW.md for my feedback, STATUS.md for progress. My job was the part neither agent could do: standing in front of the webcam at 4 AM, throwing hooks that didn't register, and recording drills until they did.
What I'll remember most is how bad AI can be at perceiving videos and animations until you review them yourself. In one cut of the demo video, two voiceovers played over each other at 50 seconds; in a later cut, the Shoulder Throw was still animated wrong at 56 to 58 seconds. Both agents got better once I gave them specific pointers, which I passed through the planner agent so it could build a better, more specific prompt, and once the agent reviewing the video looked at it at a finer grain, frame by frame. The game's own visual tests work the same way: scenes are rendered frame by frame and measured (foot slide, fist-to-target distance) before a fresh AI reviewer scores them. Frame-by-frame review is really slow across an entire video, though, so a human still has to step in periodically to review UI, animation and video changes.