A few weekends ago I got annoyed at typing prompts into a terminal and decided the fix was, obviously, to control my AI agent with hand gestures instead. This is the story of building that, and the two hours I lost fighting a GPU crash that had nothing to do with my code.
The idea: a webcam watches your hand, MediaPipe tracks the landmarks, and three gestures map to three actions on an Anthropic-powered coding agent.
No keyboard. No prompt box. Just your hand in front of a webcam, like you're a conductor telling an orchestra what to play.
I started with MediaPipe's newer Tasks API (HandLandmarker), because it's the one all the docs point you to now. It crashed immediately on my Mac with a Metal/GPU service error, even when I forced it onto the CPU delegate. Spent way too long assuming it was my setup before realizing the new API just doesn't play nice with this machine.
Switched to the legacy mp.solutions.hands API, pinned to mediapipe==0.10.21, and the problem vanished. Sometimes the fix for a shiny new API is to not use it yet.
Detecting "pinch" is easy: measure the distance between thumb and index tip, threshold it, done. The other two took more work.
"Running" fingers needed the vertical oscillation of the index and middle fingertips, counted by sign crossings, so it doesn't false trigger on a hand that's just drifting. "Spinning" tracks the index fingertip's trajectory and accumulates the signed angle around a center point, so a real circle reads differently than a shaky hand.
Both run on a rolling 1.5 second buffer of landmarks, edge triggered so a gesture fires once, not once per frame.
The runner executes whatever the agent wrote as a subprocess with a timeout, and it's intentionally not sandboxed. Watching an AI write code and then immediately run it in front of you, gesture by gesture, is the whole point of the demo. Sandboxing it would just make it less fun to watch.
First real camera test, no threshold tuning: all three gestures fired correctly. Then the full loop, real camera, real gestures, real Anthropic API calls: pinch wrote code, spin brainstormed an idea, running fingers executed it, and the HUD kept up with all of it live.
Next up is a small 3D avatar that mirrors the hand for recordings, so people watching a demo see a puppet instead of my actual hand blocking half the screen.
If you've fought MediaPipe's Tasks API on Apple Silicon, or you have opinions about gesture controlled anything, I'd like to hear about it.