Controlling the Human Body via EMS

Researchers at MIT developed Human Operator, a wearable system that uses voice commands and first-person vision to control electrical muscle stimulation (EMS) for human augmentation. The system won MIT Hard Mode 2026 and maps AI-generated actions to relay-routed EMS, enabling tasks like finger movement and wrist stimulation.

Human Operator is a wearable human-augmentation system that maps voice + first-person vision input to relay-routed Electrical Muscle Stimulation EMS actions. This project won MIT Hard Mode 2026 Learn Track . Left to right: AI stimulates wrist muscle to wave • AI stimulates fingers in sequence to play melody • AI stimulates fingers to form an OK sign app.py : Main runtime loop camera + voice trigger + LLM + command dispatch utils/receiver.py : Flask hardware gateway for timed relay/EMS command execution manual control app.py : PyQt GUI for manual calibration and direct stimulation testing firmware/human operator ems/human operator ems.ino : Arduino relay controller firmware run hardware.sh : Recommended one-command launcher for real relay hardware mode - Peter He - Portfolio https://peterhci.com | GitHub https://github.com/molegod | LinkedIn https://www.linkedin.com/in/ph475/ - Valdemar Danry - Portfolio https://valdemardanry.com | GitHub https://github.com/valleballe | LinkedIn https://www.linkedin.com/in/valdemar-danry - Daniel Kaijzer - GitHub https://github.com/danielkaijzer | LinkedIn https://www.linkedin.com/in/danielkaijzer/ - Yutong Wu - GitHub https://github.com/ichbinHallie0426 | LinkedIn https://www.linkedin.com/in/yutong-wu-4b66661b5/ - Ashley Neall - Portfolio https://aneall.github.io/ | GitHub https://github.com/aneall | LinkedIn https://www.linkedin.com/in/ashley-neall/ - Sean Hardesty Lewis - Portfolio https://seanhardestylewis.com/ | GitHub https://github.com/seanhlewis | LinkedIn https://www.linkedin.com/in/seanhardestylewis/ This repository controls physical electrical stimulation hardware. - Use only with appropriate supervision and informed consent. - Keep stimulation intensities conservative during calibration. - Ensure emergency stop access software stop and physical disconnect . - Do not use on people with contraindications to EMS. - Python 3.10+ - macOS/Linux shell environment - Arduino IDE for flashing firmware - Anthropic API key ANTHROPIC API KEY - Hardware: - Relay MCU Arduino-compatible - Relay board - EMS/stimulator device optional in relay-only mode - Camera for first-person capture - Clone and enter the project: git clone https://github.com/danielkaijzer/Human-Operator.git cd Human-Operator - Create and activate a local virtual environment: python3 -m venv .venv source .venv/bin/activate pip install -r requirements.txt pip install pyserial requests - Configure environment variables LLM key : cp .env empty .env then edit .env and set: ANTHROPIC API KEY=your key here - Flash relay firmware to your Arduino-compatible board: - Open firmware/human operator ems/human operator ems.ino - Select correct board/port in Arduino IDE - Upload at 115200 baud serial settings - Run full stack in relay-only hardware mode: ./run hardware.sh Optional custom relay port: RELAY PORT=/dev/cu.wchusbserial10 ./run hardware.sh What run hardware.sh does: - Starts utils/receiver.py in HARDWARE MODE=relay - Waits for /health - Verifies relay hardware connected=true - Launches app.py with RECEIVER URL set - Cleans up background receiver process on exit Use this for reliable relay switching and wiring validation: HARDWARE MODE=relay RELAY PORT=/dev/cu.usbserial-210 python utils/receiver.py Check health from another terminal: curl -sS http://127.0.0.1:5001/health Expected key fields: "hardware mode": "relay" "relay hardware connected": true Enable full stim path explicitly: HARDWARE MODE=full STIM PORT=/dev/cu.usbserial-XXX RELAY PORT=/dev/cu.usbserial-YYY python utils/receiver.py Notes: - Stim is intentionally disabled unless full mode is explicitly enabled. - If a device shows as SIMULATED , that serial port failed to open. Start the app if not using run hardware.sh : python app.py Runtime flow: - Wait for voice command trigger. - Capture latest camera frame. - Send prompt + image to Claude. - Transform action plan into timestamped relay/EMS payload. - POST payload to utils/receiver.py at RECEIVER URL . Run: python manual control app.py Use it to: - Connect EMS and relay serial ports manually - Select relay targets wrist left , wrist right , thumb , index , middle , ring , pinky , x - Tune amplitude/frequency/pulse width - Validate response before autonomous runtime Relay selector commands accepted by receiver/firmware: wrist left wrist right thumb index middle ring pinky x all off / isolate Firmware pin mapping in firmware/human operator ems/human operator ems.ino : - D2 - wrist left - D4 - wrist right - D3 - thumb - D5 - index - D6 - middle - D7 - ring - D8 - pinky Firmware serial debug commands: test to sweep all outputs mode low for active-low relay boards mode high for active-high relay boards x to set all outputs off Returns receiver status and hardware connection details. Accepts timestamp-keyed command dictionaries. Example: { "0.0": {"type": "RELAY", "finger": "index"}, {"type": "EMS", "channel": 1, "amplitude": 60, "duration": 1.0, "frequency": 100, "pulse width": 1000} , "1.5": {"type": "RELAY", "finger": "x"} } Ball-triggered avoidance demo: python utils/ball demo.py - Symptom: "relay": "SIMULATED" in /execute response or /health - Cause: relay serial port failed to open busy/wrong port/power issue - Fixes: - Close Arduino Serial Monitor and any other serial tool - Verify correct RELAY PORT - Re-run in explicit relay mode: HARDWARE MODE=relay RELAY PORT=/dev/cu.usbserial-210 python utils/receiver.py - Run firmware test command over serial - Try mode low and mode high - Verify relay board power and common ground - Verify IN pin wiring vs firmware pin map - Confirm camera permissions in macOS privacy settings - Confirm no other app is locking the camera - Ensure .env exists and includes ANTHROPIC API KEY - Check network connectivity and Anthropic account access php app.py - captures camera frame and voice command - asks Claude for action plan - transforms plan into receiver payload - POST /execute to utils/receiver.py utils/receiver.py Flask - validates and executes timestamped sequence - sends relay selector command to Arduino firmware - sends optional EMS/GVS/ET command to stimulator firmware/human operator ems/human operator ems.ino - maps symbolic relay targets to physical output pins Inspired by work from the Human Computer Integration Lab https://lab.plopes.org/ at UChicago: