Let multimodal models drive robots the way they use computers.
<sub>A real, unscripted run. Claude Opus 5.5, allowed to act only through programs, stacks the block on the mug in 5 turns and 18 seconds.</sub>
openRocode is an open agent harness that turns robot control into 3D computer use. The model looks at camera pictures, clicks where the gripper should go, nudges in directions it can see, or writes a short Python program for the parts that need loops and checks. The harness does the geometry, guards every motion, lets a person step in at any moment, and records everything.
The model never sees or writes a world coordinate. Depth, calibration and frames stay inside the harness. Tools speak pixels, centimetres and "left in the front camera".
git clone https://github.com/wrencanfly/openRocode.git
cd openRocode
python -m venv .venv && source .venv/bin/activate
pip install -e '.[sim,agent,openai,dev]'
rocode demo # a scripted episode, no API key: checks your install
pytest # 28 tests, about 5 seconds, no API key
Then give a model the arm, and watch it in the dashboard:
rocode run mug_to_coaster --view
export OPENAI_API_KEY=...
export OPENAI_BASE_URL=https://your-endpoint/v1
rocode run mug_to_coaster --view --provider openai --model claude-opus-5-5
rocode tasks lists the tasks. rocode view browses every recorded episode. rocode replay runs/<episode> prints one in the terminal.
move_to(camera, x, y) sends the gripper above whatever is under that pixel. Each picture is labelled (o1, o2, …) and a click is bound to the exact pictures the model looked at, so a gripper that later blocks the view cannot change what was clicked. Results come back as annotated pictures: the fingertip, its drop line to the table, and the last click are drawn on.
The model can act one step at a time, or hand a sequence to run_code when it needs loops, retries or checks on the readings. Every robot call a program makes goes through the same validation, safety hooks and emergency stop as a direct call. Programs cannot see pictures, so the model looks first and then writes code against what it saw.
While a run is going, tell the model something ("not that one, the one on the left"), it, or stop it, from the terminal or the dashboard. Messages reach the model before its next decision and stay in the record.
Every motion is force-guarded and workspace-checked by the body. Physical actions are never retried automatically, and an action that stops early cancels the rest of that turn's actions. An emergency stop halts motion immediately, including inside a running program.
Claude through the Anthropic API, or any OpenAI-compatible endpoint. History is append-only, so prompt caches stay warm and Claude's thinking blocks stay valid. Swap the model with a flag; nothing else changes.
Each episode is an event log plus every picture the model was shown, scored against the simulator's ground truth, which the model never sees.
A pi-style core of about 700 lines: loop, events, hooks and a control channel. Tools, safety, recording, the dashboard and the robot itself are plug-ins.
rocode run mug_to_coaster --view # opens on the live run, with controls
rocode view # every episode in ./runs, including runs still going
- Episodes : every run with its model, result, turns, duration and tokens.
- Debug : the turn-by-turn timeline (what the model said, each call, each program with its robot calls and output) beside both cameras, the readings for the selected step, and a telemetry chart.
- Watch : both cameras fill the page, with the telemetry chart and a strip of every picture the model saw. A running episode opens here.
Light and dark themes, arrow keys to step, space to replay, and links that open on a given step (#/e/<episode>?mode=watch&step=5). It is plain HTML and JavaScript served by Python's standard library: no build step, no Node. It listens on 127.0.0.1 only, and control requests need a token that only its own page carries.
A lightly trimmed excerpt from rocode run mug_to_coaster with Claude Opus 5.5. The model points once, then writes a program for the fiddly part: nudge, turn 90° so the fingers miss the handle, lower, grip, lift.
── turn 1
▶ move_to {"camera": "front", "x": 379, "y": 165, "hover_cm": 5}
✓ Hovering 5 cm above the point you clicked in o1. [o2]
── turn 2
▶ run_code
│ print(robot.nudge("side", right_cm=1.5).text)
│ print(robot.turn(90).text)
│ r=robot.lower(7); print(r.text, r.height_cm)
✓ Nudged. [o3]
✓ Turned 90°. [o4]
✓ Lowered 7.0 cm. [o5]
── turn 3
model: Fingers straddle the mug in side view. Lower more and grip.
▶ run_code
│ r=robot.lower(4); print(r.text)
│ r=robot.grip("close"); print(r.text, r.holding)
│ r=robot.nudge("front", up_cm=10); print(r.text)
✓ Lowered 3.6 cm and stopped on contact. [o6]
✓ Closed. Something 6.0 cm wide is between the fingers. [o7]
✓ Nudged. [o8]
...
── ended: done after 6 steps, 6 turns, 22.5 s
ground truth: success=True
| Tool | What it does |
|---|---|
look() |
Fresh pictures from every camera. |
move_to(camera, x, y, hover_cm=8, observation=None) |
Hover above the surface point under that pixel. Rises, travels, descends; stops on contact. |
lower(cm=None) |
Straight down until contact, or by cm . |
nudge(camera, right_cm, away_cm, up_cm) |
A small move in directions as seen in that camera's picture. |
turn(degrees) |
Rotate the gripper about the vertical. |
grip("close" | "open") |
Reports whether something is between the fingers, and how wide. |
run_code(code) |
Run a Python program against the same tools through a robot object. |
done(success, summary) |
End the episode. |
Every result says what actually happened (reached, touched something, blocked, refused), how high the fingertips are and what the gripper holds. Programs get the same facts as values:
r = robot.lower()
if not r.touched:
raise SystemExit("nothing under the gripper")
r = robot.grip("close")
print(r.holding, r.opening_cm, r.height_cm, r.stop)
--policy tools|code|both sets how the model may act. The default is both.
- Append-only history. Nothing already sent to the model is edited. Old pictures are trimmed only by the provider's own mechanism (
--keep-images N). - Hooks, not forks. An extension can add tools and prompt text, refuse or rewrite a call before it runs, change a result after it runs, or watch the event stream.
- Bodies are one small interface. Adding a robot means implementing
info,observe,move,rotate,grip,stopandreset. Nothing above that layer changes.
Design notes: docs/design.md (goals and scope) and docs/core-design.md (internals). The dashboard follows DESIGN.md.
Single runs on the MuJoCo tabletop, not success rates. Every run below succeeded by the simulator's ground truth.
| Task | Model | How it acted | Turns | Time |
|---|---|---|---|---|
lift_block |
Claude Opus 5.5 | tools | 4 | 13 s |
mug_to_coaster |
Claude Opus 5.5 | tools and programs | 6 | 23 s |
stack_block_on_mug |
Claude Opus 5.5 | tools | 6 | 23 s |
stack_block_on_mug |
Claude Opus 5.5 | programs only | 5 | 18 s |
mug_to_coaster |
Claude Sonnet 5.5 | tools | 7 | 26 s |
mug_to_coaster |
GPT-6.1 Sol | tools | 9 | 29 s |
rocode/
core/ loop, events, hooks, control channel, messages, tool contract
providers/ anthropic.py, openai_compat.py, scripted.py
body/ base.py (the body contract), mujoco_tabletop.py
ext/ pointing.py, code.py (run_code), safety.py, recorder.py
view/ the dashboard: episodes.py, server.py, static/
session.py tasks.py cli.py demo.py
tests/ harness, provider and dashboard tests (no API key needed)
docs/ design notes and images
openRocode is alpha. It works today on one MuJoCo tabletop with a floating gripper: the core, both providers, pointing tools, run_code, operator controls, recording, replay and the dashboard.
Known limits:
- No arm kinematics yet: the gripper floats, so reachability and joint limits are not modelled.
run_coderuns in a separate interpreter with call and time budgets, but it is not a security sandbox.- No batch evaluation yet; one
rocode runis one episode.
Next:
- Batch evaluation across tasks, seeds and models.
- A second body (LIBERO) to prove the body contract.
- A Python SDK, and an MCP server so Claude Code, pi and other agents can drive a robot directly.
- A simulated arm with real kinematics, then a first real-robot adapter.
- A hard real-time safety loop beside physical robots.
- Trained policies (VLAs) as tools.
openRocode is early, and focused contributions change a lot. Good places to start:
| You want to | Start in | Guide |
|---|---|---|
| Connect a robot or simulator | rocode/body/ |
Add a body |
| Add a task | rocode/tasks.py |
Add a task |
| Support another model API | rocode/providers/ |
Add a provider |
| Add a capability or tool | rocode/ext/ |
Rules the harness depends on |
| Improve the dashboard | rocode/view/static/ |
DESIGN.md |
Read CONTRIBUTING.md before your first pull request: it covers setup, the five rules every change keeps, and what a pull request needs. Issues labelled good first issue are ready to pick up. Tests need no API key.