cd /news/robotics/show-hn-openrocode-let-multimodal-mo… · home › topics › robotics › article
[ARTICLE · art-171659] src=github.com ↗ pub= topic=robotics verified=true sentiment=↑ positive

Show HN:openRocode–let multimodal models drive robots the way they use computers

A developer released openRocode, an open-source agent harness that lets multimodal models control robot arms by clicking on camera pixels rather than handling world coordinates, with the harness keeping depth, calibration and frames internal. In a demonstration run, Claude Opus 5.5 — acting only through programs — stacked a block on a mug in 5 turns and 18 seconds. The project ships with 28 tests that run in about 5 seconds without an API key, supports the Anthropic API and any OpenAI-compatible endpoint, and force-guards and workspace-checks every motion with an emergency stop that also halts running programs.

key takeaway

openRocode binds each click to the exact labelled camera frame the model saw, so a gripper that later blocks the view cannot change what was clicked, and it routes every robot call — direct or inside run_code — through the same validation, safety hooks and emergency stop.

read8 min views4 publishedNov 18, 2026
Show HN:openRocode–let multimodal models drive robots the way they use computers
Image: Michielbdejong (auto-discovered)

Let multimodal models drive robots the way they use computers.

<sub>A real, unscripted run. Claude Opus 5.5, allowed to act only through programs, stacks the block on the mug in 5 turns and 18 seconds.</sub>

openRocode is an open agent harness that turns robot control into 3D computer use. The model looks at camera pictures, clicks where the gripper should go, nudges in directions it can see, or writes a short Python program for the parts that need loops and checks. The harness does the geometry, guards every motion, lets a person step in at any moment, and records everything.

The model never sees or writes a world coordinate. Depth, calibration and frames stay inside the harness. Tools speak pixels, centimetres and "left in the front camera".

git clone https://github.com/wrencanfly/openRocode.git
cd openRocode
python -m venv .venv && source .venv/bin/activate
pip install -e '.[sim,agent,openai,dev]'

rocode demo        # a scripted episode, no API key: checks your install
pytest             # 28 tests, about 5 seconds, no API key

Then give a model the arm, and watch it in the dashboard:

rocode run mug_to_coaster --view

export OPENAI_API_KEY=...
export OPENAI_BASE_URL=https://your-endpoint/v1
rocode run mug_to_coaster --view --provider openai --model claude-opus-5-5

rocode tasks lists the tasks. rocode view browses every recorded episode. rocode replay runs/<episode> prints one in the terminal.

move_to(camera, x, y) sends the gripper above whatever is under that pixel. Each picture is labelled (o1, o2, …) and a click is bound to the exact pictures the model looked at, so a gripper that later blocks the view cannot change what was clicked. Results come back as annotated pictures: the fingertip, its drop line to the table, and the last click are drawn on.

The model can act one step at a time, or hand a sequence to run_code when it needs loops, retries or checks on the readings. Every robot call a program makes goes through the same validation, safety hooks and emergency stop as a direct call. Programs cannot see pictures, so the model looks first and then writes code against what it saw.

While a run is going, tell the model something ("not that one, the one on the left"), it, or stop it, from the terminal or the dashboard. Messages reach the model before its next decision and stay in the record.

Every motion is force-guarded and workspace-checked by the body. Physical actions are never retried automatically, and an action that stops early cancels the rest of that turn's actions. An emergency stop halts motion immediately, including inside a running program.

Claude through the Anthropic API, or any OpenAI-compatible endpoint. History is append-only, so prompt caches stay warm and Claude's thinking blocks stay valid. Swap the model with a flag; nothing else changes.

Each episode is an event log plus every picture the model was shown, scored against the simulator's ground truth, which the model never sees.

A pi-style core of about 700 lines: loop, events, hooks and a control channel. Tools, safety, recording, the dashboard and the robot itself are plug-ins.

rocode run mug_to_coaster --view   # opens on the live run, with controls
rocode view                        # every episode in ./runs, including runs still going
  • Episodes : every run with its model, result, turns, duration and tokens.
  • Debug : the turn-by-turn timeline (what the model said, each call, each program with its robot calls and output) beside both cameras, the readings for the selected step, and a telemetry chart.
  • Watch : both cameras fill the page, with the telemetry chart and a strip of every picture the model saw. A running episode opens here.

Light and dark themes, arrow keys to step, space to replay, and links that open on a given step (#/e/<episode>?mode=watch&step=5). It is plain HTML and JavaScript served by Python's standard library: no build step, no Node. It listens on 127.0.0.1 only, and control requests need a token that only its own page carries.

A lightly trimmed excerpt from rocode run mug_to_coaster with Claude Opus 5.5. The model points once, then writes a program for the fiddly part: nudge, turn 90° so the fingers miss the handle, lower, grip, lift.

── turn 1
   ▶ move_to {"camera": "front", "x": 379, "y": 165, "hover_cm": 5}
   ✓ Hovering 5 cm above the point you clicked in o1.  [o2]

── turn 2
   ▶ run_code
     │ print(robot.nudge("side", right_cm=1.5).text)
     │ print(robot.turn(90).text)
     │ r=robot.lower(7); print(r.text, r.height_cm)
      ✓ Nudged.  [o3]
      ✓ Turned 90°.  [o4]
      ✓ Lowered 7.0 cm.  [o5]

── turn 3
   model: Fingers straddle the mug in side view. Lower more and grip.
   ▶ run_code
     │ r=robot.lower(4); print(r.text)
     │ r=robot.grip("close"); print(r.text, r.holding)
     │ r=robot.nudge("front", up_cm=10); print(r.text)
      ✓ Lowered 3.6 cm and stopped on contact.  [o6]
      ✓ Closed. Something 6.0 cm wide is between the fingers.  [o7]
      ✓ Nudged.  [o8]
   ...
── ended: done after 6 steps, 6 turns, 22.5 s
   ground truth: success=True
Tool What it does
look() Fresh pictures from every camera.
move_to(camera, x, y, hover_cm=8, observation=None) Hover above the surface point under that pixel. Rises, travels, descends; stops on contact.
lower(cm=None) Straight down until contact, or by cm .
nudge(camera, right_cm, away_cm, up_cm) A small move in directions as seen in that camera's picture.
turn(degrees) Rotate the gripper about the vertical.
grip("close" | "open") Reports whether something is between the fingers, and how wide.
run_code(code) Run a Python program against the same tools through a robot object.
done(success, summary) End the episode.

Every result says what actually happened (reached, touched something, blocked, refused), how high the fingertips are and what the gripper holds. Programs get the same facts as values:

r = robot.lower()
if not r.touched:
    raise SystemExit("nothing under the gripper")
r = robot.grip("close")
print(r.holding, r.opening_cm, r.height_cm, r.stop)

--policy tools|code|both sets how the model may act. The default is both.

  • Append-only history. Nothing already sent to the model is edited. Old pictures are trimmed only by the provider's own mechanism (--keep-images N ).
  • Hooks, not forks. An extension can add tools and prompt text, refuse or rewrite a call before it runs, change a result after it runs, or watch the event stream.
  • Bodies are one small interface. Adding a robot means implementinginfo ,observe ,move ,rotate ,grip ,stop andreset . Nothing above that layer changes.

Design notes: docs/design.md (goals and scope) and docs/core-design.md (internals). The dashboard follows DESIGN.md.

Single runs on the MuJoCo tabletop, not success rates. Every run below succeeded by the simulator's ground truth.

Task Model How it acted Turns Time
lift_block Claude Opus 5.5 tools 4 13 s
mug_to_coaster Claude Opus 5.5 tools and programs 6 23 s
stack_block_on_mug Claude Opus 5.5 tools 6 23 s
stack_block_on_mug Claude Opus 5.5 programs only 5 18 s
mug_to_coaster Claude Sonnet 5.5 tools 7 26 s
mug_to_coaster GPT-6.1 Sol tools 9 29 s
rocode/
  core/        loop, events, hooks, control channel, messages, tool contract
  providers/   anthropic.py, openai_compat.py, scripted.py
  body/        base.py (the body contract), mujoco_tabletop.py
  ext/         pointing.py, code.py (run_code), safety.py, recorder.py
  view/        the dashboard: episodes.py, server.py, static/
  session.py   tasks.py   cli.py   demo.py
tests/         harness, provider and dashboard tests (no API key needed)
docs/          design notes and images

openRocode is alpha. It works today on one MuJoCo tabletop with a floating gripper: the core, both providers, pointing tools, run_code, operator controls, recording, replay and the dashboard.

Known limits:

  • No arm kinematics yet: the gripper floats, so reachability and joint limits are not modelled.
  • run_code runs in a separate interpreter with call and time budgets, but it is not a security sandbox.
  • No batch evaluation yet; one rocode run is one episode.

Next:

  1. Batch evaluation across tasks, seeds and models.
  2. A second body (LIBERO) to prove the body contract.
  3. A Python SDK, and an MCP server so Claude Code, pi and other agents can drive a robot directly.
  4. A simulated arm with real kinematics, then a first real-robot adapter.
  5. A hard real-time safety loop beside physical robots.
  6. Trained policies (VLAs) as tools.

openRocode is early, and focused contributions change a lot. Good places to start:

You want to Start in Guide
Connect a robot or simulator rocode/body/ Add a body
Add a task rocode/tasks.py Add a task
Support another model API rocode/providers/ Add a provider
Add a capability or tool rocode/ext/ Rules the harness depends on
Improve the dashboard rocode/view/static/ DESIGN.md

Read CONTRIBUTING.md before your first pull request: it covers setup, the five rules every change keeps, and what a pull request needs. Issues labelled good first issue are ready to pick up. Tests need no API key.

── more in #robotics 4 stories · sorted by recency
── more on @openrocode 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/show-hn-openrocode-l…] indexed:0 read:8min 2026-11-18 · —