We built WasserMan, an open-source benchmark with 10 underwater robot manipulation tasks in Isaac Sim: turning valves, opening hatches, inserting a hotstab and recovering objects.
The simulation includes water forces, currents and underwater camera effects. We evaluate ACT, Diffusion Policy and chunked BC on six tasks, and SmolVLA on three.
Code: [GitHub - dancher00/wasserman: WasserMan: Benchmark for Underwater Manipulation Policy Learning · GitHub](https://github.com/dancher00/wasserman)
Videos, results and paper: [WasserMan: Benchmark for Underwater Manipulation Policy Learning](https://dancher00.github.io/wasserman/)
The task demos show scripted experts in simulation; learned-policy results are reported separately. The README has setup instructions for native Linux, Python 3.12, a compatible NVIDIA GPU/driver and Isaac Sim 6.1.
I’m one of the authors. If you’re working on robot learning, I’d be interested in what underwater tasks or perception shifts you’d want to test next.
Sattyam 2
Nice work. The water presets are the part I’d build on.
Two shifts I’d test next:
- Closed-loop runs of the frozen checkpoints under your clear/coastal/silty presets. From the paper, these are optical replays right now, and the policy comparison uses one rendering profile. Keep the same seeds as the clean arm so every episode is paired, and report the interval clustered by task, not by episode.
- An instruction-shift arm for SmolVLA. You evaluate with a constant prompt, so this one is cheap. I ran something close on LIBERO-Object. A long reframed instruction took SmolVLA out of the keep-out zone in 42/50 episodes vs 1/50 with the normal prompt. But a scrambled-token control also moved it (18/30), so I read it as fragility, not steering. Caveats: simulation only, 5 seeds per task, the predicate is not calibrated yet, and that run is on an older release. Details: State of VLA Security - the measured result - Provael
Disclosure: I work on Provael, which does this on LIBERO. It does not run on Isaac Sim, so I can’t run WasserMan with it today.
One question: which policy was the 0.10 m/s current result on RotateValve (87.8 to 64.4)?
Thanks, Sattyam! Those numbers are for Diffusion Policy, averaged over three training seeds with paired resets. We’ve also completed closed-loop visibility tests with frozen ACT and DP checkpoints and are adding the results to the paper. We haven’t tested SmolVLA instruction shifts yet.