{"slug": "wasserman-open-source-underwater-manipulation-benchmark-with-smolvla-baselines", "title": "WasserMan: open-source underwater manipulation benchmark with SmolVLA baselines", "summary": "Researchers released WasserMan, an open-source benchmark of 10 underwater robot manipulation tasks built in Isaac Sim, with baselines for ACT, Diffusion Policy, chunked BC and SmolVLA. The benchmark evaluates ACT, Diffusion Policy and chunked BC on six tasks and SmolVLA on three, with simulation including water forces, currents and underwater camera effects; code and results are published on GitHub and a project site, and setup requires native Linux, Python 3.12, a compatible NVIDIA GPU/driver and Isaac Sim 6.1. In discussion, the authors confirmed the 0.10 m/s current result on RotateValve (87.8 to 64.4) came from Diffusion Policy averaged over three training seeds with paired resets, and that closed-loop visibility tests with frozen ACT and DP checkpoints are being added to the paper.", "body_md": "We built WasserMan, an open-source benchmark with 10 underwater robot manipulation tasks in Isaac Sim: turning valves, opening hatches, inserting a hotstab and recovering objects.\n\nThe simulation includes water forces, currents and underwater camera effects. We evaluate ACT, Diffusion Policy and chunked BC on six tasks, and SmolVLA on three.\n\nCode: [GitHub - dancher00/wasserman: WasserMan: Benchmark for Underwater Manipulation Policy Learning · GitHub](https://github.com/dancher00/wasserman)\n\nVideos, results and paper: [WasserMan: Benchmark for Underwater Manipulation Policy Learning](https://dancher00.github.io/wasserman/)\n\nThe task demos show scripted experts in simulation; learned-policy results are reported separately. The README has setup instructions for native Linux, Python 3.12, a compatible NVIDIA GPU/driver and Isaac Sim 6.1.\n\nI’m one of the authors. If you’re working on robot learning, I’d be interested in what underwater tasks or perception shifts you’d want to test next.\n\n \n \n[Sattyam](https://discuss.huggingface.co/u/Sattyam)\n2\n \nNice work. The water presets are the part I’d build on.\n\nTwo shifts I’d test next:\n\n1. Closed-loop runs of the frozen checkpoints under your clear/coastal/silty presets. From the paper, these are optical replays right now, and the policy comparison uses one rendering profile. Keep the same seeds as the clean arm so every episode is paired, and report the interval clustered by task, not by episode.\n2. An instruction-shift arm for SmolVLA. You evaluate with a constant prompt, so this one is cheap. I ran something close on LIBERO-Object. A long reframed instruction took SmolVLA out of the keep-out zone in 42/50 episodes vs 1/50 with the normal prompt. But a scrambled-token control also moved it (18/30), so I read it as fragility, not steering. Caveats: simulation only, 5 seeds per task, the predicate is not calibrated yet, and that run is on an older release. Details: [State of VLA Security - the measured result - Provael](https://www.provael.com/results)\n\nDisclosure: I work on Provael, which does this on LIBERO. It does not run on Isaac Sim, so I can’t run WasserMan with it today.\n\nOne question: which policy was the 0.10 m/s current result on RotateValve (87.8 to 64.4)?\n\n \n \nThanks, Sattyam! Those numbers are for Diffusion Policy, averaged over three training seeds with paired resets. We’ve also completed closed-loop visibility tests with frozen ACT and DP checkpoints and are adding the results to the paper. We haven’t tested SmolVLA instruction shifts yet.", "url": "https://wpnews.pro/news/wasserman-open-source-underwater-manipulation-benchmark-with-smolvla-baselines", "canonical_source": "https://discuss.huggingface.co/t/wasserman-open-source-underwater-manipulation-benchmark-with-smolvla-baselines/188270#post_3", "published_at": "2026-10-10 13:35:02+00:00", "updated_at": "2026-10-10 14:16:36.551486+00:00", "lang": "en", "topics": ["robotics", "ai-research", "machine-learning", "ai-safety"], "entities": ["WasserMan", "Isaac Sim", "SmolVLA", "Diffusion Policy", "ACT", "chunked BC", "NVIDIA", "Provael"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/wasserman-open-source-underwater-manipulation-benchmark-with-smolvla-baselines", "markdown": "https://wpnews.pro/news/wasserman-open-source-underwater-manipulation-benchmark-with-smolvla-baselines.md", "text": "https://wpnews.pro/news/wasserman-open-source-underwater-manipulation-benchmark-with-smolvla-baselines.txt", "jsonld": "https://wpnews.pro/news/wasserman-open-source-underwater-manipulation-benchmark-with-smolvla-baselines.jsonld"}}