Nvidia Releases Cosmos-H-Dreams for Real-Time Surgical-Robot Simulation

Nvidia released Cosmos-H-Dreams, an interactive generative model that lets researchers operate a surgical robot inside a synthesized video environment, generating action-conditioned video rollouts at about 160 frames per second on an RTX PRO 6000. The system, specialized for da Vinci Research Kit tabletop suturing, is a research and development platform not intended for clinical decisions or patient care, and CMR Surgical demonstrated a Versius integration in simulation.

Nvidia Releases Cosmos-H-Dreams for Real-Time Surgical-Robot Simulation - Cosmos-H-Dreams takes an initial image and robot kinematics, then generates synthetic RGB video showing predicted future scene states. It does not generate or execute robot actions. 1 https://huggingface.co/blog/nvidia/cosmos-h-dreams - Nvidia reports about 160 generated frames per second on one RTX PRO 6000, but that is a throughput figure rather than a disclosed end-to-end response-latency measurement. 1 https://huggingface.co/blog/nvidia/cosmos-h-dreams 2 https://github.com/isaac-for-healthcare/Cosmos-H-Dreams - The code and 4.36-gigabyte checkpoint are available, although the repository’s promised dedicated technical report still points to a placeholder. 2 https://github.com/isaac-for-healthcare/Cosmos-H-Dreams 3 https://huggingface.co/nvidia/Cosmos-H-Dreams/tree/main - CMR Surgical demonstrated a Versius integration in simulation. The capability is not part of the cleared Versius Plus system and is not intended for clinical decisions or patient care. 4 https://us.cmrsurgical.com/news/cmr-surgical-and-nvidia-showcase-future-of-intelligent-surgery-with-versius-plus-at-srs-2026 Nvidia has released Cosmos-H-Dreams, an interactive generative model designed to let researchers operate a surgical robot inside a synthesized video environment. The system accepts a starting camera frame and a stream of robot movements, then generates their predicted visual consequences in successive blocks. 1 https://huggingface.co/blog/nvidia/cosmos-h-dreams The distinction is important: Cosmos-H-Dreams generates action-conditioned video rollouts, which can function as a learned world simulation and a source of synthetic training or evaluation data. It does not propose a trajectory, issue motor commands or control a physical surgical robot. Nvidia describes it as a research and development platform, not a diagnostic system, intraoperative imaging replacement or robot controller. 1 https://huggingface.co/blog/nvidia/cosmos-h-dreams An open release with a narrow checkpoint The released checkpoint is derived from Nvidia’s 2-billion-parameter Cosmos-Predict2.5 video model and is specialized for da Vinci Research Kit tabletop suturing. It operates at 288 by 512 pixels, receives robot actions through a unified 44-dimensional representation and outputs blocks of 12 RGB frames. For the released dVRK setup, the effective action-control rate is 10 hertz. 5 https://huggingface.co/nvidia/Cosmos-H-Dreams Nvidia trained the system by distilling an offline, bidirectional surgical-video model into a causal student that conditions each new block on its earlier generated output. The company says the training mixture includes successful demonstrations and failures such as dropped needles, missed throws and unsuccessful knot ties. Its broader Open-H corpus contains more than 26,500 demonstrations and about 4.9 million synchronized video-and-kinematics frames, while Nvidia describes the dVRK post-training set as roughly 1 million frames. 5 https://huggingface.co/nvidia/Cosmos-H-Dreams The source code is under Apache 2.0, while the weights use Nvidia’s Open Model License, which permits commercial use and derivative models subject to attribution, guardrail and other conditions. Nvidia published the model repository and inference code on July 23, 2026. The repository lists Linux, CUDA 13-class drivers and at least 12 gigabytes of GPU memory as minimum requirements, but the company’s real-time validation used an RTX PRO 6000 rather than a 12-gigabyte card. 2 https://github.com/isaac-for-healthcare/Cosmos-H-Dreams 6 https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/ The release package is more complete than its research record. As of July 27, the repository’s “Cosmos-H-Dreams technical report” link resolves to a file named “TODO.” The model card instead references broader Cosmos research and earlier surgical-robotics work. Nvidia has not published a dedicated H-Dreams paper with a complete benchmark protocol, comparative results or independent reproduction. 2 https://github.com/isaac-for-healthcare/Cosmos-H-Dreams 5 https://huggingface.co/nvidia/Cosmos-H-Dreams What Nvidia’s real-time claim measures Nvidia says its FlashDreams inference stack raises generation from roughly 10 frames per second for its earlier offline simulator to about 160 frames per second on one RTX PRO 6000. FlashDreams uses few-step diffusion, a streaming key-value cache, model compilation and CUDA Graphs. 1 https://huggingface.co/blog/nvidia/cosmos-h-dreams That figure measures generated-frame throughput. The published materials do not report motion-to-photon latency: the complete interval from a controller movement, through action transmission and model inference, to WebRTC encoding and display. Nor does the announcement identify which two- or four-step sampler and decoder produced the 160-fps result. The repository says two-step inference offers about twice the diffusion-transformer speed at some cost to fidelity, while a lightweight decoder is about 10 times faster than the full decoder with a modest quality reduction. It also warns that the first user input takes substantially longer while the model compiles. 2 https://github.com/isaac-for-healthcare/Cosmos-H-Dreams The release therefore supports an interactive demonstration, but its latency cannot yet be compared directly with a surgical control loop. Throughput also does not establish physical accuracy. Nvidia lists tool-tip accuracy, gripper fidelity, idle stability, long-horizon drift and agreement between simulated and real policy outcomes as benchmarks that still need to be developed. 1 https://huggingface.co/blog/nvidia/cosmos-h-dreams The Versius demonstration remains outside the operating room Nvidia worked with CMR Surgical and Cambridge Consultants to adapt the model to CMR’s Versius platform. CMR presented the capability at the Society of Robotic Surgery meeting in Florida as a simulated environment that predicts what an operating field could look like after a proposed robot action. 1 https://huggingface.co/blog/nvidia/cosmos-h-dreams 4 https://us.cmrsurgical.com/news/cmr-surgical-and-nvidia-showcase-future-of-intelligent-surgery-with-versius-plus-at-srs-2026 CMR explicitly says the simulation is for research and demonstration only, is not part of the currently cleared Versius Plus system and is not intended for clinical decision-making or patient care. In the United States, the underlying Versius Plus system is FDA-cleared for cholecystectomy in eligible adult patients; CMR said gynecological indications remained pending as of its July 22 announcement. 4 https://us.cmrsurgical.com/news/cmr-surgical-and-nvidia-showcase-future-of-intelligent-surgery-with-versius-plus-at-srs-2026 CMR contributed nearly 500 hours of anonymized Versius clinical data to Open-H, according to Nvidia, spanning procedures including cholecystectomy, prostatectomy, hernia repair and hysterectomy. 7 That real-world data contribution should not be confused with clinical use of the simulator. No cited partner has reported Cosmos-H-Dreams guiding an operation, controlling a robot around a patient or receiving regulatory authorization for those uses. Research outside the H-Dreams release supports a narrower case for simulation transfer. The ORBIT-Surgical team demonstrated transfer of policies from a physics simulator to a physical dVRK across laboratory surgical subtasks. 8 A separate world-model study reported a 65% grasping success rate after simulation training in phantom and ex vivo settings. Those results show that surgical sim-to-real transfer is possible in constrained experiments, not that a generative video model can reproduce the full range of anatomy, bleeding, tissue response and complications found in an operating room. 9 https://arxiv.org/abs/2508.11200 A 2026 perspective in npj Digital Medicine concluded that synthetic surgical data could help train people and robots, while calling for physics models calibrated against physical materials, standardized benchmarks and explicit testing on atypical anatomy and rare complications. 10 FDA’s 2023 guidance provides a risk-informed credibility framework for simulation evidence in medical-device submissions, but it specifically covers physics-based, mechanistic and first-principles models. It does not directly establish a validation route for a learned generative video world model. 11 https://www.fda.gov/regulatory-information/search-fda-guidance-documents/assessing-credibility-computational-modeling-and-simulation-medical-device-submissions For now, Cosmos-H-Dreams is an available research tool and an industry simulation demonstration. The unanswered issue is whether its generated visual dynamics predict physical robot outcomes reliably enough to survive validation beyond tabletop tasks and simulated Versius sessions. Companies mentioned Further sources 1 Nvidia and Hugging Face, “NVIDIA Cosmos-H-Dreams: Bringing Real-Time Generative… ↗ https://huggingface.co/blog/nvidia/cosmos-h-dreams 2 Cosmos-H-Dreams source repository, covering release status, inference configura… ↗ https://github.com/isaac-for-healthcare/Cosmos-H-Dreams 3 Hugging Face repository listing the released Cosmos-H-Dreams checkpoint files a… ↗ https://huggingface.co/nvidia/Cosmos-H-Dreams/tree/main 4 CMR Surgical, “CMR Surgical and NVIDIA showcase future of intelligent surgery w… ↗ https://us.cmrsurgical.com/news/cmr-surgical-and-nvidia-showcase-future-of-intelligent-surgery-with-versius-plus-at-srs-2026 5 Nvidia Cosmos-H-Dreams model card, covering architecture, inputs, outputs, data… ↗ https://huggingface.co/nvidia/Cosmos-H-Dreams 6 Nvidia Open Model License Agreement, last modified October 24, 2025. ↗ https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/ +5 more The stories that matter, in one email. Free — unsubscribe anytime.