# Cosmos 3’s Robot Simulations Need More Than Video Similarity > Source: > Published: 2026-11-06 11:05:23+00:00 A robot-world video can look convincing while leaving a more basic question unanswered: did the generated motion follow the action sequence that conditioned it? A smooth arm movement and a recognizable grasp are evidence about the image. They do not, by themselves, establish that the model predicted the consequence of a particular command, especially when the command differs from familiar demonstrations or should lead to failure. That distinction matters when developers consider generated rollouts for policy evaluation or training. NVIDIA’s [Nov. 5 article](https://huggingface.co/blog/nvidia/cosmos-interactive-world-simulation) describes four open-weight Cosmos 3 Nano checkpoints. Its publication date is not the release date: the cards mark V1.0 as Oct. 20, 2026. [Transfer-Auto](https://huggingface.co/nvidia/Cosmos3-Nano-Transfer-Auto) supports multi-view driving-data generation; [Sim-Auto](https://huggingface.co/nvidia/Cosmos3-Nano-Sim-Auto) targets closed-loop driving simulation; [Sim-Transfer](https://huggingface.co/nvidia/Cosmos3-Nano-Sim-Transfer) generates visual observations from simulator control sequences; and [Sim-Bimanual](https://huggingface.co/nvidia/Cosmos3-Nano-Sim-Bimanual) generates video conditioned on robot and camera actions. The Sim-Bimanual card lists robotic policy evaluation and verification, reinforcement-learning training, and synthetic demonstrations among its uses. Its reported video/reference measures include PSNR, SSIM and LPIPS, which the card describes as proxies for action following. Those measures can say something about resemblance between generated and reference frames; they do not answer every question about whether the action-conditioned outcome is right. This is a limit of what the measures establish, not evidence that the checkpoint fails them. ## Testing the action, not just the image The [WorldEcho v2 preprint](https://arxiv.org/abs/2608.24885), revised Oct. 23, 2026, makes the comparison more direct. It evaluates 50 RoboTwin tasks using 1,750 queries. For a query, the researchers compare the generated rollout with a simulator replay of the same action from the corresponding initial state. They check visual integrity and align the robot’s end-effector trajectory, rather than treating a plausible-looking video as sufficient. The query set includes five distinct categories: Demonstrated Action, Cross-State Replay, Local Perturbation, Policy Rollout, and Feasible-Space Sampling. These off-expert query types are designed to stress-test the model’s ability to generalize beyond standard training data. By introducing variations like cross-state replays or local perturbations, researchers can determine if the model maintains physical consistency when faced with novel or slightly altered inputs. WorldEcho reports that these off-expert queries increased integrity-gated error by 0.022 to 0.048 meter-equivalent units specifically across the six expert-trained models evaluated in that study. In its Cosmos3 row, the expert-demonstration condition recorded 0.0989 integrity-gated error, 0.0572 raw NDTW and an 82.51% visual pass rate. With expanded action coverage, those values were 0.0799, 0.0419 and 85.09%. Lower error is better; the pass percentage is a separate visual measure. The paper describes the comparison as study-specific: its baselines received 40,000 fine-tuning updates, and its Cosmos3 initialization was Cosmos3-Nano. These numbers are not results for the identical interactive checkpoint released by NVIDIA. The [MiraBench preprint](https://arxiv.org/abs/2605.29360), submitted May 28, 2026, separates three questions: physical adherence, action-following fidelity, and whether a model preserves failure when an action should fail. The authors argue visual fidelity is a poor proxy for action fidelity and test failure-inducing actions, such as insufficient grip force, premature release, grip carry slip, contact oscillation, wrist tilt, and approach overshoot. Its 12 evaluated configurations include Cosmos-GR1 2B and 14B, not the four Cosmos 3 Nano checkpoints. MiraBench therefore offers a different evaluation framework, not a direct assessment of the October releases. Together, the papers clarify why a video-quality score, a command-following test and a failure-preservation test should not be collapsed into one claim. A model could produce a visually coherent outcome while missing the commanded trajectory; nominal action following also does not, by itself, show how a model responds to an input that should produce failure. Each test catches a different kind of gap. By isolating these variables, researchers can examine distinct limits in model evaluations without treating any one score as complete. Neither preprint establishes that the released Cosmos 3 Nano checkpoints fail these tests, and both remain preprints. The practical implication is narrower: teams using generated rollouts should ask what their metric actually measures. Visual resemblance, action-specific outcomes and failure preservation are separate evaluation questions. A model card’s proxy metrics can characterize parts of the system, while simulator-replay or failure-inducing tests can probe different properties. Those results should be reported as complementary evidence, not treated as interchangeable proof of reliability or safety.