{"slug": "build-world-models-for-interactive-simulation-with-nvidia-cosmos", "title": "Build World Models for Interactive Simulation with NVIDIA Cosmos", "summary": "NVIDIA released the Cosmos Framework, an open, modular framework for world model training, optimization and inference, alongside Cosmos NIM microservices for GPU-accelerated serving, the company said in a technical post. The framework adds agent skills and recipes for training on text, images, video, audio and actions, supports LoRA and full supervised fine-tuning, and adds online and offline reinforcement learning pipelines including DPO, GRPO, PPO, OPD and DiffusionNFT. NVIDIA said the framework is expanding beyond Cosmos 3 to support NVIDIA Alpamayo, GR00T, Qwen, pi0.5 and other compatible Hugging Face models, with inference runnable through Diffusers, PyTorch, SGLang, vLLM-Omni, vLLM, TRT-LLM, TRT-Edge-LLM or NIM.", "body_md": "Physical AI developers working with [world models](https://www.nvidia.com/en-us/glossary/world-models/) face two distinct challenges: adapting models to their sensors, tasks, and data, and getting those models into applications reliably.  \n\n[NVIDIA Cosmos](https://www.nvidia.com/en-us/ai/cosmos/) addresses each. Cosmos is a platform of world foundation models and development tools for [physical AI](https://www.nvidia.com/en-us/glossary/generative-physical-ai/). [Cosmos Framework](https://github.com/NVIDIA/cosmos-framework) provides agent skills and recipes for world model training, multimodal data processing, optimization and inference. [Cosmos NIM microservices](https://build.nvidia.com/models?q=cosmos) supports GPU-accelerated serving through application APIs.\n\nThis post walks through both. The simulation section shows how to use Cosmos Framework’s inference recipes and agent skills to generate synchronized camera video and LiDAR observations for AV, robot, and surgical scenarios, starting from released checkpoints you can run today. The deployment section shows how to choose a NIM configuration for your GPU, launch the container, verify readiness, and send your first inference request.\n\n## When to use Cosmos Framework for world model development\n\n[Cosmos Framework](https://github.com/NVIDIA/cosmos-framework) is an open and fully modular framework for world model training, model optimization, and inference to support a full model development lifecycle using new agent skills. It is expanding beyond Cosmos 3 to support [NVIDIA Alpamayo](https://github.com/NVlabs/alpamayo-recipes), GR00T, Qwen, pi0.5, and other compatible Hugging Face models.\n\n**Training**\n\n- Train models on text, images, video, audio, and actions faster using new skills.\n- Use LoRA for parameter-efficient adaptation, or full supervised fine-tuning (SFT) to update all model parameters.\n- Load data through JSONL, WebDataset, LeRobot, or custom adapters and train across GPUs and save checkpoints to resume runs.\n- Point your agent to post-training [skill](https://github.com/NVIDIA/cosmos-framework/tree/main/.agents/skills/cosmos3-post-training) to accelerate and automate.\n\n**Apply Reinforcement Learning** \n\n- The platform now covers [online and offline RL paths](https://github.com/NVIDIA/cosmos-framework/blob/main/docs/posttrain/howto/run-posttrain.md) . Newly supported pipelines include:\n  - DPO: offline preference optimization.\n  - GRPO: online RL with a rule or model reward and vLLM rollouts.\n  - PPO: actor-critic RL with frozen reference and critic, including RL for embodied policies in simulation.\n  - OPD: on-policy distillation from a frozen teacher.\n  - DiffusionNFT: online RL for diffusion world models.\n- Run rollout, reward, and training roles on shared GPUs or separate pools.\n- Connect supported simulators to collect experience.\n\n**Optimize models and inference**\n\n- Distill models to reduce model size or generation steps; quantize models to reduce memory use.\n- Run compatible checkpoints with Hugging Face Diffusers, PyTorch, SGLang, vLLM-Omni, vLLM, TRT-LLM and TRT-Edge-LLM or NVIDIA NIM.\n- Evaluate for quality, accuracy and GPU usage and maximize compute utilization.\n\n**Use agents with Cosmos Framework**\n\nLoad the setup, inference, post-training, and troubleshooting skills under `.agents/skills`. Give the agent a bounded task:\n\n**Prompt***: Post-train* `<checkpoint>` *on* `<dataset>` *using* `<recipe>`*. Pin all revisions, validate the dataset, and keep training and evaluation episodes separate. Run the smoke test, save the configuration, and stop at* `<N>` *GPU-hours. Export and compare against the base model on held-out data using three seeds. Repeat evaluation at a supported lower precision; record quality, peak memory, and latency.*\n\n## Which Cosmos simulation workflow should you use?\n\nCosmos Framework provides post-training and inference recipes for robot simulation, realistic scene generation, multi-camera driving simulation, and surgical policy evaluation. New Cosmos 3 Nano simulation recipes and example checkpoints enable developers to use agent runnable skills to evaluate our example checkpoints and adapt Cosmos 3 Nano to custom sensors, action space, and data domains.\n\nThese checkpoints support PyTorch and vLLM-Omni, tested on NVIDIA Hopper and Blackwell GPUs. Use the recipe to select GPU count, memory configuration, and inference settings.\n\n**Autonomous vehicle simulation:**\n\nBoth AV checkpoints take a prompt and World Scenario Map (WSM) controls representing lanes, objects, road markings, traffic signs, and road boundaries. Joint RGB–LiDAR generation additionally uses HD-map rangemap controls and supports optional initial LiDAR observations. Prompts describe weather, lighting, time of day, road conditions with prefixes providing camera-rig and field-of-view context. An optional initial camera images provide visual context to guide the generated scene’s appearance.\n\n[Cosmos3-Nano-Transfer-Auto](https://huggingface.co/nvidia/Cosmos3-Nano-Transfer-Auto) uses bidirectional generation to denoise an entire multi-view clip at once for offline [synthetic data generation](https://www.nvidia.com/en-us/glossary/synthetic-data-generation/). It supports 30 fps input and output video, with up to 400 frames at 480p or 720p; longer sequences can be generated through autoregressive chunking. Change the prompt to turn a daytime drive into a rainy night while keeping the same road layout and vehicle trajectories. Use these variations to train and evaluate perception and driving models across weather and lighting conditions.\n\nAfter Framework setup, run the four-GPU examples below from the repository root. For Transfer-Auto, use the [published sample configuration](https://huggingface.co/nvidia/Cosmos3-Nano-Transfer-Auto/blob/main/assets/sample_data/input-transfer-auto.jsonl) with seven synchronized WSM videos, per-camera captions, first-frame RGB conditioning, and HD-map LiDAR controls. See the [Transfer-Auto inference guide](https://github.com/NVIDIA/cosmos-framework/blob/main/docs/inference_auto.md) for setup, input preparation, and output details.\n\n```\nexport COSMOS_TRAINING=0 COSMOS_INTERNAL=0\n\nhf download nvidia/Cosmos3-Nano-Transfer-Auto \\\n  --local-dir checkpoints/Cosmos3-Nano-Transfer-Auto\n\nAUTO_SAMPLE_ROOT=\"$PWD/checkpoints/Cosmos3-Nano-Transfer-Auto/assets/sample_data\"\nAUTO_NEGATIVE_PROMPT=\"$(cat .agents/skills/cosmos3-nano-transfer-auto/inference/assets/negative-prompt.txt)\"\n\ntorchrun --standalone --nproc-per-node=4 \\\n  -m cosmos_framework.scripts.inference \\\n  -i \"$AUTO_SAMPLE_ROOT/input-transfer-auto.jsonl\" \\\n  -o outputs/transfer_auto \\\n  --checkpoint-path checkpoints/Cosmos3-Nano-Transfer-Auto \\\n  --resolution 720 --num-steps 35 \\\n  --per-view-negative-prompt \"$AUTO_NEGATIVE_PROMPT\" \\\n  --cp-size 4 --cfgp-size 1 \\\n  --no-use-torch-compile --no-diffusion-cache\n```\n\nThe reference run produces seven 720p videos (297 frames at 30 FPS) and 99 LiDAR sweeps. Check `sample_outputs.json` for success and `lidar.safetensors` for finite range and intensity values, then verify WSM alignment and consistency across camera views and LiDAR.\n\nFor custom scenarios and rig adaptation, see [the WSM renderer](https://github.com/NVIDIA/cosmos-framework/blob/main/.agents/skills/cookbooks/cosmos3/simulation/auto/wsm/README.md) and [Transfer-Auto post-training guides](https://github.com/NVIDIA/cosmos-framework/blob/main/docs/training_transfer_auto.md). For point-cloud conversion and visualization, see the [LiDAR visualization utilities](https://github.com/NVIDIA/cosmos-framework/blob/main/.agents/skills/cookbooks/cosmos3/simulation/auto/helpers/lidar_visualization.py)\n\n[Cosmos3-Nano-Sim-Auto](https://huggingface.co/nvidia/Cosmos3-Nano-Sim-Auto) is the causal, few-step-distilled variant of Cosmos3-Nano-Transfer-Auto, enabling autoregressive streaming generation. It generates observations incrementally as an AV policy or human driving input updates the scenario through a WSM renderer. Generated camera and LiDAR observations return to the policy, enabling closed-loop evaluation \n\nFor Sim-Auto, reproduce the reference rollout using the [published Sim-Auto sample checkpoint](https://huggingface.co/nvidia/Cosmos3-Nano-Transfer-Auto/blob/main/assets/sample_data/input-sim-auto.jsonl) and pre-rendered WSM controls to generate synchronized camera videos and LiDAR observations autoregressively. See the [Sim-Auto inference guide](https://github.com/NVIDIA/cosmos-framework/blob/main/docs/inference_sim_auto.md) for setup, input configuration, and output details.\n\n```\nexport COSMOS_TRAINING=0 COSMOS_INTERNAL=0\n\nhf download nvidia/Cosmos3-Nano-Sim-Auto \\\n  --local-dir checkpoints/Cosmos3-Nano-Sim-Auto\n\nhf download nvidia/Cosmos3-Nano-Transfer-Auto \\\n  --include 'assets/sample_data/*' \\\n  --local-dir checkpoints/Cosmos3-Nano-Transfer-Auto\n\nAUTO_SAMPLE_ROOT=\"$PWD/checkpoints/Cosmos3-Nano-Transfer-Auto/assets/sample_data\"\n\ntorchrun --standalone --nproc-per-node=4 \\\n  -m cosmos_framework.scripts.inference \\\n  -i \"$AUTO_SAMPLE_ROOT/input-sim-auto.jsonl\" \\\n  -o outputs/sim_auto \\\n  --checkpoint-path checkpoints/Cosmos3-Nano-Sim-Auto \\\n  --cp-size 4 --cfgp-size 1 \\\n  --no-use-torch-compile --no-diffusion-cache\n```\n\nFor human-controlled rollouts, see [Sim-Auto Interactive Drive](https://flashdreams.org/main/models/cosmos3_nano_sim_auto/). \n\nThe Sim-Auto reference run writes to `outputs/sim_auto/daytime_scene/`. Apply the same output checks described above for Transfer-Auto. \n\n**Robotics Simulation:** \n\n[Cosmos3-Nano-Sim-Transfer](https://huggingface.co/nvidia/Cosmos3-Nano-Sim-Transfer) converts depth video and a prompt into RGB video with varied lighting, textures, and backgrounds while preserving geometry and motion. For robot demonstrations, it changes the third-person view while retaining actions, states, and wrist-camera recordings.\n\n*Video 3. Turn depth, segmentation, edges, or blurred video into detailed RGB observations, using language prompts while the simulator maintains physical state*\n\n[Cosmos3-Nano-Sim-Bimanual](https://huggingface.co/nvidia/Cosmos3-Nano-Sim-Bimanual) generates the next visual observation from the scene and robot actions. It supports interactive two-arm simulation for YAM and AgiBot embodiments, controlled by a policy or human-operated joysticks.\n\nHere, Sim-Transfer generates visuals from a physics simulator’s updated state; Sim-Bimanual generates the next observation directly from the scene and robot actions.\n\n**Surgical Simulation:**\n\nSurgical environments are among the most challenging for classical simulators due to their complex multi-physics nature. Generative simulation offers a complementary approach: rather than explicitly modeling scene geometry, tissue deformation, and physical interactions, world foundation models learn visual dynamics directly from synchronized surgical video and robot kinematics.\n\n**Cosmos3-H-Embodied-Nano**, built on Cosmos3 and post-trained on Open-H-Embodiment, generates surgical video from robot actions, predicts actions from video, or generates both together. It takes an initial RGB frame, a task prompt, the last 16 executed actions, and subsequent robot actions. Developers can use it to generate synthetic data and evaluate policies offline without building a physics model of the surgical scene.\n\n**Cosmos3-H-Dreams** lets developers test how a surgical policy responds as a procedure unfolds, with each action shaping what happens next. Distilled from Cosmos3-H-Embodied-Nano, this causal, few-step model generates frames from live robot kinematics for real-time, closed-loop simulation controlled by a human or learned policy. This gives developers an interactive environment to explore surgical maneuvers and evaluate policies before testing on physical hardware.\n\n## When to use Cosmos NIM for accelerated inference\n\nAlongside the open inference tools, the new Cosmos NIM package supported models in optimized containers for accelerated inference, making Cosmos models easy to deploy.\n\n- **Faster inference, less memory** : Accelerate inference through quantization, speculative decoding, caching, token pruning, and KV-cache-aware routing. Speculative decoding delivers measured mean speedups of roughly 2–3× on H200.\n- **Custom models and live video understanding** : Serve fine-tuned Cosmos checkpoints and process live streams with streaming VLM inference, enabling sub-second alerts with 10x lower latency for faster event response.\n- **Simpler production deployment** : Automatic configuration selection, validated performance and quality across supported GPUs, licensing support, and quarterly feature, optimization, and security updates**.**\n\n**GPU memory requirements**\n\n| Capability | Model | Required GPU VRAM [GiB] |  |  | \n|---|---|---|---|---|\n|  |  | BF16 | FP8 | FP8 + offloading | \n| Reasoning | Cosmos3-Edge | 16 | 14.5 | – | \n|  | Cosmos3-Nano | 23.1 | 23.1 | – | \n|  | Cosmos3-Super | 135 | 67 | – | \n| Generation | Cosmos3-Edge-DROID | 14 | – | – | \n|  | Cosmos3-Nano | 58 | 44 | 31 | \n|  | Cosmos3-Super | 150 | 93 | 38 | \n\n*Table 1. GPU memory requirements per model per task*\n\n**How to Launch Cosmos NIM**\n\nChoose a GPU and model configuration using the tables above, and follow the [deployment guide](https://github.com/NVIDIA/cosmos/blob/c8353e18c607a1917eb1103557b5029060807bba/cookbooks/cosmos3/nim/deployment.md) to install the required NVIDIA driver, Docker, and NVIDIA Container Toolkit on Linux. Sign in to [NGC API Keys](https://org.ngc.nvidia.com/setup/api-keys), select **Generate Personal Key**, and enable **NGC Catalog** access. Set the key in your terminal as `NGC_API_KEY`:\n\n```\nexport NGC_API_KEY='paste-your-key-here'\n```\n\nThe following commands launch Nano Generator on GPU 0. Ensure curl and python3 are installed and sufficient disk space is available for the image and model cache. The cache permissions shown are for local development.\n\n```\n: \"${NGC_API_KEY:?Export your NGC API key first}\"\nprintf '%s' \"$NGC_API_KEY\" | docker login nvcr.io \\\n --username '$oauthtoken' --password-stdin\n\nexport NIM_IMAGE='nvcr.io/nim/nvidia/cosmos3:2.0.0'\nexport LOCAL_NIM_CACHE=\"$HOME/.cache/nim/cosmos3\"\nmkdir -p \"$LOCAL_NIM_CACHE\"\nchmod a+rwX \"$LOCAL_NIM_CACHE\"\n\ndocker run -d --name cosmos3-generator \\\n --gpus '\"device=0\"' \\\n --shm-size 16g \\\n --ulimit memlock=-1 \\\n --ulimit stack=67108864 \\\n --ulimit nofile=65536:65536 \\\n -p 8000:8000 \\\n -e NGC_API_KEY \\\n -e NIM_MODEL_TYPE=generator \\\n -e NIM_MODEL_VARIANT=nano \\\n -e NIM_PERF_PROFILE=latency \\\n -v \"$LOCAL_NIM_CACHE:/opt/nim/.cache\" \\\n \"$NIM_IMAGE\"\n```\n\nWait for readiness (~few minutes) , then inspect the selected model and profile:\n\n```\ncurl -fsS http://localhost:8000/v1/health/ready\ncurl -fsS http://localhost:8000/v1/metadata | python3 -m json.tool\n```\n\nTo inspect startup progress, run `docker logs -f cosmos3-generator` in another terminal. Now you are ready to generate.\n\n**Choose latency or throughput:**\n\n- latency (default)**:** Prioritizes finishing one request sooner. Choose it for interactive experimentation; supported multi-GPU configurations use GPUs together on a request.\n- throughput**:** Prioritizes completing more requests over time. Choose it for dataset generation or concurrent users; independent replicas process requests in parallel.\n\nSet `NIM_PERF_PROFILE` at container launch. These settings apply to Generator; throughput benefits require concurrent requests. Latency supports 1, 2, 4, or 8 GPUs, while throughput requires 2, 4, or 8 compatible GPUs. Changing only `NIM_PERF_PROFILE` to throughput in the single-GPU example will fail profile selection.\n\n## Get started\n\nExplore [Cosmos Framework](https://github.com/NVIDIA/cosmos-framework) and use its agent skills and reference recipes to train, customize, and optimize models for your application. Start with a simulation checkpoint that matches your task, reproduce the reference workflow, and adapt it to your sensors, actions, and data. \n\nRead the full [technical report](https://research.nvidia.com/labs/cosmos-lab/cosmos-simulation/technical-report.pdf).\n\nJoin us on [Thursday, Nov. 5](https://www.youtube.com/watch?v=8lNoGg_j8S8), for a livestream on how to use Cosmos Framework.\n\nDeploy Cosmos models for reasoning and generation using New Certified Cosmos NIM for best performance on 20+ NVIDIA GPUs.", "url": "https://wpnews.pro/news/build-world-models-for-interactive-simulation-with-nvidia-cosmos", "canonical_source": "https://developer.nvidia.com/blog/build-world-models-for-interactive-simulation-with-nvidia-cosmos/", "published_at": "2026-11-02 17:00:00+00:00", "updated_at": "2026-11-02 17:21:59.598071+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-research", "ai-infrastructure", "robotics", "ai-tools"], "entities": ["NVIDIA", "NVIDIA Cosmos", "Cosmos Framework", "Cosmos NIM", "NVIDIA Alpamayo", "GR00T", "Qwen", "Hugging Face"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/build-world-models-for-interactive-simulation-with-nvidia-cosmos", "markdown": "https://wpnews.pro/news/build-world-models-for-interactive-simulation-with-nvidia-cosmos.md", "text": "https://wpnews.pro/news/build-world-models-for-interactive-simulation-with-nvidia-cosmos.txt", "jsonld": "https://wpnews.pro/news/build-world-models-for-interactive-simulation-with-nvidia-cosmos.jsonld"}}