{"slug": "build-training-ready-physical-ai-datasets-with-two-prompts-in-nvidia-cosmos", "title": "Build Training-Ready Physical AI Datasets with Two Prompts in NVIDIA Cosmos", "summary": "NVIDIA released the Cosmos Data Factory (CDF), an open collection of five cloud-native blueprints and six composable skills that lets developers build training-ready physical AI datasets with two prompts, the company said. The first prompt clones the CDF Orchestration repository and runs preflight and setup; the second runs a blueprint for one of five use cases, including autonomous vehicle scenario generation, image attribute augmentation, video augmentation across weather and lighting, rare event video generation, and defect image generation for visual inspection. NVIDIA said frontier AI agents can orchestrate the blueprints to enable prompt-to-data without human steering.", "body_md": "[World model](https://www.nvidia.com/en-us/glossary/world-models/) accuracy starts with high-quality, diverse training datasets. Yet capturing rare defects, nighttime incidents, and other challenging scenarios is costly and time-consuming. Turning raw data into training-ready datasets adds complexity, particularly when using disconnected tools and manual processes for curation, labeling, generation, and evaluation.\n\nNVIDIA technology is helping developers more easily build high-quality, diverse training datasets at scale for [physical AI](https://www.nvidia.com/en-us/glossary/generative-physical-ai/). [NVIDIA Cosmos](https://www.nvidia.com/en-us/ai/cosmos/) is an open platform of [frontier models](https://www.nvidia.com/en-us/glossary/frontier-models/), datasets, frameworks, and skills for world model development. As part of this platform, [NVIDIA Cosmos Data Factory (CDF)](https://github.com/NVIDIA/cosmos-data-factory) is a collection of five cloud-native blueprints and six composable skills to accelerate data curation, processing, and [synthetic data generation (SDG)](https://www.nvidia.com/en-us/use-cases/synthetic-data-physical-ai/). Frontier [AI agents](https://www.nvidia.com/en-us/ai/) can be used to orchestrate these blueprints, enabling prompt-to-data without human steering.\n\nThis post walks you through how to curate and augment datasets or create anomalous events for physical AI model training with just two prompts in Cosmos. It features five different use cases.\n\n## Set up and configure Cosmos blueprints and skills: Prompt 1\n\nCosmos Data Factory contains five reference blueprints for common tasks. All five blueprints run using the [CDF Orchestration core skill](https://github.com/NVIDIA/cdf-orchestration), which coordinates workflow steps, deploys the required services, and manages execution on your cluster.\n\nTo set it up, enter the following prompt into your coding agent:\n\n| `Example prompt:``“Clone the CDF Orchestration repository, then run orchestration preflight and setup.”` | \n|---|\n\nPreflight checks your environment without making changes and flags anything missing. Once those checks pass, setup prepares your environment to run the blueprints.\n\n## Run the blueprint for your training data use case: Prompt 2\n\nThis section features five training data use cases for running the blueprint, including example prompts:\n\n- Generate driving scenarios for autonomous vehicle (AV) training\n- Augment image attributes for search and re-identification\n- Augment videos across weather, lighting, and environments\n- Generate rare event videos for model training\n- Generate defect images for visual inspection\n\nIf you already have footage or images and need to curate and add more variety, the first use case in this section covers generating driving scenarios, varying clothing attributes in person images, and transforming videos across weather, lighting, and environmental conditions. The augmentation use cases expand the variety in your existing data, but some training gaps call for creating entirely new events or defects.\n\nYour model might need to recognize a leaking product or a defective circuit board, but your dataset may contain few examples or none. Waiting to capture enough rare cases can take years, and staging them can be dangerous or expensive. The last two use cases cover generating rare event videos and defect images from images or 3D assets to fill these gaps.\n\n### Use case 1: Generate driving scenarios for AV training\n\nThe [Autonomous Vehicle Dataset Generation blueprint](https://github.com/NVIDIA/cdf-orchestration/blob/main/skills/cdf-autonomous-vehicle-dataset-generation/SKILL.md) helps you search large databases of driving scenarios and generate novel datasets for downstream model training and validation tasks. [NVIDIA Cosmos Curator](https://github.com/NVIDIA/cosmos-curator) extracts metadata for your video dataset inputs, and [NVIDIA Cosmos Dataset Search](https://github.com/NVIDIA-Omniverse-blueprints/cosmos-dataset-search) finds relevant videos through those embeddings. The retrieved videos are passed to NVIDIA Cosmos3-Super-Image2Video style transfer, and [Cosmos Evaluator](https://github.com/NVIDIA/cosmos-evaluator) checks the generated videos for physical plausibility.\n\n| `Example prompt:``“Search for driving scenes similar to our input video.mp4, transform them into rainy, snowy, and foggy while preserving layout and motion, and check physical plausibility.”` | \n|---|\n\n### Use case 2: **Augment image attributes for search and re-identification**\n\nThe [Image Attribute Augmentation reference blueprint](https://github.com/NVIDIA/cdf-orchestration/blob/main/skills/cdf-image-attribute-augmentation/SKILL.md) generates variations of person-crops with controlled attribute changes such as clothing type, accessories, and color. An LLM turns your selected attributes into structured modification prompts, used to generate the augmented variations of the source image. Cosmos Evaluator confirms that the requested clothing attributes appear. The blueprint then auto-labels the results with clothing attributes and search queries for person search and re-identification.\n\n| `Example prompt:``“Augment the person crops in <input-bucket>, three clothing variants per person, and write the output to <output-bucket>.”` | \n|---|\n\n### Use case 3: Augment videos across weather, lighting, and environments\n\nThe [Video Data Augmentation reference blueprint](https://github.com/NVIDIA/cdf-orchestration/blob/main/skills/cdf-video-data-augmentation/SKILL.md) transforms your existing footage across weather, lighting, and environmental conditions while preserving scene structure. A [vision lang](https://www.nvidia.com/en-us/glossary/vision-language-models/)[uage model (](https://www.nvidia.com/en-us/glossary/vision-language-models/)[VLM)](https://www.nvidia.com/en-us/glossary/vision-language-models/) describes your source video and combines that with your requested conditions into a structured prompt, which Cosmos3-Nano uses to generate the augmented variant.\n\nCosmos Evaluator checks each generated clip for motion artifacts that aren’t in your source, confirms that the conditions you asked for actually appear, and scores whether the motion is physically plausible. Next, the blueprint auto-labels both your source and generated videos with detection, tracking, and captions. It then provides you with a training-ready dataset.\n\n| `Example prompt:` `“Run video data augmentation on the videos in <input-bucket>, create 3 augmentations for yellow lighting, dim lighting and red table. Write the output to <output-bucket>.”` | \n|---|\n\n### Use case 4: Generate rare event videos for model training\n\nThe [Event Video Generation reference blueprint](https://github.com/NVIDIA/cdf-orchestration/blob/main/skills/cdf-event-video-generation/SKILL.md) uses a source image to create videos of rare events, such as fires, falls, or shoplifting. A VLM combines its description of your scene with your requested event into a structured prompt. Cosmos3-Super-Image2Video generates the clip while preserving the source viewpoint and layout.\n\nCosmos Evaluator checks whether the event appears and whether motion looks physically plausible, retrying failed clips. Passing clips are auto-labeled with detections, tracks, captions, and semantic annotations for downstream training.\n\n| `Example prompt:``“Run the Event Video Generation blueprint on the image in <input-bucket> to generate 3 clips: normal traffic, stalled vehicle, and collision at the intersection. Write the output to <output-bucket>.”` | \n|---|\n\n### Use case 5: Generate defect images for visual inspection\n\nThe [Defect Image Generation reference blueprint](https://github.com/NVIDIA/cdf-orchestration/blob/main/skills/cdf-defect-image-generation/SKILL.md) creates defective examples from clean inspection images or 3D USD assets of parts such as circuit boards, glass panels, or metal surfaces. For 3D assets, [NVIDIA Isaac Sim](https://developer.nvidia.com/isaac/sim?size=n_6_n&sort-field=featured&sort-direction=desc) renders inspection images and masks, and Qwen-Image-Edit creates realistic clean reference images, called golden images.\n\nCosmos 3 Super uses these images, defect masks, and your requested defect types to generate defects while preserving the surrounding part. It ships fine-tuned for printed circuit boards (PCBs), glass, and metal, with further fine-tuning skills available for other use cases. You get golden images, synthetic defect images, and manifests tracing each result to its source and generation settings.\n\n| `Example prompt:``“Run the Defect Image Generation blueprint and generate shift structural defects for images in <input-bucket> with full board coverage. Write the output to <output-bucket>.”` | \n|---|\n\n## How to u**se core agent skills and tools for specific tasks like curation, search, and labeling**\n\nThe blueprints outlined in the previous section are built from six core [agent skills](https://www.nvidia.com/en-us/glossary/agent-skills/): [Auto Labeling](https://github.com/NVIDIA/cdf-auto-labeling/blob/main/skills/cdf-auto-labeling/SKILL.md), [AnomalyGen](https://github.com/NVIDIA/cdf-anomalygen/blob/main/skills/cdf-anomalygen/SKILL.md), [Augmentation](https://github.com/NVIDIA/cdf-augmentation/blob/main/skills/cdf-augmentation/SKILL.md), [Curation and Retrieval](https://github.com/NVIDIA/cdf-curation-and-retrieval/blob/main/skills/cdf-curation-and-retrieval/SKILL.md), [Simulation](https://github.com/NVIDIA/cdf-simulation/blob/main/skills/cdf-simulation/SKILL.md), and [Orchestration](https://github.com/NVIDIA/cdf-orchestration/blob/main/skills/cdf-orchestration-setup/SKILL.md). You can use these skills independently or combine them to build a workflow around your own data and goals.\n\n- **Auto Labeling:** Turns raw video and images into annotated datasets with detection, tracking, captions and scene understanding.\n- **AnomalyGen** : Creates realistic defect images from a handful of examples, with masks, for visual inspection.\n- **Augmentation** : Generates visual variations of your video and images using Cosmos models, for conditions you can’t capture.\n- **Curation and Retrieval** : Finds clips and images in a large collection, using Cosmos Curator and Cosmos Dataset Search.\n- **Simulation:** Renders photorealistic, fully labeled images from 3D scenes using Isaac Sim and Omniverse Replicator.\n- **Orchestration** : Runs reference blueprints on your cluster, deploying the services each one needs and scaling them.\n\n| `Example prompt:`` Clone the auto labeling repo` [`https://github.com/NVIDIA/cdf-auto-labeling/tree/main`](https://github.com/NVIDIA/cdf-auto-labeling/tree/main)`. Perform video auto-labeling to label my videos at <input-bucket>. Save outputs to <output-bucket>. Use the locally deployed model at <endpoint> for VLM and LLM.`  | \n|---|\n\nFor workflows focused on curation and retrieval, use Cosmos Curator to process and auto-label raw video and multimodal data, and Cosmos Dataset Search to find targeted training scenarios across large datasets. Cosmos Curator now supports multicamera video, IMU, and GPS inputs, plus initial lidar support in its sensor library. Reusable Python processing components and Managed Ray on Slurm orchestration help teams connect sensor observations, test curation strategies, and use spare GPU capacity.\n\nPlanned expansion includes additional lidar devices, radar, and other sensor types. Cosmos Dataset Search, now open source under the Apache 2.0 license, combines metadata, scene semantics, spatial relationships, and temporal behavior through multi-embedding retrieval and adaptive agentic reranking (early access). Its modular APIs and agent skills retrieve relevant video segments for training datasets or the augmentation blueprints above.\n\n## **Generate** t**raining data up to 60x** f**aster and** 45**x** c**heaper**\n\nCosmos Data Factory skills and blueprints turn hard-to-collect training data into a compute workload you control. The NVIDIA team measured daily throughput on a cloud server with eight [NVIDIA RTX PRO 6000 Blackwell Series](https://www.nvidia.com/en-us/products/workstations/professional-desktop-gpus/rtx-pro-6000-family/) GPUs at roughly $40 per server-hour, or $960 per day, then divided daily cost by output to estimate cost per asset.\n\n- **A****V** **d****ataset** **g****eneration:** Generates ~130 edge-case clips/day at $2.66 in GPU compute per clip, about 45x cheaper than a fleet crew costing $125 per scenario collected on the road.\n- **Image attribute augmentation:** Augments ~42,649 annotated image pairs/day at $960/day, or $0.023 per pair, about 10x cheaper than the $0.25 average for third-party purchased data.\n- **Video data augmentation:** Augments ~305 annotated clips/day at $3.15 per ~5-minute clip, about 7x cheaper than the $22 average we’ve paid for third-party data.\n- **Defect** **i****mage** **g****eneration:** Dropped dataset assembly time from two months to one day, a 60x speedup.\n- **Event video generation:** Generates ~384 annotated clips/day at $2.50 each and ~4 minutes per clip, creating scarce training examples of fires, fights, and falls without waiting for or recreating dangerous real incidents.\n\nWith Cosmos, training gaps become generation jobs with a measurable compute budget. Note that these figures cover compute only; source data preparation, setup, agent use, storage, and human review may add costs.\n\n## What results and projected outcomes are organizations reporting with Cosmos?\n\nSeveral organizations are using Cosmos skills and blueprints to improve testing, detection, and search accuracy across autonomous vehicles, visual inspection, and robotics while targeting lower costs and effort.\n\n- Improving recall on real-world anomalies by up to [24% with synthetic data and estimated 1,400x return on data-generation](https://encord.com/data-labs/nvidia-anybotics-synthetic-data/) cost at an offshore oil and gas site.\n- Targeting up to [85% less manual engineering effort in automated optical inspection (AOI) program generation – with 40% fewer false calls](https://news.siemens.com/en-us/siemens-nvidia-pcba-digital-twin-factory/) .\n- Reducing repeated physical test drives by [up to 90% and shortening AI regression test cycles by up to 85%](https://voxel51.com/blog/volkswagen-physical-ai-data-flywheel) (expected).\n- Going to market in [under two weeks with Cosmos Curator](https://www.centific.com/blog/verifying-robot-data-at-scale-nvidia-cosmos-curator-cosmos-3-in-the-centific-physical-ai-stack) compared to an estimated six months (time required to build an equivalent pipeline in-house).\n\n## Get started with NVIDIA Cosmos\n\nTo accelerate data curation, processing, and synthetic data generation for your application, check out [NVIDIA Cosmos Data Factory](https://github.com/NVIDIA/cosmos-data-factory) blueprints and composable skills. Start with a blueprint that matches your use case, follow the setup guide, and use your AI agent to run the workflow with your data. Customize your pipeline with individual skills to generate high-quality, annotated training datasets.\n\nTo learn more, join us on November 19 for the livestream, [Build Training-Ready Datasets in Two Prompts](https://www.youtube.com/watch?v=c5or9gzM6Zg). Learn how to use Cosmos to curate and augment datasets for AVs, robotics, and vision AI agents. Featuring Voxel51 and Encord, the livestream will highlight how Porsche uses Cosmos Data Factory through the Voxel51 platform and how Encord worked with ANYbotics.", "url": "https://wpnews.pro/news/build-training-ready-physical-ai-datasets-with-two-prompts-in-nvidia-cosmos", "canonical_source": "https://developer.nvidia.com/blog/build-training-ready-physical-ai-datasets-with-two-prompts-in-nvidia-cosmos/", "published_at": "2026-11-18 17:00:00+00:00", "updated_at": "2026-11-18 17:16:10.549894+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-agents", "ai-tools", "robotics", "autonomous-vehicles"], "entities": ["NVIDIA", "NVIDIA Cosmos", "NVIDIA Cosmos Data Factory", "CDF Orchestration", "NVIDIA Cosmos Curator", "NVIDIA Cosmos Dataset Search"], "also_reported_by": [], "takeaway": "NVIDIA's CDF Orchestration skill runs preflight checks before setup and coordinates five blueprints and six skills, so a developer can generate AV scenarios, augmented images and videos, rare event videos, or defect images from a coding agent with two prompts.", "recommendation": null, "alternates": {"html": "https://wpnews.pro/news/build-training-ready-physical-ai-datasets-with-two-prompts-in-nvidia-cosmos", "markdown": "https://wpnews.pro/news/build-training-ready-physical-ai-datasets-with-two-prompts-in-nvidia-cosmos.md", "text": "https://wpnews.pro/news/build-training-ready-physical-ai-datasets-with-two-prompts-in-nvidia-cosmos.txt", "jsonld": "https://wpnews.pro/news/build-training-ready-physical-ai-datasets-with-two-prompts-in-nvidia-cosmos.jsonld"}}