Into the Omniverse: How Developers Turn Ideas Into Simulations With Frontier AI Agents

NVIDIA detailed how developers are using frontier AI models, including GPT-6 Astra and Claude Fable 5, to direct agents that assemble simulation applications on top of NVIDIA Omniverse libraries such as ovphysx, ovstage, ovrtx and ovui. NVIDIA Omniverse product manager Frank DeLise used Astra to build an interactive SimReady warehouse humanoid simulator with first- and third-person views, while simulation technology manager Doyub Kim used Astra to create Zero to Alpamayo, a reusable San Francisco Market Street autonomous-driving test environment, and RTX sensor validation engineer Ashley Reid directed Astra and Claude Fable 5 agents to compare ovrtx camera and raw LiDAR outputs against recorded data for digital twins.

Turning a simulation idea into a working application means assembling assets, connecting physics and rendering, and checking that the scene behaves as intended. Developers are combining frontier AI models with NVIDIA Omniverse libraries https://developer.nvidia.com/omniverse to help carry out that work — building applications for exploring scenarios, investigating failures and improving designs. Developers direct AI agents through natural-language instructions, review results and guide changes. Omniverse libraries provide GPU-accelerated physics, rendering and sensor simulation capabilities. Explore the projects below to see frontier AI models such as GPT-6 Astra at work, and check back for new examples from NVIDIA teams and developers across the ecosystem. Build a Humanoid Simulator for a Warehouse Environment 🔗 https://blogs.nvidia.com/blog/developers-simulation-frontier-ai-agents/ humanoid-simulator 🔗 Explore a gamified, physics-based control of a humanoid robot in first- and third-person view. Before automating warehouse tasks, developers need an interactive simulation environment to explore task behavior and evaluate how the work gets done. Frank DeLise, Omniverse product manager at NVIDIA, used Astra to turn a SimReady https://www.nvidia.com/en-us/glossary/simready/ warehouse and humanoid robot into an interactive simulator with first- and third-person views. DeLise directed Astra to connect NVIDIA Omniverse libraries https://developer.nvidia.com/omniverse for physics ovphysx https://github.com/NVIDIA-Omniverse/PhysX/blob/main/ovphysx/README.md , scene updates ovstage https://github.com/nvidia-omniverse/ovstage , rendering ovrtx https://github.com/NVIDIA-Omniverse/ovrtx and the user interface ovui https://github.com/NVIDIA-omniverse/ovui . He also used Astra with SimReady simready-foundation https://github.com/nvidia/simready-foundation to create the physical scene in simulation. Astra then generated animation and application code to bring those capabilities together. Learn how to prepare and validate SimReady robot assets with frontier AI models and NVIDIA Omniverse libraries. Connect an Autonomous-Driving Testing Workflow 🔗 https://blogs.nvidia.com/blog/developers-simulation-frontier-ai-agents/ autonomous-driving-testing 🔗 Changing a scene, sensor or driving model can significantly affect autonomous-vehicle simulations. Doyub Kim, a manager on the simulation technology team at NVIDIA, asked Astra to build Zero to Alpamayo — a reusable simulation environment based on San Francisco’s Market Street. Kim directed Astra to map out the workflow, then connect asset creation, traffic, Omniverse RTX sensor simulation https://developer.nvidia.com/blog/integrate-nvidia-omniverse-rtx-sensor-simulation-into-existing-apps/ and Alpamayo driving https://www.nvidia.com/en-us/solutions/autonomous-vehicles/alpamayo/ in stages, checking each integration. The resulting prototype was a testing ground for comparing models and tracing how scene or sensor changes affect downstream driving behavior. A separate Cosmos3-Nano experiment varied weather and lighting in recorded simulation videos, allowing Kim to compare the driving model’s responses to the same scenario under different conditions. Use Sensor Differences to Create and Improve Digital Twins 🔗 https://blogs.nvidia.com/blog/developers-simulation-frontier-ai-agents/ sensor-differences 🔗 Comparing recorded and simulated camera and lidar outputs gives developers key performance indicator-based feedback for refining a scene. To test robots and autonomous vehicles, developers need to know how closely simulated sensors match real ones. Ashley Reid, who works on RTX sensor validation at NVIDIA, directed Astra and Claude Fable 5 agents to compare ovrtx https://github.com/NVIDIA-Omniverse/ovrtx camera and raw LiDAR outputs with recorded data. The agents created two digital twins from scratch and improved two existing ones. Over about three days, Reid guided an iterative workflow in which agents measured differences, created or modified OpenUSD scenes, and checked the results. Changes addressed missing objects, geometry and materials, with acceptance depending on camera and LiDAR metrics. This gives developers a way to use measured discrepancies to guide scene creation and improvement. Start by rendering an OpenUSD scene with the ovrtx minimal Python example , then define a sensor measure to compare with recorded data. Robo Olympics: Test Robot Skills With Simulation 🔗 https://blogs.nvidia.com/blog/developers-simulation-frontier-ai-agents/ robo-olympics 🔗 Teaching robots new movements means checking whether those actions work under physical constraints. Tae Kim, who leads NVIDIA Omniverse engineering and product, used sports videos and natural-language instructions to guide Astra in building Robo Olympics, an experimental project that tests simulated Unitree G1 humanoids performing sports movements. Under Kim’s direction, Astra built controllers and refined them through physics trials. Newton Physics Engine https://developer.nvidia.com/newton-physics simulated behavior, the open source NVIDIA Warp https://developer.nvidia.com/warp-python framework accelerated calculations and ovrtx https://github.com/NVIDIA-Omniverse/ovrtx rendered scenes and virtual-camera images. In one experiment, the robot cleared a single hurdle in 64 of 100 simulation trials. The trials gave Kim feedback for improving the robot’s timing and control. Watch Kim explore building simulations with Astra and NVIDIA Omniverse libraries. Test Robotic Disassembly With Computer-Aided Design and Simulation 🔗 https://blogs.nvidia.com/blog/developers-simulation-frontier-ai-agents/ robot-disassembly 🔗 Before a robot can disassemble a product, developers need to know whether its tools can reach and remove the parts. Jens Jebens, a senior product manager for OpenUSD at NVIDIA, directed Astra to model a car suspension in PTC Onshape and configure it in NVIDIA https://developer.nvidia.com/isaac/sim/ Isaac Sim https://developer.nvidia.com/isaac/sim/ . With Astra, Jebens explored computer-aided design https://www.nvidia.com/en-us/glossary/computer-aided-engineering/ and tooling revisions for robots informed by simulation. The agent measured the available space and designed a wrench the robot could use to reach the suspension’s bolts. Jebens reported successful removal of a suspension component in simulation. This connects design and tooling decisions to disassembly results, offering a starting point for robot policy training. Explore the Onshape importer guide . Bring the International Space Station Into the Browser 🔗 https://blogs.nvidia.com/blog/developers-simulation-frontier-ai-agents/ international-space-station 🔗 Turning 3D models into an application requires connecting assets, live data and an interface. Nic Johns, an engineering director at NVIDIA, prompted Astra to assemble NASA assets into an OpenUSD https://www.nvidia.com/en-us/glossary/openusd/ International Space Station model with telemetry. Johns built the application with a single prompt, then used a follow-up prompt to shift the scene to Earth’s daytime side so the planet was visible. The workflow used Blender for asset preparation and Omniverse libraries https://developer.nvidia.com/omniverse for rendering ovrtx https://github.com/NVIDIA-Omniverse/ovrtx , scene runtime ovstage and streaming ovstream . The application brings 3D models and operational data into the browser, with Johns guiding its development through prompts and corrections. Try the Omniverse Real-Time Viewer skill . Turn Captured Rooms Into Testing Environments 🔗 https://blogs.nvidia.com/blog/developers-simulation-frontier-ai-agents/ testing-environments 🔗 A digitally reconstructed room needs editable objects and accurate physical behavior before developers can test interactions. Chirag Majithia, from the Isaac engineering applications team at NVIDIA, directed Astra to turn stereo camera captures into an editable OpenUSD https://www.nvidia.com/en-us/glossary/openusd/ studio. The workflow combined PyCuSFM, FoundationStereo and nvblox for reconstruction, with user review guiding object selection and placement. Astra assembled generated and Blender-authored assets and used USD Content Agents to configure how objects move and interact in simulation. Isaac Sim tests guided collision and contact revisions for doors and drawers. The studio connects captured geometry to interaction testing, making gaps and object behavior easier to inspect. Have a simulation idea? Explore NVIDIA Omniverse libraries https://developer.nvidia.com/omniverse to start building with an AI agent.