Building a physics-based simulation usually requires a tedious cycle of assembling assets, configuring rendering, and debugging physics interactions. Now, developers are using frontier models like GPT-6 Astra and Claude Fable 5 to orchestrate these workflows by calling specific Omniverse libraries to generate application code and refine digital twins.
How do these agents actually build simulations? #
The workflow relies on agents acting as orchestrators for NVIDIA's GPU-accelerated tools. Instead of manual setup, developers provide natural-language goals, and the agents connect the necessary libraries to make the scene functional.
For a warehouse humanoid simulator, Astra was used to bridge several specific components:
- ovphysx for physics calculations
- ovstage for scene updates
- ovrtx for rendering
- ovui for the user interface
By combining these with the simready-foundation for physical scene creation, the agent generated the animation and application code required to create an interactive environment with both first- and third-person perspectives.
Can agents automate autonomous driving tests? #
Yes, but it happens in stages. In the "Zero to Alpamayo" project, Astra mapped out a full workflow to recreate a section of San Francisco’s Market Street. The agent handled the integration of asset creation, traffic systems, and RTX sensor simulation.
The critical part of this process is the verification loop. The agent connects the Alpamayo driving model in stages, checking each integration point. This allows developers to trace exactly how a change in a sensor or a specific scene element alters the driving behavior. Additionally, Cosmos3-Nano is used to vary lighting and weather in recorded videos to see how the model reacts to the same scenario under different visual conditions.
How are digital twins being refined with AI? #
The gap between simulated sensor data and real-world recordings is a major pain point. Using a combination of Astra and Claude Fable 5, developers are now automating the alignment of ovrtx camera and raw LiDAR outputs against recorded data.
The iterative process looks like this:
- Render the OpenUSD scene using the
ovrtxminimal Python example. - Define a specific sensor measure to compare against the real-world recording.
- The agents measure the discrepancy and modify the OpenUSD scene (adjusting geometry or materials).
- The agent checks the results against LiDAR and camera metrics.
In one instance, this iterative loop took about three days to resolve missing objects and material errors, effectively using measured discrepancies to guide the scene's improvement.
Which tools are available for agentic workflows? #
For those looking to implement this, the NVIDIA Agent Toolkit provides "skills" that guide bounded workflows. These are designed to handle specific tasks like converting CAD assets into SimReady USD (complete with physics and materials) or optimizing scenes.
The fastest way to get these running is to clone the relevant library from GitHub or use a predefined blueprint for a full physical AI workflow. If you are starting from scratch, the ovrtx minimal Python example is the primary entry point for validating sensor data against real-world benchmarks.
Next Mitigating Data Center Energy Waste with AI →
All Replies （1） #
Want a live back-and-forth? Join the global AI chat room — login to talk. Since GPT-6 Astra calls Omniverse libraries, how do you verify the generated code's physics accuracy?