{"slug": "integrate-nvidia-omniverse-rtx-sensor-simulation-into-existing-apps", "title": "Integrate NVIDIA Omniverse RTX Sensor Simulation Into Existing Apps", "summary": "NVIDIA has released ovrtx, an RTX sensor simulation library now available as prerelease software on GitHub, enabling developers to generate camera, lidar, radar, and related sensor outputs from OpenUSD content within existing applications. Part of the NVIDIA Omniverse libraries now included in the NVIDIA Agent Toolkit, ovrtx provides a lightweight C and Python SDK that lets applications and AI agents inspect scenes, generate sensor data, and test changes while keeping control inside the host application. The library supports integration into workflows such as PTC Onshape Render Studio and SimReady Blender, helping developers bring physical AI capabilities into design, robotics, autonomous vehicles, and industrial digital twin applications.", "body_md": "Developers building 3D, design, simulation, robotics, and industrial digital twin applications need ways to bring [physical AI](https://www.nvidia.com/en-us/glossary/generative-physical-ai/) capabilities into the tools and services they already use. Many of these workflows already depend on [OpenUSD](https://www.nvidia.com/en-us/omniverse/usd/) scenes, [simulation-ready (SimReady)](https://www.nvidia.com/en-us/glossary/simready/) assets, [Blender](https://www.blender.org/)-based workflows, CAD pipelines, or domain-specific app stacks. The challenge is how to provide applications and AI agents with callable tools to inspect scenes, generate sensor outputs, test changes, and return evidence developers can trust.\n\n[NVIDIA Omniverse libraries](https://developer.nvidia.com/omniverse?size=n_12_n&sort-field=featured&sort-direction=desc), now part of [NVIDIA Agent Toolkit](https://nvidianews.nvidia.com/news/nvidia-agent-toolkit-expands-with-new-omniverse-libraries-putting-ai-agents-to-work-building-simulation-ready-worlds), provide modular building blocks for adding these capabilities into existing applications. [ovrtx](https://github.com/nvidia-omniverse/ovrtx) is the RTX sensor simulation library in that stack, now available as prerelease software on GitHub. It provides developers with a lightweight C and Python SDK for generating camera, lidar, radar, and related sensor outputs from OpenUSD content while keeping control inside the host application.\n\nWithin the Omniverse libraries stack, ovstage provides a shared runtime stage for OpenUSD scene data, enabling libraries such as ovphysx and ovrtx to exchange simulation state, scene changes, and prim identities through a common data substrate.\n\nThis post walks through how ovrtx integrates into existing workflows as part of the Omniverse libraries model. Steps include loading OpenUSD content, configuring render and sensor outputs, stepping from an application loop, reading results back into the app, and applying the same model in a PTC Onshape Render Studio example and a SimReady Blender sample workflow.\n\n## How do NVIDIA Omniverse libraries help application developers?\n\nSimulation and rendering are becoming part of the application layer for a wide range of workflows including design, robotics, [autonomous vehicles (AVs)](https://www.nvidia.com/en-us/solutions/autonomous-vehicles/), industrial digital twins, and physical AI.\n\nDevelopers and teams need to visualize OpenUSD scenes, generate physically grounded data, test perception systems, validate behavior, and view results inside the tools they use.\n\nThese include CAD systems, Blender-based workflows, PLM or PDM workflows, OpenUSD asset pipelines, internal simulation stacks, and domain-specific applications.\n\nA previous post, [Integrate Physical AI Capabilities into Existing Apps with NVIDIA Omniverse Libraries](https://developer.nvidia.com/blog/integrate-physical-ai-capabilities-into-existing-apps-with-nvidia-omniverse-libraries/), introduces a library-first approach for adding Omniverse capabilities to existing application stacks. That post covers how standalone libraries can expose sensors, physics, and storage capabilities through focused APIs, giving teams more control over where those capabilities run in their own tools and services. ovrtx applies that approach specifically to Omniverse RTX sensor simulation and visualization.\n\nAs [announced at SIGGRAPH 2026](https://nvidianews.nvidia.com/news/nvidia-launches-omniverse-libraries-putting-ai-agents-to-work-building-simulation-ready-worlds), NVIDIA Agent Toolkit now includes Omniverse libraries, with ovrtx serving as the RTX sensor simulation layer. It helps applications and agents generate camera, lidar, radar, semantic segmentation, and visual preflight outputs from OpenUSD scenes. Used with other Omniverse libraries, it becomes part of a tool-driven workflow for preparing and validating simulation-ready 3D content.\n\nThat application control matters whenever sensor simulation needs to fit inside an existing workflow. For physical AI teams, those outputs can support synthetic data generation (SDG), perception testing, robotics workflows, simulation-ready digital twins, and design review.\n\nWith ovrtx now available as pre-release software on GitHub, you can inspect the C and Python APIs, run the examples, test OpenUSD scene loading, and evaluate how RTX sensor outputs fit into your own application loop.\n\n## How to integrate ovrtx into existing workflows\n\nThe core principle behind ovrtx is straightforward. As a developer, you can bring NVIDIA Omniverse RTX sensor simulation into your own applications while maintaining control of the application architecture, UI, data model, and workflow. The lightweight C and Python SDK is designed for projects that need a real-time 3D viewport, RTX rendering, or sensor simulation, including camera, lidar, radar, ultrasonic, and related outputs.\n\nA typical integration follows five steps:\n\n- Create and configure an ovrtx renderer.\n- Load or reference OpenUSD content from the application’s existing pipeline.\n- Define RenderProducts and RenderVars for the camera, lidar, radar, semantic segmentation, or rendered outputs the workflow needs.\n- Step rendering or sensor simulation from the application loop.\n- Map the results back to CPU or GPU memory so the app can display, save, analyze, or pass them to another process.\n\nA pseudo-code example is shown below—the host application owns the workflow.\n\n```\nrenderer = ovrtx.Renderer()\nrenderer.open_usd(app_scene_or_usd_layer)\n\nrender_products = {\"/Render/Camera\"}  # Add lidar, radar, or other outputs as needed.\n\nwhile app.is_running():\n    app.update_scene_state()\n\n    products = renderer.step(\n        render_products=render_products,\n        delta_time=app.frame_delta_seconds,\n    )\n\n    for product in products.values():\n        for frame in product.frames:\n            color = frame.render_vars[\"LdrColor\"].map(device=ovrtx.Device.CPU)\n            app.use_render_output(color)\n```\n\nControl remains in the host application. The app owns the loop; ovrtx produces the requested render or sensor outputs; and the results return to the workflow for display, storage, analysis, or downstream processing.\n\n## Core components in the ovrtx integration flow\n\nThe host application uses a few core pieces when it calls ovrtx: the Omniverse RTX renderer, an OpenUSD stage, and RenderProducts and RenderVars that define the outputs to generate.\n\n### Omniverse RTX Renderer\n\n[Omniverse RTX Renderer](https://docs.omniverse.nvidia.com/materials-and-rendering/latest/rtx-renderer.html) is the core technology exposed by the ovrtx library. It takes advantage of NVIDIA RTX, combining “on chip” RT Cores in NVIDIA RTX GPUs with AI acceleration enabled by Tensor Cores. NVIDIA Omniverse RTX enables real-time, physically-based rendering and simulation of light transport in 3D environments, using flexible physically-based lighting and materials to model cameras, lidar, radar and related sensor outputs.\n\n### ovstage\n\novstage is the shared runtime stage for Omniverse libraries. It provides C and Python APIs for reading, writing, querying, and managing OpenUSD scene data such as transforms, materials, hierarchy, metadata, and simulation state across CPU and GPU memory. In a multilibrary workflow, ovstage allows ovphysx to write updated simulation states and ovrtx to read changed scene data for sensor simulation—without each library maintaining a separate scene representation.\n\n### OpenUSD\n\nNVIDIA has adopted OpenUSD as the foundational data layer for NVIDIA Omniverse and simulation workflows. OpenUSD defines a standard data representation for 3D scenes and provides C++ and Python APIs to read and write those scenes. Data in OpenUSD is organized into a scene graph that ovrtx can render. USD stages can define and handle complex geometry, photometric lights, OpenPBR materials, cameras, and other sensors.\n\n### RenderProducts and RenderVars\n\nRenderProducts and RenderVars provide a USD-native way to define the outputs to obtain from a renderer or sensor simulator.They provide a lot of flexibility to specify output types, resolutions, formats and other parameters. A RenderProduct is associated with one or more sensors, and ovrtx generates the outputs defined by the RenderVars associated with it. RenderProducts and RenderVars can be defined programmatically and stored in the USD scene or defined at runtime using the ovrtx C or Python APIs.\n\n## Application flow\n\nThink of ovrtx as an on-demand service your app calls when it needs to visualize an OpenUSD scene or generate render and sensor outputs. The first steps, creating a renderer, loading OpenUSD content, and configuring RenderProducts and RenderVars, usually happen once or infrequently. The final steps, stepping rendering or sensor simulation and reading outputs, typically run inside the application loop.\n\n### Create and configure an ovrtx renderer\n\nYour existing app, whether Python, C, or C++, starts by creating a renderer instance and setting basic options it needs.\n\n- Initialize ovrtx and construct a renderer object, pointing it to the correct GPUs and any global configuration options.\n- If you have multiple views or sensors, you still have a single renderer instance coordinating them. Multiple RenderProducts will be attached later.\n\n```\nrenderer = ovrtx.Renderer()\n```\n\n### Load or reference OpenUSD content\n\nNext, the renderer needs to know which USD stage to draw from.\n\n- Point the renderer at an OpenUSD file or URL with\n`open_usd`\n\n, load USDA directly from a string with open_usd_from_string, or compose additional referenced content with`add_usd_reference`\n\n. - If your app already has an Omniverse or USD pipeline, ovrtx can read that OpenUSD content directly instead of requiring the app to rebuild the scene in another format.\n\n```\nrenderer.open_usd(\"warehouse_scene.usd\")\n# or compose additional content into the stage:\n# renderer.add_usd_reference(\"extra_props.usd\", prefix=\"/World/Props\")\n```\n\n### Configure render and sensor outputs\n\nWith a renderer and stage in place, the app displays which outputs it needs.\n\nDefine RenderProducts\n\n- Example: An RGB camera on a robot, a top‑down surveillance camera, a spinning lidar sensor on a vehicle.\n- Each RenderProduct is tied to a camera or sensor prim path in the USD stage and a resolution.\n\nAttach RenderVars. For each RenderProduct, specify which outputs ovrtx should generate. Examples include:\n\n- Color, such as LdrColor or HdrColor\n- Depth or distance\n- Surface normals\n- Semantic segmentation\n- Lidar point clouds\n- Radar detections\n- Other sensor‑specific tensors\n\n### Step rendering or sensor simulation from the application loop\n\nThe app calls ovrtx from its main loop, which may already manage physics, animation, user input, networking, or other simulation state.\n\nAfter the app updates the OpenUSD stage, it advances rendering or sensor simulation by passing a time increment. ovrtx then generates the requested outputs for the configured RenderProducts and RenderVars.\n\n### Returning the output to the application\n\nFor each generated frame, the app reads the requested outputs as tensors, either on CPU or GPU memory. It consumes these in other parts of the application logic, as needed.\n\nGetting GPU and CPU views of the data:\n\n- ovrtx exposes each RenderVar as a tensor handle. Wrap it using DLPack, NumPy, or PyTorch utilities.\n- To stay on GPU (to feed a model, for example), keep the buffer on device and pass it directly to PyTorch or Warp without copies.\n- To save images or point clouds, request these outputs on CPU memory, and use other logic to write PNGs, NumPy arrays, and so on.\n\n```\nop = renderer.step_async(                                \n    render_products={\"/Render/Camera\"},\n    delta_time=1.0 / 60,\n)\npending = op.wait()                                        \nproducts = pending.fetch()                              \n  \nfor _name, product in products.items():\n    for frame in product.frames:\n        var = frame.render_vars[\"LdrColor\"].map(   \n            device=ovrtx.Device.CPU\n        )\n```\n\n## When to use ovrtx with other Omniverse libraries\n\novrtx can be used on its own or alongside other NVIDIA Omniverse libraries, depending on what the host application needs to do. Start with ovrtx when the application needs RTX sensor simulation, viewport output, or visualization. Add other libraries when the workflow also needs physics, streaming, storage, or another application service.\n\nApplication need | Use | Function |\n| RTX sensor simulation, viewport output, or visualization | ovrtx | Generates rendered frames and sensor outputs from OpenUSD content |\n| Physics simulation with rendered or sensor output | ovrtx + ovphysx | Adds physics behavior that can be visualized or consumed in the app workflow |\n| Streaming rendered output to a remote or browser-based experience | ovrtx + ovstream | Adds a streaming path for interactive viewing or review |\n| Managing OpenUSD content across storage workflows | ovrtx + ovstorage | Adds a path for reading, writing, or syncing OpenUSD scene data |\n| Sharing OpenUSD scene state across rendering, physics, sensors, and application code | ovstage + ovrtx / ovphysx | Provides a shared runtime stage so libraries can exchange scene data, simulation updates, and changed prim state |\n\n*Table 1. Common ways to use ovrtx with other Omniverse libraries based on the application capability developers need to add*\n\nUse this table as a starting point for deciding what the host application needs. Start with ovrtx when the workflow needs rendered frames, viewport output, or simulated sensor outputs from OpenUSD content. Add ovphysx, ovstream, or ovstorage only when the same workflow also needs physics behavior, interactive streaming, or OpenUSD content management. The following examples show how those choices map to practical integration patterns, from a minimal ovrtx render to interactive viewing and multi-library workflows.\n\n## Examples of ovrtx integration in practice\n\nStart with the [NVIDIA-Omniverse/ovrtx](https://github.com/NVIDIA-Omniverse/ovrtx) GitHub repo for direct SDK examples, setup guidance, and API-focused skills. For agent guidance, reference [NVIDIA Agent Skills](https://github.com/nvidia/skills). To see how Omniverse libraries fit together in early workflows, explore the [NVIDIA-Omniverse/omniverse-labs](https://github.com/NVIDIA-Omniverse/omniverse-labs) GitHub repo, which includes experimental samples, integration experiments, reference patterns, and proof-of-concept apps from the NVIDIA Omniverse team.\n\n### Example: Minimal “hello world”\n\nStart with the [minimal ovrtx example](https://github.com/NVIDIA-Omniverse/ovrtx/tree/main/examples/python/minimalhttps://github.com/NVIDIA-Omniverse/ovrtx/tree/main/examples/python/minimal) in the ovrtx repository. This “hello world” example is a simple smoke test for validating the direct SDK path before adding interaction, sensor outputs, or a larger application shell.\n\n*V ideo 1. Learn how to run the minimal example, connect Python, C, and C++ applications to Omniverse RTX rendering processes, and generate sensor outputs for robotics and synthetic data workflows*\n\nThe example shows how to create a renderer, load an OpenUSD scene, render a single image, and copy the results back to the CPU for writing as a PNG. Pass the `--png`\n\nflag and the output is written to `_output/render.png`\n\n, which can be inspected with any image viewer. Without the flag, the example opens the rendered image directly in your default image viewer.\n\n**Use this for: **Adding RTX sensor simulation capabilities to an existing application without building a full Omniverse Kit app.\n\n### Example: Build an interactive USD viewer\n\nUse the [Omniverse Realtime Viewer skill](https://github.com/NVIDIA/skills/tree/main/skills/omniverse-realtime-viewer) when the goal is to build a viewer application around OpenUSD content, especially when the workflow needs browser-streamed access to a GPU-backed renderer. ovrtx remains the rendering layer, while the viewer adds the surrounding app structure: scene loading, camera controls, selection, render settings, UI, and local desktop or browser-streamed delivery. For lower-level C++ viewport integration, the [vulkan-interop example](https://github.com/NVIDIA-Omniverse/ovrtx/tree/main/examples/c/vulkan-interop) shows how to display ovrtx output in a real-time viewport using Vulkan and CUDA.\n\nFor browser-streamed workflows, a server owns the ovrtx renderer and OpenUSD stage access, then uses ovstream to send rendered frames to a browser UI. For a hands-on path, use the [Brev launchable](https://brev.nvidia.com/launchable/deploy?launchableID=env-3EHjQXkUNYv2pOa3idjBeJOauvH) to deploy and test a real-time viewer application built with the ovrtx and ovstream libraries.\n\nTo learn more, read the [agent-assisted integration walkthrough](https://github.com/NVIDIA-Omniverse/omniverse-labs/blob/main/articles/ovrtx_agents/ovrtx_agent_article.md) on the NVIDIA-Omniverse/omniverse-labs GitHib repo.\n\n**Use this for:** Developing a streaming USD viewer with RTX rendering, browser-based camera controls, and a GPU-backed deployment path using NVIDIA Omniverse libraries and an AI coding agent.\n\n### Example: Adding physics with ovphysx\n\nAfter scene visualization, the next step is often making it behave physically, with gravity, collisions, friction, and other simulation behavior. That is where ovphysx comes in. Like ovrtx, ovphysx can be called from an application workflow so the app can simulate behavior and visualize the result in a real-time viewport.\n\nThe [Omniverse-Labs ovrtx plus ovphysx sample](https://nvidia-omniverse.github.io/omniverse-labs/projects/ovrtx-ovphysx-livestream/) shows this combined workflow in practice. The sample uses a two-process architecture: the viewer process is owned by ovrtx, while a separate ovphysx worker streams pose data back to the application loop.\n\n**Use this for:** Adding physics simulation to a rendered or sensor-aware OpenUSD workflow.\n\n### Example: SimReady Blender sample workflow\n\nThe SimReady Blender sample workflow, on display at SIGGRAPH 2026, shows how software makers can use Omniverse libraries, now part of NVIDIA Agent Toolkit, to add agent-ready simulation capabilities—including NVIDIA RTX sensor simulation, physics, and validation—into existing 3D applications while keeping creators in control. This is now openly available as a blueprint for integrating Omniverse libraries into Blender.\n\n**Use this for: **Understanding how Omniverse libraries can help software makers add agent-ready simulation capabilities into existing 3D applications while keeping creators and developers in control.\n\nDesigners and engineers who want to run this kind of agent workflow locally can deploy it on [NVIDIA DGX Station using NVIDIA NemoClaw](https://blogs.nvidia.com/blog/siggraph-news-2026/#nemoclaw-dgx-station). NemoClaw packages the Nemotron 3 Ultra open frontier model, a secure agent runtime, and Omniverse libraries into a complete local stack—a personal AI workstation with access to Blender and simulation tools.\n\n## How to integrate ovrtx into cloud-native design workflows\n\nPTC [Onshape](https://www.ptc.com/en/products/onshape) Render Studio team is exploring how ovrtx can fit into cloud-native CAD and PDM workflows, where product data, collaboration tools, rendering needs, and established user experiences already come together. That makes it a useful validation point for showing how NVIDIA Omniverse RTX rendering can fit into an existing application workflow while preserving the architecture, data model, and product experience users already rely on.\n\nThis early work uses Onshape content to explore how ovrtx can support cloud-native rendering workflows with more complex product design geometry. ovrtx enables integration into existing systems, so teams can evaluate how it fits into the services and application layers they already use, instead of treating rendering as a separate environment.\n\nFor the PTC Onshape Render Studio team, ovrtx helps address three practical integration needs:\n\n**Application-controlled rendering:** The host application controls when rendering happens, how outputs are configured, and how results return to the workflow.**Cloud-native integration:** Rendering capabilities can be exposed through the application or service layer teams already use, including cloud-native service workflows.**OpenUSD-based scene exchange:** ovrtx renders from OpenUSD scene data, giving teams a practical way to connect product design geometry, visualization, and downstream simulation workflows.\n\nFor application teams, the takeaway is that ovrtx can make RTX sensor simulation and visualization part of the workflow they already own. The PTC work shows how a lower-level API can support production-oriented exploration inside cloud-native design systems, where rendering must stay connected to product data, collaboration workflows, and the application experience.\n\nAccording to Brendan Hack, Senior Director of Software Development at PTC Onshape, “ovrtx gives us a lower-level API path for bringing NVIDIA RTX into systems cloud-native design tools while keeping the application architecture and product experience in our control.”\n\n## Get started with NVIDIA ovrtx\n\nNVIDIA Omniverse libraries provide modular building blocks for integrating RTX sensor simulation, physics, storage, and related physical AI capabilities into existing application stacks. With ovrtx, you can add camera, lidar, radar, semantic segmentation, and visual preflight outputs to OpenUSD-based workflows.\n\nFor teams developing design, simulation, robotics, industrial digital twin, synthetic data generation, or physical AI applications, ovrtx provides a practical path to evaluate RTX sensor simulation inside the tools, pipelines, and application loops they already own.\n\nReady to get started?\n\n**Explore Omniverse libraries:** Visit the[NVIDIA-Omniverse](https://github.com/NVIDIA-Omniverse/)GitHub repo to find tools and skills that help agents and developers add RTX sensor simulation, physics, OpenUSD workflows, storage, and validation to existing applications.**Run the examples:** Review[Omniverse Labs](https://nvidia-omniverse.github.io/omniverse-labs/)for early experimental samples and proof-of-concept projects that show how Omniverse libraries fit together.**Use NVIDIA physical AI agent skills:** Use[agent skills](https://github.com/NVIDIA/skills/tree/main)to turn repeatable setup, simulation, and evaluation tasks into workflows that developers can inspect, adapt, and run with an AI coding agent.**Run locally on DGX Station**: Deploy[NemoClaw on DGX Station](https://build.nvidia.com/station/nemoclaw)in just three steps, in under 30 minutes.", "url": "https://wpnews.pro/news/integrate-nvidia-omniverse-rtx-sensor-simulation-into-existing-apps", "canonical_source": "https://developer.nvidia.com/blog/integrate-nvidia-omniverse-rtx-sensor-simulation-into-existing-apps/", "published_at": "2026-07-20 15:00:00+00:00", "updated_at": "2026-07-20 15:32:37.366775+00:00", "lang": "en", "topics": ["autonomous-vehicles", "robotics", "ai-infrastructure", "developer-tools", "computer-vision"], "entities": ["NVIDIA", "Omniverse", "ovrtx", "GitHub", "OpenUSD", "NVIDIA Agent Toolkit", "PTC Onshape", "Blender"], "alternates": {"html": "https://wpnews.pro/news/integrate-nvidia-omniverse-rtx-sensor-simulation-into-existing-apps", "markdown": "https://wpnews.pro/news/integrate-nvidia-omniverse-rtx-sensor-simulation-into-existing-apps.md", "text": "https://wpnews.pro/news/integrate-nvidia-omniverse-rtx-sensor-simulation-into-existing-apps.txt", "jsonld": "https://wpnews.pro/news/integrate-nvidia-omniverse-rtx-sensor-simulation-into-existing-apps.jsonld"}}