{"slug": "make-long-running-nvidia-tensorrt-engine-builds-observable-and-cancelable-in-or", "title": "Make Long-Running NVIDIA TensorRT Engine Builds Observable and Cancelable in Python or C++", "summary": "NVIDIA TensorRT's IProgressMonitor API lets developers make long-running engine builds observable and cancelable in Python and C++, preventing wasted GPU-hours in agent workflows. The interface provides phase_start, step_complete, and phase_finish callbacks, with step_complete returning a boolean to abort the build. A tutorial demonstrates a drop-in implementation with Ctrl-C cancellation and progress streaming for IDEs, services, or agent runtimes.", "body_md": "A TensorRT engine build can take seconds to many minutes. Large strongly typed models, deep tactic search, and a cold timing cache on a brand-new GPU SKU can leave developers, end users, or AI agents staring at a frozen terminal with no idea whether to wait, retry, or kill the process. Most NVIDIA TensorRT integrations report nothing during a build or provide no way to abort early. In a long-running agent workflow, this turns into wasted GPU-hours and stuck sessions.\n\nTensorRT provides `IProgressMonitor`\n\n, an API for fixing this issue, and it has been in `NvInfer.h`\n\nfor several releases. This tutorial walks through a minimal drop-in implementation for Python and C++, adds a cancel path that responds to Ctrl-C or a programmatic stop signal from an outer event loop, and shows where to surface the resulting progress stream so an IDE, a service, or an agent runtime can use it.\n\nEvery code block in this post is lifted from or modeled on two NVIDIA-maintained OSS samples:\n\n**Python:**`samples/python/simple_progress_monitor/`\n\n(ResNet-50, strongly typed network)**C++:**`samples/sampleProgressMonitor/`\n\n(MNIST)\n\n## What IProgressMonitor gives you\n\n`IProgressMonitor`\n\nis an abstract base class that TensorRT calls during the engine build. You subclass it and override three methods. The shape is identical in Python and C++; only the spelling differs.\n\n| Concept | Python method | C++ method | What you do |\n|---|---|---|---|\n| Phase entered | `phase_start(phase_name, parent_phase, num_steps)` | `phaseStart(phaseName, parentPhase, nbSteps)` | Reserve a progress row and record `num_steps` . |\n| Step within phase complete | `step_complete(phase_name, step) -> bool` | `stepComplete(phaseName, step) -> bool` | Advance the bar. Return `False` /`false` to cancel the build. |\n| Phase exited | `phase_finish(phase_name)` | `phaseFinish(phaseName)` | Tear down the row. |\n\n*Table 1. The*\n\n`IProgressMonitor`\n\ninterface mirrored across Python and C++. The three methods have identical semantics, and `step_complete`\n\nis the only callback whose return value changes the builder’s behaviorA phase whose `parent_phase`\n\nis non-null is nested inside another phase, so the monitor sees a tree of progress rather than a flat list. The implementation must be thread-safe because TensorRT can call the same monitor instance from multiple internal threads.\n\nWire the monitor to the builder by setting it on the `IBuilderConfig`\n\n. It is a single call in either language:\n\n```\nconfig.progress_monitor = MyMonitor()      # Python\n\nconfig-&gt;setProgressMonitor(&amp;myMonitor);     // C++\n```\n\nRead the diagram from top to bottom. The builder opens the *Building Engine* phase with `phase_start`\n\n, then opens *Tactic Selection* nested inside it with its `parent_phase`\n\npointing back at *Building Engine*. As the build proceeds, the builder calls `step_complete`\n\n(the solid arrows) and your monitor returns a Boolean (the dashed arrows): `true`\n\nlets the build continue and `false`\n\nrequests cancellation. In the run shown here, the monitor returns `false`\n\nat step 47, which is the red cancel path, and the builder stops issuing new steps and unwinds. It calls `phase_finish`\n\nearly on *Tactic Selection* and then on *Building Engine*, closing every active phase in reverse order.\n\n## What this tutorial builds\n\nThis tutorial shows how to implement `IProgressMonitor`\n\nin Python and C++, add cancellation through `step_complete`\n\n, and route progress updates to a terminal, IDE, service, or agent runtime.\n\n## Prerequisites\n\n- One NVIDIA GPU.\n- TensorRT (current OSS release) and its Python bindings, or a build of the C++ samples.\n- Python 3.10 or newer (Python path).\n- The TensorRT sample data: ResNet-50 ONNX for Python and MNIST ONNX for C++. Both ship with the sample-data archive or are mounted under\n`/usr/src/tensorrt/data`\n\nin the official NGC containers. - A terminal that supports ANSI virtual-terminal escapes. Any modern Linux shell qualifies; Windows Terminal works if VT is enabled.\n\n### 1. Subclass `IProgressMonitor`\n\nin Python\n\nThe subclass is small. It only tracks which phases are active and how many steps each phase contains.\n\n``` python\nimport tensorrt as trt\nfrom dataclasses import dataclass, field\nfrom threading import Lock\n \n@dataclass\nclass _PhaseState:\n    num_steps: int\n    current_step: int = 0\n    parent: str | None = None\n \nclass RichProgressMonitor(trt.IProgressMonitor):\n    def __init__(self):\n        super().__init__()\n        self._lock = Lock()\n        self._phases: dict[str, _PhaseState] = {}\n        self._cancelled = False\n\t     self._rendered_lines = 0\n \n    def phase_start(self, phase_name, parent_phase, num_steps):\n        with self._lock:\n        \tself._phases[phase_name] = _PhaseState(\n            \tnum_steps=num_steps, parent=parent_phase\n        \t)\n        \tself._render()\n \n    def step_complete(self, phase_name, step) -> bool:\n        with self._lock:\n        \tif phase_name in self._phases:\n                self._phases[phase_name].current_step = step\n        \tself._render()\n        \treturn not self._cancelled\n \n    def phase_finish(self, phase_name):\n        with self._lock:\n        \tself._phases.pop(phase_name, None)\n        \tself._render()\n```\n\nTwo things to notice. First, the `Lock`\n\nis not optional. TensorRT will call into the monitor from multiple internal threads, and rendering from a thread that doesn’t own the state will tear the display. Second, `step_complete`\n\nis the only callback that can stop the build. `phase_start`\n\nreturns `None`\n\n, so you cannot reject a phase before it begins. The earliest cancellation point is the first `step_complete`\n\nof that phase.\n\n**2. Render nested progress bars with virtual-terminal escapes**\n\nThe renderer is the part that varies most by environment, so this section gives the shape and points to the upstream sample for the production-grade implementation. The pattern is:\n\n``` python\ndef _render(self):\n    # Order phases by nesting depth so children draw under parents.\n    rows = sorted(\n        self._phases.items(),\n        key=lambda kv: (kv[1].parent or \"\", kv[0]),\n    )\n    # Move the cursor up by the number of lines the PREVIOUS render printed,\n    # not the current row count — phases are added on nesting and removed on\n    # phase_finish, so the two differ exactly when the tree changes shape.\n    if self._rendered_lines:\n        print(f\"\\x1b[{self._rendered_lines}A\", end=\"\")\n    for name, st in rows:\n        # step is a 0-based index in [0, num_steps); +1 turns it into a\n        # completed count so the bar can actually reach 100%.\n        done = min(st.current_step + 1, st.num_steps)\n        pct = done / max(st.num_steps, 1)\n        bar = \"█\" * int(40 * pct) + \"·\" * (40 - int(40 * pct))\n        indent = \"  \" if st.parent else \"\"\n        print(f\"\\x1b[2K{indent}{name:<28} [{bar}] {done}/{st.num_steps}\")\n    # Clear rows left behind when a phase finishes and the count shrinks.\n    for _ in range(self._rendered_lines - len(rows)):\n        print(\"\\x1b[2K\")\n    self._rendered_lines = len(rows)\n```\n\nThe upstream `simple_progress_monitor.py`\n\nrenders the same shape with improved color and width handling. The escape sequence `\\x1b[NA`\n\nmoves the cursor up *N* lines, and `\\x1b[2K`\n\nclears a line. The first render call writes blank rows; subsequent calls overwrite them in place.\n\nWhen this monitor is attached, do not redirect stdout to a file or pipe. The escape codes will be written verbatim into the log and make it unreadable. For non-terminal sinks, replace `_render()`\n\nwith a structured emitter.\n\n**3. Add a cancel path**\n\nCancellation is a three-line addition once the monitor exists. Install a SIGINT handler that flips the flag, then let `step_complete`\n\nhonor it.\n\n``` python\nimport signal\n\ndef install_cancel(monitor: RichProgressMonitor):\n    def handler(signum, frame):\n        monitor._cancelled = True\n        print(\"\\nCancelling TensorRT build at next step boundary...\")\n\n    signal.signal(signal.SIGINT, handler)\n```\n\nWire the monitor and run the builder:\n\n```\nbuilder = trt.Builder(TRT_LOGGER)\nnetwork = builder.create_network(\n    1 << int(trt.NetworkDefinitionCreationFlag.STRONGLY_TYPED)\n)\n\nparser = trt.OnnxParser(network, TRT_LOGGER)\n\nwith open(onnx_path, \"rb\") as f:\n    parser.parse(f.read())\n\nconfig = builder.create_builder_config()\n\nmonitor = RichProgressMonitor()\nconfig.progress_monitor = monitor\n\ninstall_cancel(monitor)\n\nserialized = builder.build_serialized_network(network, config)\n\nif serialized is None:\n    if monitor._cancelled:\n        print(\"Build cancelled cleanly.\")\n    else:\n        print(\"Build failed.\")\n```\n\n`build_serialized_network()`\n\nreturns `None`\n\non cancellation. The builder unwinds at the next step boundary, usually quickly, but not instantaneously, especially inside a long tactic-search step.\n\nApplications should surface cancellation latency to users. A simple *“Cancelling…”* message during the unwind window goes a long way.\n\nThe same flag can be set from any non-signal path, such as an IDE Stop button, an agent timeout, or a CI cancel webhook. Set `monitor._cancelled = True`\n\n, and the build aborts at the next step boundary.\n\n### 4**. The same pattern in C++**\n\n```\n#include <NvInfer.h>\n#include <atomic>\n#include <mutex>\n#include <unordered_map>\n\nclass RichProgressMonitor : public nvinfer1::IProgressMonitor {\npublic:\n    void phaseStart(char const* phaseName,\n                    char const* parentPhase,\n                    int32_t nbSteps) noexcept override {\n        std::lock_guard<std::mutex> g(mu_);\n        phases_[phaseName] = {nbSteps, 0, parentPhase ? parentPhase : \"\"};\n        render();\n    }\n\n    bool stepComplete(char const* phaseName,\n                      int32_t step) noexcept override {\n        std::lock_guard<std::mutex> g(mu_);\n        auto it = phases_.find(phaseName);\n        if (it != phases_.end())\n            it->second.current = step;\n        render();\n        return !cancelled_.load();\n    }\n\n    void phaseFinish(char const* phaseName) noexcept override {\n        std::lock_guard<std::mutex> g(mu_);\n        phases_.erase(phaseName);\n        render();\n    }\n\n    void requestCancel() noexcept {\n        cancelled_.store(true);\n    }\n\nprivate:\n    struct Phase {\n        int32_t nbSteps;\n        int32_t current;\n        std::string parent;\n    };\n\n    std::mutex mu_;\n    std::unordered_map<std::string, Phase> phases_;\n    std::atomic<bool> cancelled_{false};\n\n    void render() noexcept;\n};\n```\n\nAttach it the same way:\n\n``` php\nauto config =\n    std::unique_ptr<nvinfer1::IBuilderConfig>(\n        builder->createBuilderConfig());\n\nRichProgressMonitor monitor;\n\nconfig->setProgressMonitor(&monitor);\n```\n\n`std::atomic<bool>`\n\nfor the cancel flag matters because `requestCancel()`\n\nmay be called from another thread or a signal handler. Everything else mirrors the Python version.\n\n## Where to wire it in real systems\n\n*Figure 3. IProgressMonitor is the single integration point between the builder and an application’s surfaces*\n\nThe cancel arrow is drawn from the `agent runtime`\n\nfor concreteness, but the same mechanism applies to every sink. A Ctrl-C from the terminal, an IDE Stop button, an HTTP cancel webhook, or an agent timeout all flip the same `monitor._cancelled`\n\nflag, and the cancel takes effect at the next `step_complete`\n\nreturn.Where to wire it in real systems\n\nThe terminal is the easy case. The interesting integrations route progress somewhere else:\n\n**IDE extension:** Override`_render()`\n\nto emit`$/progress`\n\nnotifications in the Language Server Protocol, or equivalent`window/showProgress`\n\nin protocol. Each phase becomes one progress token;`step_complete()`\n\nbecomes a report message;`phase_finish()`\n\nbecomes end.**FastAPI / HTTP service:** Run the build on a background thread, and have`_render()`\n\npush entries into an`asyncio.Queue`\n\nthat the request handler drains via Server-Sent Events. The client gets a live stream; the cancel hook is just a`POST /builds/{id}/cancel`\n\nthat calls`monitor.requestCancel()`\n\n.**Agent tool call:** Emit one structured chunk per phase transition (`{\"phase\": ..., \"step\": ..., \"total\": ...}`\n\n) into the tool-call stream. The agent runtime renders it in the user-visible trace, and the same`requestCancel()`\n\nhook is what an agent timeout calls when the build exceeds the budget. This pattern also matters for agent runtimes. Long-running builds need to be observable and cancelable so agents can report progress, enforce time budgets, and stop cleanly.\n\nIn all three cases, `IProgressMonitor`\n\nis the right boundary. Anything above it (rendering, streaming, transport) is application-level; anything below it (tactic timing, kernel selection) is the builder’s business.\n\n## Edge cases to handle\n\nThese behaviors are common sources of integration bugs:\n\n- Do not redirect\n`stdout`\n\nwhile the terminal renderer is attached. The escape sequences will pollute the log. For non-interactive sinks, swap the renderer for a structured emitter. `phase_start()`\n\ncan’t cancel. It returns`None`\n\n. The earliest cancel point is the first`step_complete()`\n\nof that phase. If the user cancels during a long`phase_start()`\n\n, the build will continue until the first step boundary.`phase_finish()`\n\nmay fire before all`num_steps`\n\nare reported. This can happen during error recovery, builder-internal short-circuits, or when`step_complete()`\n\nreturns`False`\n\n. Treat it as the authoritative end-of-phase signal; do not assume`current_step == num_steps`\n\n.- Cancel latency is bounded but not zero. The builder finishes the current step before checking the return value. Long tactic-search steps can push this into the seconds-to-tens-of-seconds range.\n- Thread safety is required. The same monitor instance is called from multiple builder threads; uninstrumented\n`dict`\n\nor`unordered_map`\n\naccess from`_render()`\n\nwill eventually crash or tear.\n\n## Get started\n\nThe fastest way to run this end to end is:\n\n```\ngit clone --depth 1 https://github.com/NVIDIA/TensorRT.git\ncd TensorRT/samples/python/simple_progress_monitor\npython3 simple_progress_monitor.py\n```\n\nThis starts a live, animated build of a ResNet-50 engine. Replace `simple_progress_monitor.py`\n\n‘s monitor class with the version above or attach a cancel handler around the existing class. C++ equivalent is available in `samples/sampleProgressMonitor/`\n\n.\n\nFor larger systems, the right next step is replacing the terminal renderer with the transport the application already uses such as Language Server Protocol notifications, server-sent events, or structured tool-call chunks. `IProgressMonitor`\n\nbecomes the point where TensorRT build progress is translated into the application’s progress model.\n\n### Learn more\n\nRefer to the following resources for more information:", "url": "https://wpnews.pro/news/make-long-running-nvidia-tensorrt-engine-builds-observable-and-cancelable-in-or", "canonical_source": "https://developer.nvidia.com/blog/make-long-running-nvidia-tensorrt-engine-builds-observable-and-cancelable-in-python-or-c/", "published_at": "2026-07-22 16:35:04+00:00", "updated_at": "2026-07-22 16:56:53.067261+00:00", "lang": "en", "topics": ["developer-tools", "ai-infrastructure", "ai-tools"], "entities": ["NVIDIA TensorRT", "IProgressMonitor", "Python", "C++", "ResNet-50", "MNIST"], "alternates": {"html": "https://wpnews.pro/news/make-long-running-nvidia-tensorrt-engine-builds-observable-and-cancelable-in-or", "markdown": "https://wpnews.pro/news/make-long-running-nvidia-tensorrt-engine-builds-observable-and-cancelable-in-or.md", "text": "https://wpnews.pro/news/make-long-running-nvidia-tensorrt-engine-builds-observable-and-cancelable-in-or.txt", "jsonld": "https://wpnews.pro/news/make-long-running-nvidia-tensorrt-engine-builds-observable-and-cancelable-in-or.jsonld"}}