Scaling NVIDIA TensorRT Performance Optimization Beyond a Single Build Route with the Global Performance Tuner, Part 2

NVIDIA's Global Performance Tuner, released in TensorRT 11.1, automatically tunes performance by treating alternative scheduling strategies as explicit, searchable build routes, each a concrete combination of compiler knob values. Part 2 of NVIDIA's series describes the feature's algorithm and architecture and offers best practices, noting that TensorRT still performs remaining engine construction stages such as tactic selection after a route is applied. Because no single fusion strategy is universally better across models, tensor shapes and target GPUs, NVIDIA says selecting the better route requires building candidate engines and comparing their end-to-end performance.

The Global Performance Tuner https://docs.nvidia.com/deeplearning/tensorrt/latest/performance/tuning.html is a feature for automatic performance tuning released in NVIDIA TensorRT 11.1 https://docs.nvidia.com/deeplearning/tensorrt/latest/getting-started/release-notes-11/11.1.0.html . It provides automatic, accuracy-aware and end-to-end performance tuning within TensorRT by customizing different optimizations for different models. For more details, see Part 1 https://developer.nvidia.com/blog/scaling-nvidia-tensorrt-performance-optimization-beyond-a-single-build-route-with-the-global-performance-tuner-part-1/ of this series, which introduces the challenges associated with scheduling a critical stage in deep learning compiler optimization , the motivation for developing the Global Performance Tuner feature in TensorRT, and the mathematical theories established to formalize the problem. This post, Part 2, uses the notation from Part 1 to describe the algorithm and architectural design of the Global Performance Tuner feature. It also provides best practices for using it. Designing the Global Performance Tuner feature in TensorRT The central design idea of the Global Performance Tuner is to represent alternative scheduling strategies as explicit and searchable build routes. A compiler knob also called a compiler flag or compiler option is a configuration setting that changes some aspect of the compilation pipeline, such as a heuristic, fusion decision, or code generation path. Each tunable knob exposes a finite set of allowed values and has a default value. A build route is a concrete combination of knob values. When no build route is specified, TensorRT uses the default build route, in which every knob retains its default value. Consider a hypothetical compiler consisting of three passes, $f 1$, $f 2$, and $f 3$. Suppose a binary knob $p$ controls the fusion strategy used by $f 2$: when $p=1$, the pass applies horizontal fusion; when $p=0$, it applies vertical fusion: $f 2 s = \begin{cases} f’ 2 s , & \text{if } p = 1 \\ f” 2 s , & \text{if } p = 0 \end{cases}$ The two possible values of $p$ define two build routes and therefore produce a candidate solution set of size two for the input model $m$: $B m = \{f 3 f’ 2 f 1 m ,\, f 3 f” 2 f 1 m \}$ Neither fusion strategy is universally better. Horizontal fusion may produce the faster engine for one model, while vertical fusion may be more effective for another. The result may also change across tensor shapes and target GPUs, even when the model itself remains unchanged. Selecting the better route therefore requires building both candidate engines and comparing their end-to-end performance. Even this two-route example captures the essential idea of tuning: the compiler exposes meaningful scheduling alternatives, and measured behavior determines which complete solution performs best for the deployment case at hand. In the mathematical framework introduced earlier, a build route is a composition of compiler passes. In the implementation, a combination of knob assignments provides a compact specification of that route. Changing one or more knob values changes how the model travels through the tunable parts of the compilation pipeline. Within this abstraction, a knob combination can be treated as identifying a build route. More precisely, the knob combination is a build-route specification, not a complete description of the resulting engine. TensorRT still performs the remaining stages of engine construction, including tactic selection, and may reconcile individual knob values with constraints imposed by the network. The route specifies the requested choices within the tunable configuration space. It does not replace the rest of the TensorRT builder. TensorRT exposes a compact API surface around two fundamental capabilities: - Apply and inspect one build route : A setter applies a fixed route to the builder configuration, while the corresponding getter returns the currently specified route. - Discover the tunable configuration space : A separate getter describes the available knobs, their allowed values, their defaults, and their purposes. The corresponding C++ APIs are: bool IBuilderConfig::setBuildRoute char const buildRoute ; char const IBuilderConfig::getBuildRoute const; char const IBuilderConfig::getAllBuildRoutes const; You can use these APIs to implement custom tuners. A sample C++ implementation is also provided in trtexec . At a high level, trtexec organizes tuning as follows: - When a user specifies an input model and a regex-like tunable build route expression, trtexec will expand the expression into a list of concrete build routes according to the searching algorithm. - There will be a loop to traverse all fixed build routes. In each iteration, the model will be built with that single build route, then accuracy and performance will be measured during each iteration. - All build routes that crash during building or fail accuracy during inference are excluded from best-engine selection. The engine with the best measured end-to-end performance is selected. TensorRT 11.1 provides more than 200 knobs for tuning. For more details, refer to the Global Performance Tuner documentation https://docs.nvidia.com/deeplearning/tensorrt/latest/performance/tuning.html . The same implementation in Python will be released in Torch-TensorRT 2.15. The practical value of exploring multiple build routes becomes visible in end-to-end measurements. Figure 1 compares the default TensorRT build route, normalized to 1.0x for each workload, with the best eligible route found during tuning. Across these ten evaluated workloads, tuning delivered performance improvements ranging from 1.14x to 1.63x. The variation across workloads is central to the result. Different models expose different optimization opportunities, so a single default scheduling strategy cannot be expected to realize every model’s best performance. The Global Performance Tuner explores alternative build routes, rejects candidates that do not satisfy the accuracy requirement, and selects the fastest eligible engine measured in the target environment. Case study: Performance-accuracy entanglement The Global Performance Tuner feature applies accuracy-aware tuning, so the performance-accuracy entanglement problem can theoretically be solved perfectly. Consider the following thought experiment. For a simple neural network model, as shown in Figure 2, imagine there are knobs controlling whether to force some IRs to use a low-precision kernel, which is equivalent to converting the graph into quantized graph by inserting Q/DQ nodes between ops. This is not supported by TensorRT 11.0 and later because weakly typed APIs are deprecated. This model has four places to insert Q/DQ. Suppose these are controlled by four different binary knobs. The deep learning compiler could then provide 2<sup 4</sup = 16 solutions in the solution space. It is readily apparent that, among the 16 candidates, the solution with the best performance is the one with all quantizations inserted. However, this solution may not meet the accuracy requirement. The solution with no Q/DQ will have the best accuracy, but with the worst performance. The performance and accuracy are entangled such that it is difficult to know which solution in the “middle” satisfies both requirements. Recall that the concept of the builder algorithm contains an accuracy tolerance. The key to disentangle accuracy and performance is to define a clear accuracy acceptance boundary, and the boundary needs to be backed by user supplied accuracy metric and datasets as golden values. The Global Performance Tuner feature in TensorRT also has an accuracy tolerance threshold of $ε$. Consider the following three cases: - $ε = 0$: Under the exact-match metric assumed by this thought experiment, the candidate outputs must be bit-wise identical to the reference outputs. If every lower-precision substitution changes at least one evaluated output, only the one solution with no lower-precision substitutions remains eligible. - $ε = +\infty$: The accuracy requirement imposes no effective restriction, so all 16 candidate solutions remain eligible. More generally, imagine that the theoretical solution space admitted an “empty” solution that performed no useful computation and therefore had a runtime of zero. An optimizer guided only by performance could select this meaningless solution. This limiting case demonstrates why performance alone is not a sufficient optimization objective. - $ε$ changes gradually from $+\infty$ to 0: As the accuracy requirement becomes stricter, the eligible candidate set contracts—for example, from 16 solutions to 14, then 13, 8, 7, and eventually 1. These counts are illustrative; the actual transition points depend on the model, reference data, and selected accuracy metric. At each value of $ε$, the selection algorithm chooses the fastest measured solution among the evaluated candidates that remain eligible. The Global Performance Tuner does not remove the underlying trade-off. Instead, it disentangles the roles of performance and accuracy in the optimization process: - Accuracy determines which candidate solutions are acceptable - Performance ranks the acceptable candidate solutions As a developer, you define an accuracy boundary appropriate for your application, while tuning identifies the best-performing evaluated build route within that boundary. The result is a systematic constrained-optimization process rather than an ad hoc choice between speed and accuracy. This thought experiment is independent of the particular compiler knobs exposed by any TensorRT release. Its purpose is to demonstrate the broader design principle; that is, when scheduling choices are represented as tunable build routes and accuracy is treated as an explicit constraint, performance–accuracy optimization becomes measurable, reproducible, and application-defined. Best practices for using the Global Performance Tuner The Global Performance Tuner makes build-route exploration systematic, but effective tuning still benefits from a deliberate search strategy. When an expression contains many variable knobs, the number of possible combinations can grow rapidly. The following practices help keep tuning controlled, reproducible, and aligned with the target application. Choose the search strategy according to the search space Treat the default build route as the baseline against which alternative routes are compared. For a broad initial scan, the fast search evaluates the default route and varies one knob at a time. The mixed search is often a practical starting point for a larger expression: it first performs the fast scan and then explores combinations of the knob values that produced improvements. Reserve full search for a small, carefully selected group of knobs because it evaluates their complete Cartesian product. Before beginning a fast or full sweep, --dryRun can display the concrete route list without building engines. This provides a quick way to inspect the search space before committing to the full tuning run. --dryRun is not supported with mixed search because the second phase depends on measurements collected during the first phase. Treat tuning as an incremental and resumable process When --tuningCacheFile is specified, each completed iteration is recorded together with its build route, measured GPU time, and accuracy result. If the sweep is interrupted or reaches a time limit, it can later resume from the recorded state using --continue . With this approach, it’s possible to conduct tuning as a progressive experiment rather than requiring every search to finish in one uninterrupted run. When route-specific failures or accuracy regressions require deeper investigation, --saveAllEngines can preserve the engine produced by every iteration. Because this option can consume substantial storage, it is most useful for debugging rather than routine tuning. Adapt the tuning workflow to the application The supplied inputs and reference outputs should represent the deployment conditions that matter to the application. An engine satisfying the accuracy requirement on one narrow dataset has only been validated against that dataset, metric, and threshold. Include representative inputs, important boundary cases, and an accuracy metric that reflects the application’s actual quality requirements. The trtexec implementation provides a ready-to-use tuner, but it is not the only possible tuning policy. The builder-configuration APIs expose the available knobs and enable individual build routes to be applied, while the open-source trtexec implementation demonstrates how to organize the surrounding tuning loop. Developers can build application-specific search strategies, introduce heuristic filters, or apply application-level accuracy validation before comparing candidate performance. Treat previous results as guidance, not as universally transferable solutions The best build route is specific to the model, input shapes, target GPU, software stack, and TensorRT and tuner versions used during the search. A route that improves one deployment may have no effect or cause a regression in another. Re-tune after changing the model, shapes, target hardware, CUDA or driver environment, or relevant TensorRT version. Knowledge from previous tuning experiments can still be valuable. If certain knobs repeatedly improve a related family of workloads, that experience can guide future search-space design and help prioritize promising candidates. Such heuristics should act as priors that focus exploration, while every final route is validated and measured in its actual deployment environment. Get started with the Global Performance Tuner feature in TensorRT The Global Performance Tuner feature changes TensorRT performance optimization from relying on one universal scheduling strategy to evaluating model-specific alternatives. Customizable build routes address performance customization by allowing different models and deployment environments to explore different scheduling decisions. Separating candidate generation from route search provides a scalable path out of heuristic sprawl, reducing the need to encode every model-specific discovery as another hard-coded exception. Accuracy-aware tuning also addresses performance–accuracy entanglement. Accuracy determines which candidate engines are acceptable, and end-to-end performance determines which acceptable engine wins. The selected result is therefore the fastest measured eligible engine among the build routes evaluated for the target model and environment. TensorRT provides an excellent base recipe, compiler knobs provide controlled variations, and tuning identifies the best acceptable variation for each customer. This shift from universal prediction to model-specific evidence is how the Global Performance Tuner feature scales TensorRT optimization beyond a single build route. To get started, explore the Global Performance Tuner documentation https://docs.nvidia.com/deeplearning/tensorrt/latest/performance/tuning.html and the open source https://github.com/NVIDIA/TensorRT/tree/main/samples/trtexec trtexec implementation https://github.com/NVIDIA/TensorRT/tree/main/samples/trtexec . Extend the sample with your own accuracy validation algorithms, search strategies, or heuristics that prioritize promising build routes. For greater control, use TensorRT build-route APIs to build a tuner from scratch, tailoring candidate exploration and validation to your application’s accuracy requirements and performance goals.