# CUDA Graph In The Context of Multi-Stream Execution

> Source: <https://leimao.github.io/blog/CUDA-Graph-Multi-Stream/>
> Published: 2026-08-19 15:12:05.196550+00:00

# CUDA Graph In The Context of Multi-Stream Execution

Introduction

CUDA Graph is typically known to reduce CPU overhead, remove GPU bubbles, and improve GPU utilizations when launching many kernels. In the context of single stream execution, the direct consequence of enabling CUDA Graph is reducing the processing latency. In the context of multi-stream execution, however, this might not be the case.

In this blog post, I would like to quickly discuss the caveats of using CUDA Graph in the context of multi-stream execution.

CUDA Graph In The Context of Multi-Stream Execution

To maximize GPU utilization and reduce latency service-level agreement (SLA), especially during high-throughput model serving, it is common to have multiple workers on multiple threads launching kernel sequences in multiple CUDA streams in production.

Because different kernels consume different amount of SM resources, some kernels might execute preemptively and block the execution of other kernels in other streams, some kernels might execute concurrently with other kernels in other streams. As a consequence, for the CUDA kernel sequence that is launched in a single stream, it is common to have bubbles.

Given this situation, sometimes it is tempting to use CUDA Graph because CUDA Graph is known to remove GPU bubbles, at least in the context of single stream execution. However, in the context of multi-stream execution, CPU overhead is not the only reason that causes GPU bubbles, as mentioned above, other blocking CUDA kernels in other streams can also cause GPU bubbles. **CUDA Graph can only remove GPU bubbles that are caused by CPU overhead, but cannot remove GPU bubbles that are caused by blocking CUDA kernels in other streams.** As a consequence, if most of the GPU bubbles are caused by blocking CUDA kernels in other streams, enabling CUDA Graph cannot reduce the processing latency and improve GPU utilization. In such situations, usually the GPU utilization is already very high and we should not spend engineering effort to test the effect of enabling CUDA Graph. Enabling CUDA Graph will not reduce the processing latency either. The GPU utilization, SM utilization in particular, can be monitored using [NVIDIA NVML APIs](/blog/NVIDIA-NVML-GPU-Statistics/).

CUDA Graph has lots of restrictions on models and we should not spend excessive engineering effort to enable it in a system whose GPU is already highly utilized, even if it appears that there are a lot of GPU bubbles from the profiling trace.

Of course, it is also completely possible that even with multi-stream execution, most of the GPU bubbles are caused by CPU overhead. This can happen when the system does not have too many threads, i.e. GPU is much more “powerful” than CPU, and the batch size for each worker to process is small in order to satisfy the SLA. In this situation, then enabling CUDA Graph might still be worthwhile. But one also has to be very careful because such GPU bubbles caused by CPU overhead observed in the profiling trace can be an artifact of the profiling process. One should always check if the profiling trace is consistent with the system utilizations when profiling is not enabled.

Conclusions

Foundamentally, the purpose of CUDA Graph is to improve GPU utilization. If the GPU utilization is already very high, the benefit of enabling CUDA Graph is likely to be minimal.

CUDA Graph In The Context of Multi-Stream Execution