CUDA Device Max Connections

NVIDIA's CUDA_DEVICE_MAX_CONNECTIONS environment variable defaults to 8 concurrent hardware connections to the GPU, capping real concurrency regardless of how many CUDA streams a developer creates, according to a technical blog post by Lei Mao. The post benchmarks 32 CUDA streams with 32 threads and 200 queries per thread, showing that GPU utilization stays limited unless CUDA_DEVICE_MAX_CONNECTIONS is raised. The author published the example code in the leimao/CUDA-Device-Max-Connections GitHub repository with profiling traces.

CUDA Device Max Connections To maximize GPU utilization, it is common to have multiple workers processing tasks concurrently on GPU. However, by default, the GPU hardware concurrency is limited, no matter how much software concurrency is implemented. As a consequence, GPU might still be under utilized, even if at the software level the concurrency appears high in the implementation. CUDA DEVICE MAX CONNECTIONS is an environment variable that can be set to control the number of hardware concurrency on GPU. In this blog post, I would like to quickly discuss the importance of setting CUDA DEVICE MAX CONNECTIONS https://docs.nvidia.com/cuda/cuda-programming-guide/05-appendices/environment-variables.html cuda-device-max-connections for maximizing GPU concurrency and overall utilization. In CUDA programming, a CUDA stream is an abstraction which allows the programmer to express a sequence of operations. The developer could create multiple streams to enable concurrent execution of different tasks on the GPU, thereby improving overall utilization and performance. CUDA kernels launched in different streams can run concurrently, subject to hardware limitations and resource availability, such as the number of available Streaming Multiprocessors. There is one key factor that the developer might overlook, which is the CUDA DEVICE MAX CONNECTIONS environment variable that controls the maximum number of concurrent connections to the GPU. If this variable is not set appropriately, no matter how many CUDA streams are created, how lightweight the kernels are on each stream, the GPU concurrency will still be limited. In the following example https://github.com/leimao/CUDA-Device-Max-Connections , we created 32 CUDA streams to run concurrent tasks on GPU. The inference performances are benchmarked and profiling traces are collected. | 123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132 | include "multi stream.h" include "cuda utils.h" include "cuda worker.h" include <chrono include <cstdlib include <iostream include <set include <string include <thread include <torch/torch.h include <vector int run multi stream int argc, char argv { char const max conn env = std::getenv "CUDA DEVICE MAX CONNECTIONS" ; std::string max conn str = max conn env ? max conn env : "default"; std::string trace filename = "multi stream trace max conn " + max conn str + ".json"; if argc == 3 && std::string argv 1 == "--trace-file" { trace filename = argv 2 ; } else if argc = 1 { std::cerr << "Usage: " << argv 0 << " --trace-file <path-to-trace.json " << std::endl; return EXIT FAILURE; } std::cout << "CUDA DEVICE MAX CONNECTIONS = " << max conn env ? max conn env : "Not Set Defaults to 8 " << std::endl; int const num threads = 32; int const queries per thread = 200; unsigned long long const cycles = 1000000ULL; int const total queries = num threads queries per thread; std::vector<cudaStream t streams num threads ; for int thread index = 0; thread index < num threads; ++thread index { CHECK CUDA ERROR cudaStreamCreateWithFlags &streams thread index , cudaStreamNonBlocking ; } at::ThreadLocalState tls state; std::cout << "\n--- Phase 1: Measuring Pure System Throughput QPS ---" << std::endl; auto start time = std::chrono::high resolution clock::now ; std::vector<std::thread benchmark workers; for int thread index = 0; thread index < num threads; ++thread index { benchmark workers.emplace back enqueue delay kernels, streams thread index , queries per thread, cycles, tls state ; } for auto& worker : benchmark workers { worker.join ; } for cudaStream t stream : streams { CHECK CUDA ERROR cudaStreamSynchronize stream ; } auto end time = std::chrono::high resolution clock::now ; std::chrono::duration<double elapsed = end time - start time; double elapsed seconds = elapsed.count ; double throughput = total queries / elapsed seconds; std::cout << "Total Queries Processed : " << total queries << std::endl; std::cout << "Elapsed Time : " << elapsed seconds << " seconds" << std::endl; std::cout << "Pure System Throughput : " << throughput << " queries/second" << std::endl; std::cout << "\n--- Phase 2: Collecting Profiling Trace ---" << std::endl; torch::autograd::profiler::ProfilerConfig profiler config torch::autograd::profiler::ProfilerState::KINETO, / report input shapes= /false, / profile memory= /false, / with stack= /false, / with flops= /false, / with modules= /false ; std::set<torch::autograd::profiler::ActivityType activities = { torch::autograd::profiler::ActivityType::CUDA}; torch::autograd::profiler::prepareProfiler profiler config, activities ; torch::autograd::profiler::enableProfiler profiler config, activities ; std::vector<std::thread profile workers; for int thread index = 0; thread index < num threads; ++thread index { profile workers.emplace back enqueue delay kernels, streams thread index , queries per thread, cycles, tls state ; } for auto& worker : profile workers { worker.join ; } std::this thread::sleep for std::chrono::milliseconds 500 ; auto profiler result = torch::autograd::profiler::disableProfiler ; for cudaStream t stream : streams { CHECK CUDA ERROR cudaStreamSynchronize stream ; } if profiler result { profiler result- save trace filename ; std::cout << "Saved PyTorch Profiler trace to: " << trace filename << std::endl; } for cudaStream t stream : streams { CHECK CUDA ERROR cudaStreamDestroy stream ; } return 0;} | By varying CUDA DEVICE MAX CONNECTIONS , we can control the maximum number of concurrent connections to the GPU device, which affects the performance of multi-stream workloads. | 12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061626364656667686970 | bash $ for connections in 1 2 4 8 16 32; do CUDA DEVICE MAX CONNECTIONS="$connections" ./build/multi stream \ --trace-file "build/max conn ${connections}.json"doneCUDA DEVICE MAX CONNECTIONS = 1--- Phase 1: Measuring Pure System Throughput QPS ---Total Queries Processed : 6400Elapsed Time : 1.66418 secondsPure System Throughput : 3845.73 queries/second--- Phase 2: Collecting Profiling Trace ---USDT:2026-09-24 03:17:03 553:553 SyncActivityProfilerHandler.cpp:39 profiler startUSDT:2026-09-24 03:17:04 553:553 SyncActivityProfilerHandler.cpp:46 profiler stopSaved PyTorch Profiler trace to: build/max conn 1.jsonCUDA DEVICE MAX CONNECTIONS = 2--- Phase 1: Measuring Pure System Throughput QPS ---Total Queries Processed : 6400Elapsed Time : 1.13147 secondsPure System Throughput : 5656.38 queries/second--- Phase 2: Collecting Profiling Trace ---USDT:2026-09-24 03:17:06 626:626 SyncActivityProfilerHandler.cpp:39 profiler startUSDT:2026-09-24 03:17:07 626:626 SyncActivityProfilerHandler.cpp:46 profiler stopSaved PyTorch Profiler trace to: build/max conn 2.jsonCUDA DEVICE MAX CONNECTIONS = 4--- Phase 1: Measuring Pure System Throughput QPS ---Total Queries Processed : 6400Elapsed Time : 0.438797 secondsPure System Throughput : 14585.3 queries/second--- Phase 2: Collecting Profiling Trace ---USDT:2026-09-24 03:17:08 699:699 SyncActivityProfilerHandler.cpp:39 profiler startUSDT:2026-09-24 03:17:09 699:699 SyncActivityProfilerHandler.cpp:46 profiler stopSaved PyTorch Profiler trace to: build/max conn 4.jsonCUDA DEVICE MAX CONNECTIONS = 8--- Phase 1: Measuring Pure System Throughput QPS ---Total Queries Processed : 6400Elapsed Time : 0.35048 secondsPure System Throughput : 18260.7 queries/second--- Phase 2: Collecting Profiling Trace ---USDT:2026-09-24 03:17:11 772:772 SyncActivityProfilerHandler.cpp:39 profiler startUSDT:2026-09-24 03:17:11 772:772 SyncActivityProfilerHandler.cpp:46 profiler stopSaved PyTorch Profiler trace to: build/max conn 8.jsonCUDA DEVICE MAX CONNECTIONS = 16--- Phase 1: Measuring Pure System Throughput QPS ---Total Queries Processed : 6400Elapsed Time : 0.142692 secondsPure System Throughput : 44851.8 queries/second--- Phase 2: Collecting Profiling Trace ---USDT:2026-09-24 03:17:12 845:845 SyncActivityProfilerHandler.cpp:39 profiler startUSDT:2026-09-24 03:17:13 845:845 SyncActivityProfilerHandler.cpp:46 profiler stopSaved PyTorch Profiler trace to: build/max conn 16.jsonCUDA DEVICE MAX CONNECTIONS = 32--- Phase 1: Measuring Pure System Throughput QPS ---Total Queries Processed : 6400Elapsed Time : 0.0756491 secondsPure System Throughput : 84601.2 queries/second--- Phase 2: Collecting Profiling Trace ---USDT:2026-09-24 03:17:14 918:918 SyncActivityProfilerHandler.cpp:39 profiler startUSDT:2026-09-24 03:17:15 918:918 SyncActivityProfilerHandler.cpp:46 profiler stopSaved PyTorch Profiler trace to: build/max conn 32.json | The system throughputs benchmarked and the profiling traces collected for different values of CUDA DEVICE MAX CONNECTIONS are summarized in the table below. | CUDA DEVICE MAX CONNECTIONS | Number of CUDA Streams | System Throughput QPS | Perfetto Trace | |---|---|---|---| | 1 | 32 | 3,845.73 | Trace https://ui.perfetto.dev/ /?url=https://raw.githubusercontent.com/leimao/CUDA-Device-Max-Connections/refs/heads/main/traces/max conn 1.json | | 2 | 32 | 5,656.38 | Trace https://ui.perfetto.dev/ /?url=https://raw.githubusercontent.com/leimao/CUDA-Device-Max-Connections/refs/heads/main/traces/max conn 2.json | | 4 | 32 | 14,585.30 | Trace https://ui.perfetto.dev/ /?url=https://raw.githubusercontent.com/leimao/CUDA-Device-Max-Connections/refs/heads/main/traces/max conn 4.json | | 8 | 32 | 18,260.70 | Trace https://ui.perfetto.dev/ /?url=https://raw.githubusercontent.com/leimao/CUDA-Device-Max-Connections/refs/heads/main/traces/max conn 8.json | | 16 | 32 | 44,851.80 | Trace https://ui.perfetto.dev/ /?url=https://raw.githubusercontent.com/leimao/CUDA-Device-Max-Connections/refs/heads/main/traces/max conn 16.json | | 32 | 32 | 84,601.20 | Trace https://ui.perfetto.dev/ /?url=https://raw.githubusercontent.com/leimao/CUDA-Device-Max-Connections/refs/heads/main/traces/max conn 32.json | We could see that the system throughput nearly doubles as CUDA DEVICE MAX CONNECTIONS is doubled, indicating a strong correlation between the number of allowed CUDA connections and the overall system performance. By examining the Perfetto traces, we could see that despite the very lightweight kernel, there are lots of bubbles in CUDA stream which are not caused by CPU launch overhead, if CUDA DEVICE MAX CONNECTIONS is not the same as the number of CUDA streams. Technically, each CUDA stream is associated with a hardware queue on GPU, and the number of hardware queues is configured by the CUDA DEVICE MAX CONNECTIONS environment variable. By default, CUDA DEVICE MAX CONNECTIONS is set to 8 . Therefore, in our application, if CUDA DEVICE MAX CONNECTIONS is not set, the system will be significantly underutilized. We could check what hardware queue each CUDA stream is mapped to by examining the stream and the channel attributes of CUDA kernels. For example, in the Perfetto trace of CUDA DEVICE MAX CONNECTIONS=1 , all CUDA streams are mapped to the same hardware queue 0 . AMD GPUs have similar concepts of hardware queues and stream-to-queue mapping, which can be controlled through environment variables specific to the ROCm platform. In the case of AMD GPUs, GPU MAX HW QUEUES https://rocm.docs.amd.com/projects/HIP/en/latest/reference/env variables.html specifies the maximum number of hardware queues available for mapping streams and hsa queue is the hardware queue associated with a particular stream that can be checked from the Perfetto trace attributes. Note that the default value of GPU MAX HW QUEUES is 4 , which means the maximum GPU concurrency is very limited unless this environment variable is increased. CUDA Device Max Connections