Gpu_ai_runtime_memory_proposal

A new proposal calls for a common GPU execution and memory-management layer above CUDA, ROCm, and Intel XPU/oneAPI so that AI workloads are not tightly coupled to a single vendor, targeting 13B-class INT4 inference on 12 GB consumer GPUs. The proposal states its goal is not to replace CUDA, which would remain the NVIDIA backend, but to move model, memory, quantization, and kernel logic above the vendor-specific layers. It argues the main challenge is controlling the entire runtime memory footprint, including KV cache, rather than merely quantizing weights.

AI hardware is improving faster than the software stack that is supposed to use it. NVIDIA, AMD, and Intel all produce increasingly capable GPUs with better matrix acceleration, lower-precision compute, higher memory bandwidth, and stronger AI-oriented hardware. However, users still often have to choose a GPU based on software compatibility first and hardware capability second. In practice, the first question is often not: Which GPU has the best VRAM capacity, bandwidth, compute performance, power efficiency, and price? Instead, it is: Does this model, extension, quantization library, or attention kernel work with CUDA? That is not a healthy long-term situation for a multi-vendor GPU market. The goal of this proposal is not to replace CUDA . CUDA, ROCm, and Intel XPU / oneAPI should continue to exist as optimized vendor backends. The goal is to build a stronger common execution and memory-management layer above them so that application code, model code, and common AI workloads do not have to be tightly coupled to one GPU vendor. Local AI users repeatedly encounter the same classes of problems: The problem is therefore not just raw GPU compute. The software stack needs better control over memory placement, low-bit execution, backend abstraction, and runtime planning . A useful practical target would be: Make 13B-class INT4 inference practical on 12 GB consumer GPUs without requiring users to manually combine many separate libraries and backend-specific workarounds. Approximate model weight sizes for a 13B model are: This means that INT4 weights can theoretically fit comfortably within 12 GB. The real problem is everything else: So the main challenge is not merely quantizing the weights. It is controlling the entire runtime memory footprint . Instead of forcing users to manually tune device map , max memory , offload folders, quantization settings, and backend-specific flags, the runtime should accept a simple memory budget. For example: python model = load model path=“model”, device=“gpu”, memory budget=“11GB”, weight dtype=“int4”, kv cache dtype=“int8”, offload=“auto”, The runtime should automatically decide: The user should be able to say: My GPU has 12 GB. Make the best execution plan that fits. The framework should then perform the planning. An ideal architecture could look like this: text Application / Transformers / Local LLM Runtime | v PyTorch Graph | v Common GPU Runtime / IR | ±----------±----------+ | | | v v v CUDA ROCm XPU NVIDIA AMD Intel Again, this does not require removing CUDA. CUDA would remain the NVIDIA backend. ROCm would remain the AMD backend. XPU / oneAPI would remain the Intel backend. The difference is that more model logic, memory logic, quantization logic, and kernel definitions would exist above those vendor-specific layers. Ideally, users could write: python model.to “gpu” and the runtime would internally choose: text NVIDIA → CUDA AMD → ROCm Intel → XPU / oneAPI without requiring major changes in application code. INT4, INT8, FP8, and mixed precision should not behave like optional tricks provided by unrelated third-party libraries. They should be first-class execution and memory-management concepts. Examples include: A critical point is that an INT4 model should not repeatedly allocate large BF16 copies of its weights during execution. The preferred path should look more like: text INT4 Weight | v Fused Dequant + MatMul | v BF16 / FP16 Output rather than: text INT4 Weight | v Temporary BF16 Weight Copy | v MatMul The latter can destroy much of the memory advantage of quantization. Reducing weight size is not enough. As context length grows, KV cache can become one of the largest consumers of VRAM. A modern runtime should support: For example, when given an 11 GB memory budget, the runtime might automatically decide: text Weights: INT4 KV Cache: INT8 Attention: memory-efficient implementation Context Length: automatically limited to available memory This should be planned before execution, not after an OOM crash. Current consumer systems effectively look like this: text GPU VRAM | PCIe | System RAM | Storage For AI workloads, a more explicit hierarchical memory model would be useful: text Tier 0: GPU Local VRAM Tier 1: GPU-Side Expansion Memory Tier 2: System RAM Tier 3: NVMe / mmap Storage The fastest memory. Best suited for: A future large-capacity memory tier physically close to the GPU. Possible capacities: It may be slower than local VRAM but much faster and lower-latency than system RAM over PCIe. Possible uses: Large and relatively inexpensive, but slower and farther from the GPU. Useful for less frequently accessed tensors and overflow capacity. A last-resort storage tier. Useful for: It should not be treated as a substitute for fast working memory. Consumer GPUs continue to gain compute performance faster than they gain VRAM capacity. This creates situations where a GPU has enough compute throughput for a model but cannot hold the model and its runtime state. One possible hardware direction would be: text GPU | ±- 12 GB High-Speed VRAM | ±- 32-64 GB Expansion Memory | ±- Dedicated High-Speed Link The expansion memory does not need to be as fast as local GDDR or HBM. However, it should be significantly better than ordinary system RAM or NVMe for GPU access. Useful properties would include: In such a design, 12 GB of VRAM could act as the high-speed working set while a much larger pool stores the rest of the model. This could be especially valuable for local AI workloads. NVMe offloading can prevent complete failure when a model is too large. It does not solve the performance problem. If every layer requires data to be read from storage and transferred to the GPU, then execution becomes transfer-bound: text Compute Time << Data Transfer Time At that point, additional GPU compute performance is mostly wasted. NVMe is useful for: It is not an ideal primary memory tier for interactive inference. The real solution is to reduce how much data must cross slow links during execution. If PyTorch continues evolving toward a broader AI execution runtime, the following features could provide significant value. Example: python torch.set memory budget “11GB” or a model-specific equivalent. The runtime could understand placements such as: text gpu local gpu extended system ram mmap and move tensors automatically. Low-bit formats should be directly represented and optimized: An attention, quantization, or matrix-multiplication kernel could be defined once and compiled to: backend-specific implementations. The runtime should automatically select between: depending on available hardware and memory. Instead of crashing first and forcing the user to change settings, the runtime could estimate memory requirements before execution. text Expected Usage: 13.8 GB Available Budget: 11.0 GB Replanned Execution: This is a better user experience than trial-and-error OOM debugging. From a Transformers user perspective, the ideal interface could be extremely simple: python model = AutoModel.from pretrained model id, device=“auto”, memory budget=“11GB”, precision=“auto”, Internally, the framework could coordinate: Today, users often have to read documentation for several separate libraries and manually combine them. That complexity should gradually move into the runtime itself. This point is important. CUDA is a mature and highly optimized NVIDIA backend. There is no need to remove it. The problem is that CUDA compatibility is often treated as a requirement for using a model, extension, or kernel at all. The ideal market would allow users to choose GPUs based mainly on: rather than mainly asking: Does this require CUDA? If NVIDIA provides the best hardware, users can choose NVIDIA. If AMD provides more VRAM at a better price, users should be able to choose AMD without losing major software capabilities. If Intel offers competitive hardware, it should also be a viable option. That is healthier competition. A stronger common runtime could change how consumer GPUs compete. Current situation: text Software compatibility → GPU vendor decision Preferred situation: text Price / VRAM / Bandwidth / Efficiency / Performance → GPU vendor decision This would encourage all three major GPU vendors to compete on actual hardware value while still maintaining their own optimized backend stacks. The result should be better for users, developers, and the broader AI ecosystem. Run a 13B-class INT4 model on a 12 GB NVIDIA GPU. Requirements: Run the same model code on a 16 GB AMD GPU. The application code should not require major modification. Run the same model code on an Intel GPU. Backend differences should remain mostly inside the framework. When local VRAM is insufficient, automatically move selected tensors to another memory tier. The user should not have to manually assign every layer. Proposal: Vendor-neutral, memory-budget-aware execution for consumer GPUs I would like to suggest a direction for PyTorch, Transformers, and local LLM runtimes. Today, consumer GPU users face two major problems: A 12 GB GPU may have enough compute power for a 13B-class model, especially with INT4 weights, but KV cache, temporary buffers, backend-specific kernels, and fragmented memory allocation can still cause OOM. I think frameworks should move toward a memory-budget-aware execution model . python model = load model path, device=“gpu”, memory budget=“11GB”, weight dtype=“int4”, kv cache dtype=“int8”, offload=“auto”, The goal is not to replace CUDA . CUDA, ROCm, and XPU can remain optimized vendor backends. The goal is to build a stronger common execution layer above them so that model code does not need to be heavily rewritten or restricted to one GPU vendor. I also think frameworks should prepare for hierarchical GPU memory: text Tier 0: GPU local VRAM Tier 1: future GPU-side expansion memory Tier 2: system RAM Tier 3: NVMe / mmap NVMe offloading alone is not a real performance solution because bandwidth and latency are much worse than local GPU memory. A future GPU-side expansion-memory tier could allow a fast 12 GB consumer GPU to use a much larger model without relying on system RAM or SSD as the primary working memory. A concrete target could be: There are already many relevant technologies: However, these capabilities are still fragmented. I think a future major version of the stack would be more valuable if it focused on unified memory management, low-bit inference, and hardware-neutral execution. The end goal should be simple: Users should choose GPUs based on VRAM, bandwidth, compute performance, power efficiency, and price — not primarily because a particular model or kernel only works on one vendor. I would be interested to hear whether there are existing projects, RFCs, or research efforts moving in this direction. AI frameworks should hide GPU-vendor differences, automatically optimize models for a fixed VRAM budget, and prepare for future GPU-side expansion memory so that 12 GB consumer GPUs can run larger models efficiently.