AI hardware is improving faster than the software stack that is supposed to use it.
NVIDIA, AMD, and Intel all produce increasingly capable GPUs with better matrix acceleration, lower-precision compute, higher memory bandwidth, and stronger AI-oriented hardware. However, users still often have to choose a GPU based on software compatibility first and hardware capability second.
In practice, the first question is often not:
Which GPU has the best VRAM capacity, bandwidth, compute performance, power efficiency, and price?
Instead, it is:
Does this model, extension, quantization library, or attention kernel work with CUDA?
That is not a healthy long-term situation for a multi-vendor GPU market.
The goal of this proposal is not to replace CUDA.
CUDA, ROCm, and Intel XPU / oneAPI should continue to exist as optimized vendor backends.
The goal is to build a stronger common execution and memory-management layer above them so that application code, model code, and common AI workloads do not have to be tightly coupled to one GPU vendor.
Local AI users repeatedly encounter the same classes of problems:
The problem is therefore not just raw GPU compute.
The software stack needs better control over memory placement, low-bit execution, backend abstraction, and runtime planning.
A useful practical target would be:
Make 13B-class INT4 inference practical on 12 GB consumer GPUs without requiring users to manually combine many separate libraries and backend-specific workarounds.
Approximate model weight sizes for a 13B model are:
This means that INT4 weights can theoretically fit comfortably within 12 GB.
The real problem is everything else:
So the main challenge is not merely quantizing the weights. It is controlling the entire runtime memory footprint.
Instead of forcing users to manually tune device_map, max_memory, offload folders, quantization settings, and backend-specific flags, the runtime should accept a simple memory budget.
For example:
model = load_model(
path=“model”,
device=“gpu”,
memory_budget=“11GB”,
weight_dtype=“int4”,
kv_cache_dtype=“int8”,
offload=“auto”,
)
The runtime should automatically decide:
The user should be able to say:
My GPU has 12 GB. Make the best execution plan that fits.
The framework should then perform the planning.
An ideal architecture could look like this:
Application / Transformers / Local LLM Runtime
|
v
PyTorch Graph
|
v
Common GPU Runtime / IR
|
±----------±----------+
| | |
v v v
CUDA ROCm XPU
NVIDIA AMD Intel
Again, this does not require removing CUDA.
CUDA would remain the NVIDIA backend.
ROCm would remain the AMD backend.
XPU / oneAPI would remain the Intel backend.
The difference is that more model logic, memory logic, quantization logic, and kernel definitions would exist above those vendor-specific layers.
Ideally, users could write:
model.to(“gpu”)
and the runtime would internally choose:
NVIDIA → CUDA
AMD → ROCm
Intel → XPU / oneAPI
without requiring major changes in application code.
INT4, INT8, FP8, and mixed precision should not behave like optional tricks provided by unrelated third-party libraries.
They should be first-class execution and memory-management concepts.
Examples include:
A critical point is that an INT4 model should not repeatedly allocate large BF16 copies of its weights during execution.
The preferred path should look more like:
INT4 Weight
|
v
Fused Dequant + MatMul
|
v
BF16 / FP16 Output
rather than:
INT4 Weight
|
v
Temporary BF16 Weight Copy
|
v
MatMul
The latter can destroy much of the memory advantage of quantization.
Reducing weight size is not enough.
As context length grows, KV cache can become one of the largest consumers of VRAM.
A modern runtime should support:
For example, when given an 11 GB memory budget, the runtime might automatically decide:
Weights: INT4
KV Cache: INT8
Attention: memory-efficient implementation
Context Length: automatically limited to available memory
This should be planned before execution, not after an OOM crash.
Current consumer systems effectively look like this:
GPU VRAM
|
PCIe
|
System RAM
|
Storage
For AI workloads, a more explicit hierarchical memory model would be useful:
Tier 0: GPU Local VRAM
Tier 1: GPU-Side Expansion Memory
Tier 2: System RAM
Tier 3: NVMe / mmap Storage
The fastest memory.
Best suited for:
A future large-capacity memory tier physically close to the GPU.
Possible capacities:
It may be slower than local VRAM but much faster and lower-latency than system RAM over PCIe.
Possible uses:
Large and relatively inexpensive, but slower and farther from the GPU.
Useful for less frequently accessed tensors and overflow capacity.
A last-resort storage tier.
Useful for:
It should not be treated as a substitute for fast working memory.
Consumer GPUs continue to gain compute performance faster than they gain VRAM capacity.
This creates situations where a GPU has enough compute throughput for a model but cannot hold the model and its runtime state.
One possible hardware direction would be:
GPU
|
±- 12 GB High-Speed VRAM
|
±- 32-64 GB Expansion Memory
|
±- Dedicated High-Speed Link
The expansion memory does not need to be as fast as local GDDR or HBM.
However, it should be significantly better than ordinary system RAM or NVMe for GPU access.
Useful properties would include:
In such a design, 12 GB of VRAM could act as the high-speed working set while a much larger pool stores the rest of the model.
This could be especially valuable for local AI workloads.
NVMe off can prevent complete failure when a model is too large.
It does not solve the performance problem.
If every layer requires data to be read from storage and transferred to the GPU, then execution becomes transfer-bound:
Compute Time << Data Transfer Time
At that point, additional GPU compute performance is mostly wasted.
NVMe is useful for:
It is not an ideal primary memory tier for interactive inference.
The real solution is to reduce how much data must cross slow links during execution.
If PyTorch continues evolving toward a broader AI execution runtime, the following features could provide significant value.
Example:
torch.set_memory_budget(“11GB”)
or a model-specific equivalent.
The runtime could understand placements such as:
gpu_local
gpu_extended
system_ram
mmap
and move tensors automatically.
Low-bit formats should be directly represented and optimized:
An attention, quantization, or matrix-multiplication kernel could be defined once and compiled to:
backend-specific implementations.
The runtime should automatically select between:
depending on available hardware and memory.
Instead of crashing first and forcing the user to change settings, the runtime could estimate memory requirements before execution.
Expected Usage: 13.8 GB
Available Budget: 11.0 GB
Replanned Execution:
This is a better user experience than trial-and-error OOM debugging.
From a Transformers user perspective, the ideal interface could be extremely simple:
``` python
model = AutoModel.from_pretrained(
model_id,
device=“auto”,
memory_budget=“11GB”,
precision=“auto”,
)
Internally, the framework could coordinate:
Today, users often have to read documentation for several separate libraries and manually combine them.
That complexity should gradually move into the runtime itself.
This point is important.
CUDA is a mature and highly optimized NVIDIA backend.
There is no need to remove it.
The problem is that CUDA compatibility is often treated as a requirement for using a model, extension, or kernel at all.
The ideal market would allow users to choose GPUs based mainly on:
rather than mainly asking:
Does this require CUDA?
If NVIDIA provides the best hardware, users can choose NVIDIA.
If AMD provides more VRAM at a better price, users should be able to choose AMD without losing major software capabilities.
If Intel offers competitive hardware, it should also be a viable option.
That is healthier competition.
A stronger common runtime could change how consumer GPUs compete.
Current situation:
Software compatibility → GPU vendor decision
Preferred situation:
Price / VRAM / Bandwidth / Efficiency / Performance → GPU vendor decision
This would encourage all three major GPU vendors to compete on actual hardware value while still maintaining their own optimized backend stacks.
The result should be better for users, developers, and the broader AI ecosystem.
Run a 13B-class INT4 model on a 12 GB NVIDIA GPU.
Requirements:
Run the same model code on a 16 GB AMD GPU.
The application code should not require major modification.
Run the same model code on an Intel GPU.
Backend differences should remain mostly inside the framework.
When local VRAM is insufficient, automatically move selected tensors to another memory tier.
The user should not have to manually assign every layer.
Proposal: Vendor-neutral, memory-budget-aware execution for consumer GPUs
I would like to suggest a direction for PyTorch, Transformers, and local LLM runtimes.
Today, consumer GPU users face two major problems:
A 12 GB GPU may have enough compute power for a 13B-class model, especially with INT4 weights, but KV cache, temporary buffers, backend-specific kernels, and fragmented memory allocation can still cause OOM.
I think frameworks should move toward a memory-budget-aware execution model.
model = load_model(
path,
device=“gpu”,
memory_budget=“11GB”,
weight_dtype=“int4”,
kv_cache_dtype=“int8”,
offload=“auto”,
)
The goal is not to replace CUDA.
CUDA, ROCm, and XPU can remain optimized vendor backends.
The goal is to build a stronger common execution layer above them so that model code does not need to be heavily rewritten or restricted to one GPU vendor.
I also think frameworks should prepare for hierarchical GPU memory:
Tier 0: GPU local VRAM
Tier 1: future GPU-side expansion memory
Tier 2: system RAM
Tier 3: NVMe / mmap
NVMe off alone is not a real performance solution because bandwidth and latency are much worse than local GPU memory.
A future GPU-side expansion-memory tier could allow a fast 12 GB consumer GPU to use a much larger model without relying on system RAM or SSD as the primary working memory.
A concrete target could be:
There are already many relevant technologies:
However, these capabilities are still fragmented.
I think a future major version of the stack would be more valuable if it focused on unified memory management, low-bit inference, and hardware-neutral execution.
The end goal should be simple:
Users should choose GPUs based on VRAM, bandwidth, compute performance, power efficiency, and price — not primarily because a particular model or kernel only works on one vendor.
I would be interested to hear whether there are existing projects, RFCs, or research efforts moving in this direction.
AI frameworks should hide GPU-vendor differences, automatically optimize models for a fixed VRAM budget, and prepare for future GPU-side expansion memory so that 12 GB consumer GPUs can run larger models efficiently.