{"slug": "gpu-ai-runtime-memory-proposal", "title": "Gpu_ai_runtime_memory_proposal", "summary": "A new proposal calls for a common GPU execution and memory-management layer above CUDA, ROCm, and Intel XPU/oneAPI so that AI workloads are not tightly coupled to a single vendor, targeting 13B-class INT4 inference on 12 GB consumer GPUs. The proposal states its goal is not to replace CUDA, which would remain the NVIDIA backend, but to move model, memory, quantization, and kernel logic above the vendor-specific layers. It argues the main challenge is controlling the entire runtime memory footprint, including KV cache, rather than merely quantizing weights.", "body_md": "AI hardware is improving faster than the software stack that is supposed to use it.\n\nNVIDIA, AMD, and Intel all produce increasingly capable GPUs with better matrix acceleration, lower-precision compute, higher memory bandwidth, and stronger AI-oriented hardware. However, users still often have to choose a GPU based on software compatibility first and hardware capability second.\n\nIn practice, the first question is often not:\n\nWhich GPU has the best VRAM capacity, bandwidth, compute performance, power efficiency, and price?\n\nInstead, it is:\n\nDoes this model, extension, quantization library, or attention kernel work with CUDA?\n\nThat is not a healthy long-term situation for a multi-vendor GPU market.\n\nThe goal of this proposal is **not to replace CUDA**.\n\nCUDA, ROCm, and Intel XPU / oneAPI should continue to exist as optimized vendor backends.\n\nThe goal is to build a stronger common execution and memory-management layer above them so that application code, model code, and common AI workloads do not have to be tightly coupled to one GPU vendor.\n\nLocal AI users repeatedly encounter the same classes of problems:\n\nThe problem is therefore not just raw GPU compute.\n\nThe software stack needs better control over **memory placement, low-bit execution, backend abstraction, and runtime planning**.\n\nA useful practical target would be:\n\nMake 13B-class INT4 inference practical on 12 GB consumer GPUs without requiring users to manually combine many separate libraries and backend-specific workarounds.\n\nApproximate model weight sizes for a 13B model are:\n\nThis means that INT4 weights can theoretically fit comfortably within 12 GB.\n\nThe real problem is everything else:\n\nSo the main challenge is not merely quantizing the weights. It is controlling the **entire runtime memory footprint**.\n\nInstead of forcing users to manually tune `device_map`, `max_memory`, offload folders, quantization settings, and backend-specific flags, the runtime should accept a simple memory budget.\n\nFor example:\n\n``` python\n\nmodel = load_model(\n\npath=“model”,\n\ndevice=“gpu”,\n\nmemory_budget=“11GB”,\n\nweight_dtype=“int4”,\n\nkv_cache_dtype=“int8”,\n\noffload=“auto”,\n\n)\n\n```\n\nThe runtime should automatically decide:\n\nThe user should be able to say:\n\nMy GPU has 12 GB. Make the best execution plan that fits.\n\nThe framework should then perform the planning.\n\nAn ideal architecture could look like this:\n\n``` text\n\nApplication / Transformers / Local LLM Runtime\n\n|\n\nv\n\nPyTorch Graph\n\n|\n\nv\n\nCommon GPU Runtime / IR\n\n|\n\n±----------±----------+\n\n| | |\n\nv v v\n\nCUDA ROCm XPU\n\nNVIDIA AMD Intel\n\n```\n\nAgain, this does not require removing CUDA.\n\nCUDA would remain the NVIDIA backend.\n\nROCm would remain the AMD backend.\n\nXPU / oneAPI would remain the Intel backend.\n\nThe difference is that more model logic, memory logic, quantization logic, and kernel definitions would exist **above** those vendor-specific layers.\n\nIdeally, users could write:\n\n``` python\n\nmodel.to(“gpu”)\n\n```\n\nand the runtime would internally choose:\n\n``` text\n\nNVIDIA → CUDA\n\nAMD → ROCm\n\nIntel → XPU / oneAPI\n\n```\n\nwithout requiring major changes in application code.\n\nINT4, INT8, FP8, and mixed precision should not behave like optional tricks provided by unrelated third-party libraries.\n\nThey should be first-class execution and memory-management concepts.\n\nExamples include:\n\nA critical point is that an INT4 model should not repeatedly allocate large BF16 copies of its weights during execution.\n\nThe preferred path should look more like:\n\n``` text\n\nINT4 Weight\n\n|\n\nv\n\nFused Dequant + MatMul\n\n|\n\nv\n\nBF16 / FP16 Output\n\n```\n\nrather than:\n\n``` text\n\nINT4 Weight\n\n|\n\nv\n\nTemporary BF16 Weight Copy\n\n|\n\nv\n\nMatMul\n\n```\n\nThe latter can destroy much of the memory advantage of quantization.\n\nReducing weight size is not enough.\n\nAs context length grows, KV cache can become one of the largest consumers of VRAM.\n\nA modern runtime should support:\n\nFor example, when given an 11 GB memory budget, the runtime might automatically decide:\n\n``` text\n\nWeights: INT4\n\nKV Cache: INT8\n\nAttention: memory-efficient implementation\n\nContext Length: automatically limited to available memory\n\n```\n\nThis should be planned before execution, not after an OOM crash.\n\nCurrent consumer systems effectively look like this:\n\n``` text\n\nGPU VRAM\n\n|\n\nPCIe\n\n|\n\nSystem RAM\n\n|\n\nStorage\n\n```\n\nFor AI workloads, a more explicit hierarchical memory model would be useful:\n\n``` text\n\nTier 0: GPU Local VRAM\n\nTier 1: GPU-Side Expansion Memory\n\nTier 2: System RAM\n\nTier 3: NVMe / mmap Storage\n\n```\n\nThe fastest memory.\n\nBest suited for:\n\nA future large-capacity memory tier physically close to the GPU.\n\nPossible capacities:\n\nIt may be slower than local VRAM but much faster and lower-latency than system RAM over PCIe.\n\nPossible uses:\n\nLarge and relatively inexpensive, but slower and farther from the GPU.\n\nUseful for less frequently accessed tensors and overflow capacity.\n\nA last-resort storage tier.\n\nUseful for:\n\nIt should not be treated as a substitute for fast working memory.\n\nConsumer GPUs continue to gain compute performance faster than they gain VRAM capacity.\n\nThis creates situations where a GPU has enough compute throughput for a model but cannot hold the model and its runtime state.\n\nOne possible hardware direction would be:\n\n``` text\n\nGPU\n\n|\n\n±- 12 GB High-Speed VRAM\n\n|\n\n±- 32-64 GB Expansion Memory\n\n|\n\n±- Dedicated High-Speed Link\n\n```\n\nThe expansion memory does not need to be as fast as local GDDR or HBM.\n\nHowever, it should be significantly better than ordinary system RAM or NVMe for GPU access.\n\nUseful properties would include:\n\nIn such a design, 12 GB of VRAM could act as the high-speed working set while a much larger pool stores the rest of the model.\n\nThis could be especially valuable for local AI workloads.\n\nNVMe offloading can prevent complete failure when a model is too large.\n\nIt does not solve the performance problem.\n\nIf every layer requires data to be read from storage and transferred to the GPU, then execution becomes transfer-bound:\n\n``` text\n\nCompute Time << Data Transfer Time\n\n```\n\nAt that point, additional GPU compute performance is mostly wasted.\n\nNVMe is useful for:\n\nIt is not an ideal primary memory tier for interactive inference.\n\nThe real solution is to reduce how much data must cross slow links during execution.\n\nIf PyTorch continues evolving toward a broader AI execution runtime, the following features could provide significant value.\n\nExample:\n\n``` python\n\ntorch.set_memory_budget(“11GB”)\n\n```\n\nor a model-specific equivalent.\n\nThe runtime could understand placements such as:\n\n``` text\n\ngpu_local\n\ngpu_extended\n\nsystem_ram\n\nmmap\n\n```\n\nand move tensors automatically.\n\nLow-bit formats should be directly represented and optimized:\n\nAn attention, quantization, or matrix-multiplication kernel could be defined once and compiled to:\n\nbackend-specific implementations.\n\nThe runtime should automatically select between:\n\ndepending on available hardware and memory.\n\nInstead of crashing first and forcing the user to change settings, the runtime could estimate memory requirements before execution.\n\n``` text\n\nExpected Usage: 13.8 GB\n\nAvailable Budget: 11.0 GB\n\nReplanned Execution:\n\nThis is a better user experience than trial-and-error OOM debugging.\n\nFrom a Transformers user perspective, the ideal interface could be extremely simple:\n\n``` python\n\nmodel = AutoModel.from_pretrained(\n\nmodel_id,\n\ndevice=“auto”,\n\nmemory_budget=“11GB”,\n\nprecision=“auto”,\n\n)\n\n```\n\nInternally, the framework could coordinate:\n\nToday, users often have to read documentation for several separate libraries and manually combine them.\n\nThat complexity should gradually move into the runtime itself.\n\nThis point is important.\n\nCUDA is a mature and highly optimized NVIDIA backend.\n\nThere is no need to remove it.\n\nThe problem is that CUDA compatibility is often treated as a requirement for using a model, extension, or kernel at all.\n\nThe ideal market would allow users to choose GPUs based mainly on:\n\nrather than mainly asking:\n\nDoes this require CUDA?\n\nIf NVIDIA provides the best hardware, users can choose NVIDIA.\n\nIf AMD provides more VRAM at a better price, users should be able to choose AMD without losing major software capabilities.\n\nIf Intel offers competitive hardware, it should also be a viable option.\n\nThat is healthier competition.\n\nA stronger common runtime could change how consumer GPUs compete.\n\nCurrent situation:\n\n``` text\n\nSoftware compatibility → GPU vendor decision\n\n```\n\nPreferred situation:\n\n``` text\n\nPrice / VRAM / Bandwidth / Efficiency / Performance → GPU vendor decision\n\n```\n\nThis would encourage all three major GPU vendors to compete on actual hardware value while still maintaining their own optimized backend stacks.\n\nThe result should be better for users, developers, and the broader AI ecosystem.\n\nRun a 13B-class INT4 model on a 12 GB NVIDIA GPU.\n\nRequirements:\n\nRun the same model code on a 16 GB AMD GPU.\n\nThe application code should not require major modification.\n\nRun the same model code on an Intel GPU.\n\nBackend differences should remain mostly inside the framework.\n\nWhen local VRAM is insufficient, automatically move selected tensors to another memory tier.\n\nThe user should not have to manually assign every layer.\n\n**Proposal: Vendor-neutral, memory-budget-aware execution for consumer GPUs**\n\nI would like to suggest a direction for PyTorch, Transformers, and local LLM runtimes.\n\nToday, consumer GPU users face two major problems:\n\nA 12 GB GPU may have enough compute power for a 13B-class model, especially with INT4 weights, but KV cache, temporary buffers, backend-specific kernels, and fragmented memory allocation can still cause OOM.\n\nI think frameworks should move toward a **memory-budget-aware execution model**.\n\n``` python\n\nmodel = load_model(\n\npath,\n\ndevice=“gpu”,\n\nmemory_budget=“11GB”,\n\nweight_dtype=“int4”,\n\nkv_cache_dtype=“int8”,\n\noffload=“auto”,\n\n)\n\n```\n\nThe goal is **not to replace CUDA**.\n\nCUDA, ROCm, and XPU can remain optimized vendor backends.\n\nThe goal is to build a stronger common execution layer above them so that model code does not need to be heavily rewritten or restricted to one GPU vendor.\n\nI also think frameworks should prepare for hierarchical GPU memory:\n\n``` text\n\nTier 0: GPU local VRAM\n\nTier 1: future GPU-side expansion memory\n\nTier 2: system RAM\n\nTier 3: NVMe / mmap\n\n```\n\nNVMe offloading alone is not a real performance solution because bandwidth and latency are much worse than local GPU memory.\n\nA future GPU-side expansion-memory tier could allow a fast 12 GB consumer GPU to use a much larger model without relying on system RAM or SSD as the primary working memory.\n\nA concrete target could be:\n\nThere are already many relevant technologies:\n\nHowever, these capabilities are still fragmented.\n\nI think a future major version of the stack would be more valuable if it focused on unified memory management, low-bit inference, and hardware-neutral execution.\n\nThe end goal should be simple:\n\nUsers should choose GPUs based on VRAM, bandwidth, compute performance, power efficiency, and price — not primarily because a particular model or kernel only works on one vendor.\n\nI would be interested to hear whether there are existing projects, RFCs, or research efforts moving in this direction.\n\n**AI frameworks should hide GPU-vendor differences, automatically optimize models for a fixed VRAM budget, and prepare for future GPU-side expansion memory so that 12 GB consumer GPUs can run larger models efficiently.**", "url": "https://wpnews.pro/news/gpu-ai-runtime-memory-proposal", "canonical_source": "https://discuss.huggingface.co/t/gpu-ai-runtime-memory-proposal/180288#post_1", "published_at": "2026-09-11 23:47:21+00:00", "updated_at": "2026-09-11 23:52:09.213252+00:00", "lang": "en", "topics": ["ai-infrastructure", "ai-chips", "large-language-models", "ai-tools", "mlops"], "entities": ["NVIDIA", "AMD", "Intel", "CUDA", "ROCm", "oneAPI", "PyTorch"], "alternates": {"html": "https://wpnews.pro/news/gpu-ai-runtime-memory-proposal", "markdown": "https://wpnews.pro/news/gpu-ai-runtime-memory-proposal.md", "text": "https://wpnews.pro/news/gpu-ai-runtime-memory-proposal.txt", "jsonld": "https://wpnews.pro/news/gpu-ai-runtime-memory-proposal.jsonld"}}