{"slug": "how-much-vram-do-you-really-need-to-run-a-70b-llm", "title": "How Much VRAM Do You Really Need to Run a 70B LLM?", "summary": "A developer's technical breakdown of local LLM memory requirements shows that a 70-billion-parameter model needs roughly 140 GB of VRAM at FP16, 70 GB at INT8, and about 35 GB at 4-bit quantization, meaning even an 80 GB accelerator cannot hold raw FP16 weights. The analysis notes that 24 GB cards fall short of the 4-bit weight requirement and must rely on CPU offloading or multi-GPU setups, where system RAM bandwidth becomes the bottleneck for token generation, while 32 GB cards can fit some 70B-class models only with more aggressive quantization. It also flags that quantization overhead, KV cache, and context length add memory beyond the raw weight estimate.", "body_md": "Running a large language model locally sounds simple until you start looking at GPU memory.\n\nA model has 70 billion parameters. Your GPU has 24 GB, 32 GB, 48 GB, or maybe 80 GB of VRAM.\n\nSo will it fit?\n\nUnfortunately, **parameter count alone does not answer that question**.\n\nTo estimate how much VRAM an LLM actually needs, you need to consider at least four things:\n\nAnd if the model does not fit entirely in VRAM, you also need to think about **CPU offloading and multi-GPU inference**.\n\nLet's break it down.\n\nAt the most basic level, the memory required for model weights can be estimated with:\n\n```\nModel memory ≈ parameters × bits per parameter ÷ 8\n```\n\nFor a 70-billion-parameter model:\n\n| Precision | Approx. raw weight memory | \n|---|---|\n| FP16 / BF16 | 140 GB | \n| INT8 | 70 GB | \n| 6-bit | 52.5 GB | \n| 5-bit | 43.75 GB | \n| 4-bit | 35 GB | \n| 3-bit | 26.25 GB | \n\nThese numbers are only a starting point.\n\nA \"4-bit\" model does **not necessarily occupy exactly 35 GB in VRAM**.\n\nQuantization formats often include scales, metadata, higher-precision tensors, and other overhead. Different quantization methods can therefore produce noticeably different memory requirements even when both are described as \"4-bit.\"\n\nThis is why looking only at the advertised quantization level can be misleading.\n\nWithout quantization, a 70B model is far beyond the memory capacity of normal consumer GPUs.\n\nAt FP16:\n\n```\n70 billion × 2 bytes ≈ 140 GB\n```\n\nThat means even an 80 GB accelerator cannot hold the raw weights entirely in memory.\n\nAt INT8:\n\n```\n70 billion × 1 byte ≈ 70 GB\n```\n\nNow the model becomes feasible on very large accelerator cards, although there still needs to be room for runtime overhead and KV cache.\n\nAt approximately 4-bit:\n\n```\n70 billion × 0.5 bytes ≈ 35 GB\n```\n\nSuddenly the model becomes practical on configurations with around 48 GB of GPU memory.\n\nThis is why quantization has been so important for local LLM inference.\n\nInstead of needing several enterprise accelerators, heavily quantized models can sometimes run on workstation GPUs or multiple consumer cards.\n\nThe tradeoff is that increasingly aggressive quantization can affect model quality.\n\nSo the goal should not simply be:\n\nUse the smallest model possible.\n\nA better goal is:\n\nUse the highest-quality quantization that fits comfortably within the hardware you have.\n\nCards with 24 GB of VRAM are extremely useful for local AI.\n\nBut 24 GB is still substantially below the roughly 35 GB theoretical weight requirement of a 70B model at exactly 4 bits per parameter.\n\nThat means something has to change.\n\nYou can:\n\nCPU offloading is particularly interesting because it allows models much larger than GPU memory to run.\n\nBut capacity and performance are two different problems.\n\nSuppose you have:\n\n```\nGPU VRAM: 24 GB\nSystem RAM: 64 GB\n```\n\nYou potentially have enough total memory to store a quantized 70B model.\n\nThe runtime can keep some layers in GPU memory while storing the remaining layers in normal system RAM.\n\nThat works.\n\nBut system RAM bandwidth is dramatically lower than modern GPU VRAM bandwidth.\n\nFor autoregressive LLM inference, weights may need to be accessed repeatedly as each token is generated.\n\nIf part of those weights must travel between CPU memory and the GPU, token generation can slow considerably.\n\nSo when evaluating hardware for local AI, I separate two questions:\n\nand\n\nThose are not the same thing.\n\nA 32 GB GPU gets much closer.\n\nThe theoretical size of a 4-bit 70B model is still around:\n\n```\n35 GB\n```\n\nSo a straightforward 4-bit model will generally still exceed 32 GB before accounting for additional memory requirements.\n\nHowever, more aggressive quantizations can bring some 70B-class models within range.\n\nThat makes 32 GB cards interesting for users willing to trade some model fidelity for the ability to stay mostly—or entirely—on the GPU.\n\nBut fitting the weights is only part of the problem.\n\nThere is another large consumer of GPU memory:\n\n**context.**\n\nWhen an LLM processes a conversation, it stores information associated with previous tokens in a structure called the **KV cache**.\n\nThe longer the conversation becomes, the larger that cache becomes.\n\nSo a model that fits comfortably at:\n\n```\n4,096 tokens\n```\n\nmay consume considerably more memory at:\n\n```\n32,768 tokens\n```\n\nor:\n\n```\n131,072 tokens\n```\n\nThis creates one of the most common mistakes when estimating GPU requirements.\n\nSomeone downloads a model that appears to require 22 GB of memory and assumes it will fit comfortably on a 24 GB GPU.\n\nThen the runtime loads:\n\nand suddenly there isn't enough memory.\n\nConsider two people running the exact same model.\n\nRuns:\n\n```\n4K context\n1 concurrent request\n128K context\n4 concurrent requests\n```\n\nThey may have dramatically different memory requirements even though they're using the same model.\n\nThis is especially important for:\n\nIf you're buying hardware for AI, **model size should never be considered separately from intended context length**.\n\nEven after accounting for weights and KV cache, you should avoid planning a system that uses exactly 100% of available VRAM.\n\nInference frameworks need working memory.\n\nDepending on the runtime, that can include:\n\nA machine that technically fits a model with 200 MB of VRAM remaining may be much less useful than one with several gigabytes of headroom.\n\nFor that reason, I generally think of GPU memory as a **budget**, not a hard model-size limit.\n\nAround 48 GB of VRAM is where 70B-class local inference becomes significantly easier.\n\nA 4-bit 70B model with a theoretical weight size around 35 GB leaves substantially more room for:\n\nThat does not mean every 70B model and every context configuration will fit.\n\nBut compared with 24 GB or 32 GB, you have far more flexibility.\n\nThis is one reason older professional GPUs with large VRAM capacities can remain interesting for AI even when newer gaming GPUs have considerably more raw compute.\n\nFor LLM inference, sometimes:\n\n**the slower GPU that fits the whole model is more useful than the faster GPU that doesn't.**\n\nThis is another common question.\n\nAt first glance:\n\n```\n24 GB + 24 GB = 48 GB\n```\n\nSo two 24 GB GPUs should behave exactly like one 48 GB GPU.\n\nNot quite.\n\nA runtime can distribute model layers or tensors across both GPUs, allowing the combined memory capacity to hold a larger model.\n\nBut the GPUs still have physically separate memory pools.\n\nCommunication must occur over:\n\nThe topology and inference framework therefore matter.\n\nTwo GPUs can dramatically expand the models you are able to run, but they do not magically become a single GPU.\n\nStill, for local AI enthusiasts, used high-VRAM consumer GPUs can sometimes create very interesting price-to-memory configurations.\n\nOnce the model fits, another specification becomes increasingly important:\n\n**memory bandwidth.**\n\nLLM token generation frequently involves moving large amounts of model data through memory.\n\nThat means two GPUs with similar compute capability can behave very differently depending on:\n\nOther important factors include:\n\nThis is why comparing AI GPUs purely by TFLOPS is often misleading.\n\nInstead of asking:\n\nWhat is the fastest GPU?\n\nI recommend asking these questions in order.\n\n8B?\n\n32B?\n\n70B?\n\nMixture-of-Experts model?\n\nFP16?\n\nFP8?\n\nINT8?\n\nQ6?\n\nQ5?\n\nQ4?\n\n4K?\n\n32K?\n\n128K?\n\nMore?\n\nIf not:\n\nThere is a huge difference between:\n\n```\n2 tokens/sec\n```\n\nand:\n\n```\n50 tokens/sec\n```\n\nBoth configurations technically \"run\" the model.\n\nOnly one may be pleasant to use interactively.\n\nHere is a deliberately simplified way to think about it.\n\nA normal 4-bit 70B model will not fit entirely in VRAM.\n\nExpect aggressive quantization, CPU offloading, or multiple GPUs.\n\nCloser, but still below the theoretical size of a standard 4-bit 70B model.\n\nAggressive quantization may make some configurations possible.\n\nA much more comfortable target for 4-bit 70B-class inference.\n\nContext length and runtime overhead still matter.\n\nEnough for approximately 8-bit weights in theory, although KV cache and runtime overhead must still be accounted for.\n\nFP16 70B remains far above the capacity of a single 80 GB GPU.\n\nThe most important lesson is that:\n\n```\nparameters ≠ VRAM requirement\n```\n\nThe real calculation is closer to:\n\n```\nModel weights\n+ KV cache\n+ runtime overhead\n+ safety margin\n= required GPU memory\n```\n\nAnd even after answering that question, you still need to consider memory bandwidth and software support to estimate actual performance.\n\nI built [CompareAIHardware](https://compareaihardware.com/) around exactly this problem: comparing GPUs, accelerators, VRAM capacity, memory bandwidth, and model requirements from the perspective of people actually trying to run AI workloads locally.\n\nThe next time you see someone ask:\n\n\"Can I run a 70B model on my GPU?\"\n\nthe correct answer probably isn't simply yes or no.\n\nThe better answer is:\n\n**Which 70B model, which quantization, which context length, and how much of it needs to stay in VRAM?**\n\nIf you're building a local AI machine, those four questions can save you a very expensive GPU purchase.", "url": "https://wpnews.pro/news/how-much-vram-do-you-really-need-to-run-a-70b-llm", "canonical_source": "https://dev.to/peter_gedeon/how-much-vram-do-you-really-need-to-run-a-70b-llm-3jn8", "published_at": "2026-09-23 06:20:26+00:00", "updated_at": "2026-09-23 06:22:36.423971+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "ai-chips", "mlops", "ai-tools"], "entities": [], "alternates": {"html": "https://wpnews.pro/news/how-much-vram-do-you-really-need-to-run-a-70b-llm", "markdown": "https://wpnews.pro/news/how-much-vram-do-you-really-need-to-run-a-70b-llm.md", "text": "https://wpnews.pro/news/how-much-vram-do-you-really-need-to-run-a-70b-llm.txt", "jsonld": "https://wpnews.pro/news/how-much-vram-do-you-really-need-to-run-a-70b-llm.jsonld"}}