Sizing VRAM for a Self-Hosted LLM: 7B to 70B on a Rented GPU Server

Most "will it fit" questions about self-hosted models get answered with a rule of thumb: two bytes per parameter, add some headroom, done. That rule tells you whether the weights load. It does not tell you whether the server survives fifty concurrent users with long prompts, which is the question that actually decides what card you rent.

This post walks through the arithmetic I use before picking hardware for inference and fine-tuning, with a small Python calculator, a working vLLM launch config and the mistakes that cost the most time.

During inference, GPU memory is split between:

Training adds gradients and optimizer state on top, which changes the picture completely. We will get to that.

Bytes per parameter by precision:

The 4-bit column is not exactly 0.5 because quantized formats store scales and zero points per group of weights, and some layers (embeddings, the output head) often stay in higher precision. Check the actual size of the safetensors files rather than trusting the multiplication.

So a 70B model in BF16 does not fit on one 80 GB card, needs two of them with tensor parallelism, and fits on a single 141 GB card with room for cache. The same model in 4-bit fits on a 48 GB card, but with very little left over for users.

For every token in every active sequence, each transformer layer stores a key and a value vector. The formula:

kv_bytes_per_token = 2 * num_layers * num_kv_heads * head_dim * bytes_per_element

The 2 is key plus value. num_kv_heads is the important term: models with grouped-query attention (GQA) share key/value heads across several query heads, which shrinks the cache dramatically.

Real numbers, FP16 cache:

Now multiply by load. Say you serve Llama 3 70B with an 8,192-token context and want 32 sequences in flight:

320 KiB * 8192 tokens = 2.5 GiB per sequence
2.5 GiB * 32 sequences = 80 GiB

That is 80 GiB of cache on top of 140 GB of weights. The "it fits on two 80 GB cards" plan turns into "it loads on two cards and serves about four users at full context". The model starts, the demo works, and throughput collapses under real traffic.

In practice not every request uses the full context, and paged attention (vLLM, SGLang, TensorRT-LLM) allocates cache in blocks only for tokens that exist. So size against your real prompt length distribution, not the theoretical maximum. Log prompt plus completion lengths from a pilot and use the 90th percentile.

from dataclasses import dataclass

GiB = 1024 ** 3

@dataclass
class Model:
    params_b: float      # billions of parameters
    layers: int
    kv_heads: int
    head_dim: int

def inference_vram_gib(m: Model, weight_bytes: float, kv_bytes: float,
                       ctx_tokens: int, concurrent: int,
                       overhead_gib: float = 2.0, activation_frac: float = 0.05):
    weights = m.params_b * 1e9 * weight_bytes / GiB
    kv_per_token = 2 * m.layers * m.kv_heads * m.head_dim * kv_bytes
    kv = kv_per_token * ctx_tokens * concurrent / GiB
    activations = (weights + kv) * activation_frac
    total = weights + kv + activations + overhead_gib
    return round(weights, 1), round(kv, 1), round(total, 1)

llama3_8b = Model(8.0, 32, 8, 128)
llama3_70b = Model(70.6, 80, 8, 128)

print(inference_vram_gib(llama3_8b, 2, 2, 8192, 16))    # BF16 weights, FP16 cache
print(inference_vram_gib(llama3_70b, 2, 2, 8192, 32))
print(inference_vram_gib(llama3_70b, 0.55, 1, 8192, 32)) # 4-bit weights, FP8 cache

Output:

(14.9, 16.0, 34.4)
(131.5, 80.0, 224.1)
(36.2, 40.0, 82.0)

Read the last line carefully. Quantizing weights to 4-bit and the cache to FP8 brings a 32-user 70B deployment from "three or four 80 GB cards" down to "one 80 GB card, slightly over budget". Drop concurrency to 24 or context to 6k and it fits. Whether the quality loss from 4-bit weights is acceptable is a separate question that you answer with your own eval set, not with a benchmark table.

The activation_frac of 5% is a rough allowance. Long-prompt prefill on large batches can need more; vLLM handles this by reserving memory during its profiling run at startup, which is why its numbers differ from naive math.

vLLM takes a fraction of GPU memory, loads weights, and gives everything left to the KV cache. The flags that matter:

vllm serve meta-llama/Meta-Llama-3-70B-Instruct \
  --tensor-parallel-size 2 \
  --gpu-memory-utilization 0.90 \
  --max-model-len 8192 \
  --max-num-seqs 32 \
  --kv-cache-dtype fp8 \
  --enable-prefix-caching

--gpu-memory-utilization 0.90 leaves 10% for the CUDA context and fragmentation. Going to 0.95 works until a monitoring agent or a second process grabs memory, then you get OOM at 3 a.m.--max-model-len caps context. If you leave it at the model's native 128k, vLLM may refuse to start because one maximum-length sequence does not fit in the cache, or it starts with very few concurrent slots.--max-num-seqs limits how many sequences are scheduled at once. Too high and requests get preempted and recomputed when the cache fills; too low and the GPU idles.--kv-cache-dtype fp8 halves cache memory. Supported on Hopper and Ada cards; measure output quality before shipping it.--enable-prefix-caching reuses cache for shared prompt prefixes. If every request starts with the same 2,000-token system prompt, this is a large win. At startup vLLM logs the number of GPU blocks it allocated for the cache and the maximum concurrency it can sustain at full context. Read that line before running a load test. It is the single best check of whether your sizing was right.

Full fine-tuning with AdamW in mixed precision costs about 16 bytes per parameter before activations:

For a 7B model that is about 112 GB, so full fine-tuning even a small model needs several cards or ZeRO/FSDP sharding plus optimizer offload. Activations come on top and scale with batch size and sequence length; gradient checkpointing trades roughly 30% more compute for a large cut there.

LoRA freezes the base weights and trains small adapter matrices, so optimizer state almost disappears. A 7B model with LoRA fits on a 24 to 48 GB card. QLoRA loads the base model in 4-bit and trains adapters on top: the original QLoRA work fine-tuned a 65B model on a single 48 GB GPU. For most business use cases (tone, format, domain vocabulary) LoRA on an 8B or 70B base is the sensible starting point.

Rough mapping for inference with moderate concurrency, assuming you size the cache as above:

Memory bandwidth matters as much as capacity. Token generation is bandwidth-bound: every new token reads all the weights once. A card with more HBM bandwidth generates tokens faster at the same model size, which is a big part of why H100 and H200 are worth the premium for latency-sensitive serving.

Multi-GPU without NVLink. Tensor parallelism across PCIe-only cards works, but the all-reduce on every layer eats into throughput. Ask how the cards are connected before you plan a two-card split.

Driver and CUDA mismatch. A wheel built for one CUDA version on a host driver that is too old gives cryptic errors. Pin your container base image and check nvidia-smi for the driver's maximum supported CUDA version first.

Disk and download time. A 70B BF16 checkpoint is around 140 GB. On a fresh server, down it can take longer than the first experiment. Put model weights on local NVMe, not a network mount, or load times stretch into tens of minutes.

Benchmarking with one user. A single-stream test measures latency, not capacity. Use a load generator with your real prompt length distribution and watch time-to-first-token as concurrency grows.

Ignoring data residency. If the prompts contain customer records, where the GPU physically sits can be a legal constraint, not a preference. Check it before choosing a provider, not after the security review.

The utilisation threshold is the deciding number. A card that is busy less than about half the time is usually cheaper to rent, because owned hardware depreciates while idle. Prototyping, evaluation of several model sizes and periodic fine-tuning runs almost always fall below that line. A model serving production traffic around the clock for several years can justify buying.

The practical sequence: rent, measure real utilisation and real VRAM use for a month with the calculator above as your starting estimate, then decide. If you need hardware located inside Kazakhstan with CUDA, PyTorch and Docker already set up, billed by the hour for experiments or monthly for long runs, here is how our GPU server rental with H100, H200, A100 and L40S cards works.