Llama 3.1 8b Instruct - Memory Usage More than Reported

## (“Memory Usage More than Reported”)

Late to this thread but it keeps being the #1 surprise, so for future readers — the missing 16 GB is the KV cache, and you can compute it from config.json alone:

KV bytes = 2 (K+V) × layers × kv_heads × head_dim × context × bytes/elem

Llama-3.1-8B: 2 × 32 × 8 × 128 × 131072 × 2 (fp16) = 16 GB (128 KB per token!)

So at the advertised 128K context: 15 GB of weights + 16 GB of cache + scratch ≈ 32 GB — it was never going to fit in 24 GB, and it’s not the model’s “fault”: it’s the context. Three levers, cheapest first: quantize the cache (q8_0 halves it), cap the context (~66K fits in 24 GB with fp16 weights), or drop weight precision.

I packaged this arithmetic (plus the “which side is the problem” verdict and the max-context solver) into a free browser tool — geometry fetched from the model card, nothing downloaded: TAF Agent — Test ANY Transformer LLM in Your Browser