cd /news/large-language-models/show-hn-what-llm-fits-my-gpu-vram-me… · home › topics › large-language-models › article
[ARTICLE · art-155772] src=quantized.uk ↗ pub= topic=large-language-models verified=true sentiment=↑ positive

Show HN: What LLM fits my GPU? – VRAM measured on real hardware for 117 models

A Show HN project launched on 2026-10-23 that indexes 117 LLMs and 78 GPUs, pairing each model with VRAM measured on real hardware at 4K context and batch 1, including weights, KV cache and 10% activations. The site reports throughput and memory figures per quantization — for example, Llama 3.1 8B on an RTX 4090 24G with ExLlamaV2 EXL2 4.65bpw at 235 tok/s and 5.4 GB VRAM, versus 68 tok/s and 5.7 GB on an M3 Max 48G with Ollama GGUF Q4_K_M — and lists quality loss per quant, such as 3.8% for Mistral Small 24B Instruct AWQ INT4. The index aims to let users pick a card and task to see which models fit, the memory math behind each, and the command to run it.

read7 min views1 publishedOct 23, 2026
Show HN: What LLM fits my GPU? – VRAM measured on real hardware for 117 models
Image: source

2026-10-23latest update on your card?

Quantize everything.

Pick your card and what you want it for. You get models that actually fit, the memory maths behind each one, and the command to run it.

New here? Start with the 8GB guide

What fits your card #

Sized at 4K context, batch 1 — weights + KV cache + 10% activations, straight from the model index.

ExampleShowing RTX 4060 Ti 16G as an example — pick your own hardware above to see yours.

  • Most capable that fitsMistral Small 24B InstructAWQ · INT4
    • 13 GB of 16 GB
    • 2.8 GB free
    • 3.8% quality loss
    • 78 tok/s (RTX 4090)Estimated The largest model in the index that still fits this card comfortably at 4K context.
  • Room to growFalcon 3 10B InstructGGUF · Q4_K_M
    • 7.3 GB of 16 GB
    • 8.7 GB free
    • 3.1% quality loss
    • 118 tok/s (RTX 4090)Estimated The largest model that uses under 60% of the card, leaving space for much longer context.
  • FastestQwen2.5 0.5B InstructGGUF · Q8_0
    • 0.6 GB of 16 GB
    • 15 GB free
    • 0.5% quality loss
    • 540 tok/s (RTX 4090)Estimated Highest measured throughput of everything that fits — usually a small model. Measured on an RTX 4090, so read it as a ranking, not as your number.
[All models that fit RTX 4060 Ti 16G →](https://quantized.uk/gpu/rtx-4060-ti-16g/)

Data updated 2026-10-23

Latest additions #

The newest models and hardware in the index · data last updated2026-10-23

Full changelog →

By hardware

Every card has its own page: what fits comfortably, what is tight, what to step up to.

All 78 GPUs → By task

Filtered views of the index.

Start here

Measured, not guessed #

A sample of the benchmark table. Every row is a run on real hardware, recorded with the stack it ran on.

Model Hardware Stack tok/s VRAM
Llama 3.1 8B RTX 4090 24G ExLlamaV2 · EXL2 4.65bpw 235 5.4 GB
Qwen3 30B-A3B RTX 4090 24G llama.cpp · GGUF Q4_K_M 95 19 GB
Llama 3.1 8B M3 Max 48G Ollama · GGUF Q4_K_M 68 5.7 GB
Llama 3.1 8B RTX 3090 24G ExLlamaV2 · EXL2 4.65bpw 175 5.4 GB

Method: Meta Llama 3.1 8B Instruct · WikiText-2 · 4096 ctx · batch 1

Local inference runs on RTX 4090 / 3090 / M3 Max / M2 Ultra; llama.cpp b4000+, ExLlamaV2 0.2.x, vLLM 0.6.x, Ollama 0.3.x

Full benchmark table and methodology →

Curated standout quant releases — click to view model details

[View All →](https://quantized.uk/quant-hub/)

- [NEW](https://quantized.uk/quant-hub/k2-horizon-7b/) K2 Horizon 7BGGUFQ4_K_M · 6.5 GB at 4K · Fully open, data included, 8 GBIFMComfortable from RTX 4060 (8 GB)
- [NEW](https://quantized.uk/quant-hub/minicpm-v-4-7-35b-a3b/) MiniCPM-V 4.7 35B-A3BGGUFQ4_K_S · 20.9 GB at 4K · Vision MoE, 3B active, 24 GBOpenBMBComfortable from RTX 3090 (24 GB)
- [NEW](https://quantized.uk/quant-hub/north-mini-code-1.0/) North Mini Code 1.0 30B-A3BGGUFQ4_K_M · 20.0 GB at 4K · Coding MoE, 3B active, 24 GBbartowskiComfortable from RTX 3090 (24 GB)
- [NEW](https://quantized.uk/quant-hub/gemma-4-12b/) Gemma 4 12B ITGGUFQ4_K_M · 8.0 GB at 4K · 12B, 256K window on 16 GBunslothComfortable from Arc B570 10G (10 GB)
- [NEW](https://quantized.uk/quant-hub/beam-1.0-501b-a23b/) Beam 1.0 501B-A23BGGUFQ2_K · 190.9 GB at 4K · 501B, Apache 2.0, 1M windowDevQuasarComfortable from Mac M5 Ultra 512G (512 GB)
- [NEW](https://quantized.uk/quant-hub/ling-3.1-flash/) Ling-3.1-flashGGUFQ4_K_M · 370.6 GB at 4K · 560B, 25B active, 9 of 72 layers cacheAtomicChatNo single card fits it comfortably
- [NEW](https://quantized.uk/quant-hub/lfm2-5-24b-a2b/) LFM2.5 24B-A2BGGUFQ4_K_M · 15.3 GB at 4K · 2.45B active, 128K windowLiquidAIComfortable from Radeon RX 7900 XT (20 GB)
- [NEW](https://quantized.uk/quant-hub/nemotron-3.5-lightning-30b-a3b/) Nemotron 3.5 Lightning 30B-A3BGGUFQ4_0 · 19.8 GB at 4K · 3B active, 256K on 24 GBggml-orgComfortable from RTX 3090 (24 GB)
- [NEW](https://quantized.uk/quant-hub/nemotron-3.5-bolt-74b-a9b/) Nemotron 3.5 Bolt 74B-A9BGGUFIQ2_XXS · 23.1 GB at 4K · 74B built for 24 GBnvidiaComfortable from RTX 5090 (32 GB)
- [NEW](https://quantized.uk/quant-hub/granite-4.2-8b/) Granite 4.2 8BGGUFQ4_K_M · 6.3 GB at 4K · dense, thinking by defaultibm-graniteComfortable from RTX 4060 (8 GB)
- [NEW](https://quantized.uk/quant-hub/olmo-3-7b/) OLMo 3 7B InstructGGUFQ4_K_M · 6.8 GB at 4K · fully open, 64K windowallenaiComfortable from RTX 4060 (8 GB)
- [NEW](https://quantized.uk/quant-hub/ernie-4.5-21b-a3b/) ERNIE 4.5 21B-A3BGGUFQ4_K_M · 14.2 GB at 4K · 3B active, 128K windowbaiduComfortable from Radeon RX 7900 XT (20 GB)
- [NEW](https://quantized.uk/quant-hub/smollm3-3b/) SmolLM3 3BGGUFQ4_K_M · 2.3 GB at 4K · training data publishedHuggingFaceTBComfortable from Mac M1 8G (8 GB)
- [NEW](https://quantized.uk/quant-hub/gemma-4-e4b/) Gemma 4 E4B ITGGUFQ4_K_M · 4.9 GB at 4K · image + audio on 8 GBgoogleComfortable from Mac M1 8G (8 GB)
- [NEW](https://quantized.uk/quant-hub/gemma-4-26b-a4b/) Gemma 4 26B-A4B ITGGUFQ4_K_M · 18.2 GB at 4K · 4B active, 256K windowgoogleComfortable from RTX 3090 (24 GB)
- [NEW](https://quantized.uk/quant-hub/gemma-4-31b/) Gemma 4 31B ITGGUFQ4_K_M · 21.1 GB at 4K · 10 of 60 layers cachegoogleComfortable from RTX 3090 (24 GB)
- [NEW](https://quantized.uk/quant-hub/glm-4.7-flash/) GLM-4.7-FlashGGUFQ4_K_M · 19.2 GB at 4K · 3B active, MLA cachezai-orgComfortable from RTX 3090 (24 GB)
- [NEW](https://quantized.uk/quant-hub/kimi-linear-48b-a3b/) Kimi Linear 48B-A3B InstructGGUFQ4_K_M · 30.4 GB at 4K · 1M context, 7 of 27 layers cachemoonshotaiComfortable from A100 40G (40 GB)
- [NEW](https://quantized.uk/quant-hub/qwen3-8-27b/) Qwen3.8 27BGGUFQ4_K_M · 17.5 GB at 4K · 16 of 64 layers cache KVunslothComfortable from Radeon RX 7900 XT (20 GB)
- [NEW](https://quantized.uk/quant-hub/ministral-3-8b/) Ministral 3 8B InstructGGUFQ4_K_M · 6.0 GB at 4K · official Mistral GGUFmistralaiComfortable from RTX 4060 (8 GB)
- [NEW](https://quantized.uk/quant-hub/qwen3-vl-8b/) Qwen3-VL 8B InstructGGUFQ4_K_M · 6.2 GB at 4K · vision-language at 8BbartowskiComfortable from RTX 4060 (8 GB)
- [HOT](https://quantized.uk/quant-hub/gpt-oss-20b/) GPT-OSS 20BGGUFMXFP4 · 12.8 GB at 4K · native 4-bitopenaiComfortable from RTX 4060 Ti 16G (16 GB)
- [HOT](https://quantized.uk/quant-hub/qwen3-coder-30b-a3b/) Qwen3-Coder 30B-A3B InstructGGUFQ4_K_M · 19.7 GB at 4K · agentic coderbartowskiComfortable from RTX 3090 (24 GB)

Before you download a 20GB file #

All 27 questions

### How much VRAM do I need to run a 7B model? #

About 5.1 GB at Q4_K_M with a 4K context window — 4.2 GB of weights, 0.5 GB of KV cache and a 0.5 GB activation buffer, measured on WizardLM-2 7B (7B). Every 8 GB card in this index clears that with room left. The figure rises with context, not with how hard you use the model: the same file at 32K needs about 9.0 GB, because the KV cache grows to 4.0 GB.

### Can I run a 24B model on a 16GB card? #

At 4 bits and short context, yes — Mistral Small 24B Instruct at AWQ INT4 needs about 13.2 GB of 16 GB, leaving 2.8 GB. The level matters more than the parameter count: the same model at Q4_K_M is 15.9 GB, which is 99% of the card — it loads on a card with nothing else on it and has no margin for a longer window. A 16 GB card runs 64 of the 117 models here comfortably.

### How much quality do you lose at 4-bit? #

At Q4_K_M the median published perplexity loss across the 79 models here that report one is 2.9% — 97.1% retained — with the range running 1.4% to 5.2%. Two caveats that matter more than the median: smaller models lose more than larger ones at the same level, and perplexity is a language-modelling measure, not a measure of whether the model still does your task. Models without a published figure are shown as a dash here rather than assumed to be lossless.

### GGUF or AWQ — which should I use? #

GGUF unless you are running a server. GGUF runs on anything — CPU, NVIDIA, AMD, Apple — and 117 of the 117 models here ship it, against 53 for AWQ. AWQ's case is throughput under vLLM with batched requests, on an NVIDIA card or one of the Radeon cards vLLM's ROCm build supports; for one person talking to one model, it buys you nothing GGUF does not already do. 53 models here ship both, which is the only set where the two can be compared on the same weights.

Found a number that looks wrong? #

This index is maintained by hand and checked against public model cards. Figures marked “estimated” are calculated from the architecture, not measured on a card — if one does not match what you see on your own hardware, say so and it gets corrected.

Data Changelog #

Last updated 2026-10-23

[Show all 195 entries](https://quantized.uk/changelog/)

2026-10-23llama.cpp is now at b11829; Ollama (v0.40.8), vLLM (v0.32.0) and ExLlamaV3 (v1.6.4) are unchanged. Poolside's Laguna S 2.1, a 118B mixture-of-experts coding model with about 8B parameters active per token, is now on the not-runnable-yet list: llama.cpp supports its architecture, but its layer layout and file sizes have not been checked.

All data is manually curated and verified against community sources. Always cross-check with official Hugging Face model cards before deployment.

source & further reading
quantized.uk — original article
── more in #large-language-models 4 stories · sorted by recency
── more on @show hn 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/show-hn-what-llm-fit…] indexed:0 read:7min 2026-10-23 · —