{"slug": "the-16-gb-cliff-measured-llm-memory-and-speed-on-an-rtx-5080", "title": "The 16 GB Cliff: Measured LLM Memory and Speed on an RTX 5080", "summary": "A hands-on benchmark of 36 measured local language-model configurations on a retail Nvidia RTX 5080 found that llama3.2:3b at Q4_K_M quantization and 4,096 context ran at a median 351.22 tok/s decode while occupying 3,760 MiB of the card's 16,303 MiB, or 23.1% of total memory. The same model dropped from 351.22 tok/s at 4,096 context to 102.68 tok/s at 131,072 context, and a separate stock-power run measured 318.3 tok/s at 265.2 W board power, or 0.833 J per token. All figures were captured with Ollama 0.32.15 on its vendored llama.cpp build 9d77fa172, with whole-card memory read from nvidia-smi including the Windows desktop.", "body_md": "Every pin is a local language model I ran on my own retail RTX 5080, placed by the memory in use on the whole card and the decode speed that came with it. Nothing here is estimated. For a card I do not own or a setup I have not run, the LLM VRAM calculator and the LLM speed calculator estimate, and the VRAM tiers guide covers other card sizes. This page only reports.\n\nMeasured. Copied from a capture file, shown with its capture date.\n\nComputed. Arithmetic on measured values, shown with its formula.\n\nShowing 36 of 36 measured configurations\n\nThe map\n\nAcross: memory in use on the whole card. Up: decode speed.\n\ntok/s0100200300400\n\nllama3.2:3b, 4,096 to 131,072 context351.22 to 102.68 tok/s\n\n04,0968,19212,28816,303MiB in usecard total\n\nA measured configuration\n\nRan partly off the card\n\nReading includes another model\n\nSame setup, context raised\n\nEvery pin is one row from a capture file. Nothing between two pins was measured, and nothing here is an estimate.\n\nReadout\n\nNothing selected\n\nClick or tap near a pin, or pick a row in the list, and its measured record appears here.\n\nMeasured at 3,760 MiB resident on with Ollama 0.32.15. That figure is the whole card as nvidia-smi reported it, Windows desktop included.\n\nResident, whole card\n\n3,760 MiB\n\nMeasured\n\nDecode\n\n351.22 tok/s\n\nMeasured\n\nLeft on the card\n\n12,543 MiB\n\nComputed: 16,303 − 3,760\n\nShare of the card\n\n23.1%\n\nComputed: 3,760 ÷ 16,303\n\nFull measured row\n\nCaptured\n\nfrom the file's name\n\nApp\n\nollama 0.32.15\n\nEngine\n\nvendored llama.cpp 9d77fa172\n\nCondition\n\ndefaults\n\nModel name in the file\n\nllama3.2:3b\n\nQuantization\n\nQ4_K_M. Not a column in this file. Taken from the same model in the throughput CSV.\n\nContext\n\n4096\n\nRuns\n\n5\n\nDecode, median (server counters)\n\n351.22 tok/s\n\nDecode, median (wall clock)\n\n345.19 tok/s\n\nEach run\n\n351.15 · 350.9 · 351.86 · 351.32 · 351.22 tok/s\n\nPrefill, long prompt, median\n\n17802.7 tok/s at 434 prompt tokens\n\nTime to first token\n\nEmpty in this file\n\nWhole-card memory, median\n\n3760 MiB\n\nWhole-card memory before the load\n\n1053 MiB\n\nAdded by loading the model\n\n2,707 MiB Computed: 3,760 − 1,053\n\nLoad\n\n2.67 s\n\nCold first request\n\n2.79 s\n\nApp RAM while loaded\n\n1234.0 MB\n\nDriver\n\nNot recorded in this file\n\nResident in GiB\n\n3.67 GiB Computed: 3,760 ÷ 1,024\n\nEnergy for this model, from its own power run\n\nA separate run of llama3.2:3b on (UTC), stock power limit: 318.3 tok/s at 265.2 W board power, 0.833 J per token in the file. That is 833 J per 1,000 tokensComputed: 0.833 × 1,000. The power file records no context length, driver or runtime version, so read it as the model's energy at stock, not this row's.\n\nMeasured at 3,760 MiB resident on with Ollama 0.32.15. That figure is the whole card as nvidia-smi reported it, Windows desktop included.\n\nResident, whole card\n\n3,760 MiB\n\nMeasured\n\nDecode\n\n350.52 tok/s\n\nMeasured\n\nLeft on the card\n\n12,543 MiB\n\nComputed: 16,303 − 3,760\n\nShare of the card\n\n23.1%\n\nComputed: 3,760 ÷ 16,303\n\nFull measured row\n\nCaptured\n\nfrom the file's name\n\nApp\n\nollama 0.32.15\n\nEngine\n\nvendored llama.cpp 9d77fa172\n\nCondition\n\nnormalized\n\nModel name in the file\n\nllama3.2:3b\n\nQuantization\n\nQ4_K_M. Not a column in this file. Taken from the same model in the throughput CSV.\n\nContext\n\n4096\n\nRuns\n\n5\n\nDecode, median (server counters)\n\n350.52 tok/s\n\nDecode, median (wall clock)\n\n338.11 tok/s\n\nEach run\n\n349.35 · 350.38 · 350.52 · 350.63 · 350.96 tok/s\n\nPrefill, long prompt, median\n\n17862.3 tok/s at 432 prompt tokens\n\nTime to first token\n\nEmpty in this file\n\nWhole-card memory, median\n\n3760 MiB\n\nWhole-card memory before the load\n\n1057 MiB\n\nAdded by loading the model\n\n2,703 MiB Computed: 3,760 − 1,057\n\nLoad\n\n2.27 s\n\nCold first request\n\n2.38 s\n\nApp RAM while loaded\n\n1239.6 MB\n\nDriver\n\nNot recorded in this file\n\nResident in GiB\n\n3.67 GiB Computed: 3,760 ÷ 1,024\n\nEnergy for this model, from its own power run\n\nA separate run of llama3.2:3b on (UTC), stock power limit: 318.3 tok/s at 265.2 W board power, 0.833 J per token in the file. That is 833 J per 1,000 tokensComputed: 0.833 × 1,000. The power file records no context length, driver or runtime version, so read it as the model's energy at stock, not this row's.\n\nMeasured at 4,343 MiB resident on with Ollama 0.32.1. That figure is the whole card as nvidia-smi reported it, Windows desktop included.\n\nResident, whole card\n\n4,343 MiB\n\nMeasured\n\nDecode\n\n318.51 tok/s\n\nMeasured\n\nLeft on the card\n\n11,960 MiB\n\nComputed: 16,303 − 4,343\n\nShare of the card\n\n26.6%\n\nComputed: 4,343 ÷ 16,303\n\nFull measured row\n\nCaptured\n\nfrom the file's footer notes\n\nRuntime\n\nollama 0.32.1\n\nModel name in the file\n\nllama3.2:3b\n\nQuantization\n\nQ4_K_M. Not a column in this file. Taken from the same model in the throughput CSV.\n\nContext\n\n4096 from the file's settings note\n\nOffload\n\nFull offload from the file's settings note\n\nRuns\n\n5\n\nDecode, median (server counters)\n\n318.51 tok/s\n\nDecode, median (wall clock)\n\n313.25 tok/s\n\nEach run\n\n319.0 · 318.7 · 318.2 · 318.5 · 317.5 tok/s\n\nPrefill, short prompt, median\n\n7038.1 tok/s\n\nPrefill, long prompt\n\n17444 tok/s at 433 prompt tokens\n\nWhole-card memory, median\n\n4343 MiB\n\nLoad\n\n3.2 s\n\nDesktop baseline at bench start\n\n1642 MiB one reading for the whole capture\n\nDriver\n\nNot recorded in this file\n\nResident in GiB\n\n4.24 GiB Computed: 4,343 ÷ 1,024\n\nEnergy for this model, from its own power run\n\nA separate run of llama3.2:3b on (UTC), stock power limit: 318.3 tok/s at 265.2 W board power, 0.833 J per token in the file. That is 833 J per 1,000 tokensComputed: 0.833 × 1,000. The power file records no context length, driver or runtime version, so read it as the model's energy at stock, not this row's.\n\nMeasured at 5,495 MiB resident on with Ollama 0.32.1. That figure is the whole card as nvidia-smi reported it, Windows desktop included.\n\nResident, whole card\n\n5,495 MiB\n\nMeasured\n\nDecode\n\n312.89 tok/s\n\nMeasured\n\nLeft on the card\n\n10,808 MiB\n\nComputed: 16,303 − 5,495\n\nShare of the card\n\n33.7%\n\nComputed: 5,495 ÷ 16,303\n\nFull measured row\n\nCaptured (UTC)\n\nGPU\n\nNVIDIA GeForce RTX 5080\n\nDriver\n\n610.88\n\nRuntime\n\nOllama 0.32.1\n\nMethod\n\nllm-throughput-ollama-v1\n\nModel\n\nllama3.2:3b digest a80c4f17acd5\n\nParameters\n\n3.2B\n\nQuantization\n\nQ4_K_M\n\nContext (num_ctx)\n\n16384\n\nPrompt cell 1\n\nThe row pinned on the map\n\nPrompt tokens\n\n471\n\nRepetitions\n\n3\n\nDecode, median\n\n312.89 tok/s\n\nPrefill, median\n\n17885.62 tok/s\n\nLoad, median\n\n162.9 ms\n\nWhole-card memory after the cell\n\n5495 MiB\n\nPrompt cell 2\n\nSame configuration, longer prompt\n\nPrompt tokens\n\n1657\n\nRepetitions\n\n3\n\nDecode, median\n\n298.59 tok/s\n\nPrefill, median\n\n19397.36 tok/s\n\nLoad, median\n\n158.0 ms\n\nWhole-card memory after the cell\n\n5501 MiB\n\nResident in GiB\n\n5.37 GiB Computed: 5,495 ÷ 1,024\n\nEnergy for this model, from its own power run\n\nA separate run of llama3.2:3b on (UTC), stock power limit: 318.3 tok/s at 265.2 W board power, 0.833 J per token in the file. That is 833 J per 1,000 tokensComputed: 0.833 × 1,000. The power file records no context length, driver or runtime version, so read it as the model's energy at stock, not this row's.\n\nMeasured at 6,190 MiB resident on with Ollama 0.32.1. That figure is the whole card as nvidia-smi reported it, Windows desktop included.\n\nResident, whole card\n\n6,190 MiB\n\nMeasured\n\nDecode\n\n180.02 tok/s\n\nMeasured\n\nLeft on the card\n\n10,113 MiB\n\nComputed: 16,303 − 6,190\n\nShare of the card\n\n38.0%\n\nComputed: 6,190 ÷ 16,303\n\nFull measured row\n\nCaptured (UTC)\n\nGPU\n\nNVIDIA GeForce RTX 5080\n\nDriver\n\n610.88\n\nRuntime\n\nOllama 0.32.1\n\nMethod\n\nllm-throughput-ollama-v1\n\nModel\n\nqwen2.5:7b digest 845dbda0ea48\n\nParameters\n\n7.6B\n\nQuantization\n\nQ4_K_M\n\nContext (num_ctx)\n\n4096\n\nPrompt cell 1\n\nThe row pinned on the map\n\nPrompt tokens\n\n488\n\nRepetitions\n\n3\n\nDecode, median\n\n180.02 tok/s\n\nPrefill, median\n\n10152.28 tok/s\n\nLoad, median\n\n129.8 ms\n\nWhole-card memory after the cell\n\n6190 MiB\n\nPrompt cell 2\n\nSame configuration, longer prompt\n\nPrompt tokens\n\n1674\n\nRepetitions\n\n3\n\nDecode, median\n\n176.79 tok/s\n\nPrefill, median\n\n9672.39 tok/s\n\nLoad, median\n\n123.4 ms\n\nWhole-card memory after the cell\n\n6190 MiB\n\nResident in GiB\n\n6.04 GiB Computed: 6,190 ÷ 1,024\n\nEnergy for this model, from its own power run\n\nA separate run of qwen2.5:7b on (UTC), stock power limit: 182.2 tok/s at 343.6 W board power, 1.886 J per token in the file. That is 1,886 J per 1,000 tokensComputed: 1.886 × 1,000. The power file records no context length, driver or runtime version, so read it as the model's energy at stock, not this row's.\n\nMeasured at 7,022 MiB resident on with Ollama 0.32.1. That figure is the whole card as nvidia-smi reported it, Windows desktop included.\n\nResident, whole card\n\n7,022 MiB\n\nMeasured\n\nDecode\n\n179.66 tok/s\n\nMeasured\n\nLeft on the card\n\n9,281 MiB\n\nComputed: 16,303 − 7,022\n\nShare of the card\n\n43.1%\n\nComputed: 7,022 ÷ 16,303\n\nFull measured row\n\nCaptured (UTC)\n\nGPU\n\nNVIDIA GeForce RTX 5080\n\nDriver\n\n610.88\n\nRuntime\n\nOllama 0.32.1\n\nMethod\n\nllm-throughput-ollama-v1\n\nModel\n\nqwen2.5:7b digest 845dbda0ea48\n\nParameters\n\n7.6B\n\nQuantization\n\nQ4_K_M\n\nContext (num_ctx)\n\n16384\n\nPrompt cell 1\n\nThe row pinned on the map\n\nPrompt tokens\n\n488\n\nRepetitions\n\n3\n\nDecode, median\n\n179.66 tok/s\n\nPrefill, median\n\n10133.1 tok/s\n\nLoad, median\n\n127.9 ms\n\nWhole-card memory after the cell\n\n7022 MiB\n\nPrompt cell 2\n\nSame configuration, longer prompt\n\nPrompt tokens\n\n1674\n\nRepetitions\n\n3\n\nDecode, median\n\n176.61 tok/s\n\nPrefill, median\n\n9984.37 tok/s\n\nLoad, median\n\n127.7 ms\n\nWhole-card memory after the cell\n\n7034 MiB\n\nResident in GiB\n\n6.86 GiB Computed: 7,022 ÷ 1,024\n\nEnergy for this model, from its own power run\n\nA separate run of qwen2.5:7b on (UTC), stock power limit: 182.2 tok/s at 343.6 W board power, 1.886 J per token in the file. That is 1,886 J per 1,000 tokensComputed: 1.886 × 1,000. The power file records no context length, driver or runtime version, so read it as the model's energy at stock, not this row's.\n\nMeasured at 10,356 MiB resident on with Ollama 0.32.15. That figure is the whole card as nvidia-smi reported it, Windows desktop included.\n\nResident, whole card\n\n10,356 MiB\n\nMeasured\n\nDecode\n\n102.25 tok/s\n\nMeasured\n\nLeft on the card\n\n5,947 MiB\n\nComputed: 16,303 − 10,356\n\nShare of the card\n\n63.5%\n\nComputed: 10,356 ÷ 16,303\n\nFull measured row\n\nCaptured\n\nfrom the file's name\n\nApp\n\nollama 0.32.15\n\nEngine\n\nvendored llama.cpp 9d77fa172\n\nCondition\n\ndefaults\n\nModel name in the file\n\nqwen2.5:14b\n\nQuantization\n\nQ4_K_M. Not a column in this file. Taken from the same model in the throughput CSV.\n\nContext\n\n4096\n\nRuns\n\n5\n\nDecode, median (server counters)\n\n102.25 tok/s\n\nDecode, median (wall clock)\n\n101.06 tok/s\n\nEach run\n\n102.13 · 102.34 · 102.25 · 102.3 · 102.25 tok/s\n\nPrefill, long prompt, median\n\n4983.8 tok/s at 453 prompt tokens\n\nTime to first token\n\nEmpty in this file\n\nWhole-card memory, median\n\n10356 MiB\n\nWhole-card memory before the load\n\n1053 MiB\n\nAdded by loading the model\n\n9,303 MiB Computed: 10,356 − 1,053\n\nLoad\n\n5.51 s\n\nCold first request\n\n5.67 s\n\nApp RAM while loaded\n\n1604.1 MB\n\nDriver\n\nNot recorded in this file\n\nResident in GiB\n\n10.11 GiB Computed: 10,356 ÷ 1,024\n\nEnergy for this model, from its own power run\n\nA separate run of qwen2.5:14b on (UTC), stock power limit: 98.8 tok/s at 311.6 W board power, 3.154 J per token in the file. That is 3,154 J per 1,000 tokensComputed: 3.154 × 1,000. The power file records no context length, driver or runtime version, so read it as the model's energy at stock, not this row's.\n\nMeasured at 10,356 MiB resident on with Ollama 0.32.15. That figure is the whole card as nvidia-smi reported it, Windows desktop included.\n\nResident, whole card\n\n10,356 MiB\n\nMeasured\n\nDecode\n\n102.21 tok/s\n\nMeasured\n\nLeft on the card\n\n5,947 MiB\n\nComputed: 16,303 − 10,356\n\nShare of the card\n\n63.5%\n\nComputed: 10,356 ÷ 16,303\n\nFull measured row\n\nCaptured\n\nfrom the file's name\n\nApp\n\nollama 0.32.15\n\nEngine\n\nvendored llama.cpp 9d77fa172\n\nCondition\n\nnormalized\n\nModel name in the file\n\nqwen2.5:14b\n\nQuantization\n\nQ4_K_M. Not a column in this file. Taken from the same model in the throughput CSV.\n\nContext\n\n4096\n\nRuns\n\n5\n\nDecode, median (server counters)\n\n102.21 tok/s\n\nDecode, median (wall clock)\n\n101.23 tok/s\n\nEach run\n\n102.21 · 102.21 · 102.17 · 102.21 · 102.21 tok/s\n\nPrefill, long prompt, median\n\n5038.3 tok/s at 452 prompt tokens\n\nTime to first token\n\nEmpty in this file\n\nWhole-card memory, median\n\n10356 MiB\n\nWhole-card memory before the load\n\n1053 MiB\n\nAdded by loading the model\n\n9,303 MiB Computed: 10,356 − 1,053\n\nLoad\n\n5.52 s\n\nCold first request\n\n5.68 s\n\nApp RAM while loaded\n\n1617.6 MB\n\nDriver\n\nNot recorded in this file\n\nResident in GiB\n\n10.11 GiB Computed: 10,356 ÷ 1,024\n\nEnergy for this model, from its own power run\n\nA separate run of qwen2.5:14b on (UTC), stock power limit: 98.8 tok/s at 311.6 W board power, 3.154 J per token in the file. That is 3,154 J per 1,000 tokensComputed: 3.154 × 1,000. The power file records no context length, driver or runtime version, so read it as the model's energy at stock, not this row's.\n\nMeasured at 10,841 MiB resident on with Ollama 0.32.1. That figure is the whole card as nvidia-smi reported it, Windows desktop included.\n\nResident, whole card\n\n10,841 MiB\n\nMeasured\n\nDecode\n\n96.65 tok/s\n\nMeasured\n\nLeft on the card\n\n5,462 MiB\n\nComputed: 16,303 − 10,841\n\nShare of the card\n\n66.5%\n\nComputed: 10,841 ÷ 16,303\n\nFull measured row\n\nCaptured (UTC)\n\nGPU\n\nNVIDIA GeForce RTX 5080\n\nDriver\n\n610.88\n\nRuntime\n\nOllama 0.32.1\n\nMethod\n\nllm-throughput-ollama-v1\n\nModel\n\nqwen2.5:14b digest 7cdf5a0187d5\n\nParameters\n\n14.8B\n\nQuantization\n\nQ4_K_M\n\nContext (num_ctx)\n\n4096\n\nPrompt cell 1\n\nThe row pinned on the map\n\nPrompt tokens\n\n488\n\nRepetitions\n\n3\n\nDecode, median\n\n96.65 tok/s\n\nPrefill, median\n\n5202.23 tok/s\n\nLoad, median\n\n182.5 ms\n\nWhole-card memory after the cell\n\n10841 MiB\n\nPrompt cell 2\n\nSame configuration, longer prompt\n\nPrompt tokens\n\n1674\n\nRepetitions\n\n3\n\nDecode, median\n\n94.48 tok/s\n\nPrefill, median\n\n4876.41 tok/s\n\nLoad, median\n\n125.8 ms\n\nWhole-card memory after the cell\n\n10855 MiB\n\nResident in GiB\n\n10.59 GiB Computed: 10,841 ÷ 1,024\n\nEnergy for this model, from its own power run\n\nA separate run of qwen2.5:14b on (UTC), stock power limit: 98.8 tok/s at 311.6 W board power, 3.154 J per token in the file. That is 3,154 J per 1,000 tokensComputed: 3.154 × 1,000. The power file records no context length, driver or runtime version, so read it as the model's energy at stock, not this row's.\n\nMeasured at 10,941 MiB resident on with Ollama 0.32.1. That figure is the whole card as nvidia-smi reported it, Windows desktop included.\n\nResident, whole card\n\n10,941 MiB\n\nMeasured\n\nDecode\n\n99.43 tok/s\n\nMeasured\n\nLeft on the card\n\n5,362 MiB\n\nComputed: 16,303 − 10,941\n\nShare of the card\n\n67.1%\n\nComputed: 10,941 ÷ 16,303\n\nFull measured row\n\nCaptured\n\nfrom the file's footer notes\n\nRuntime\n\nollama 0.32.1\n\nModel name in the file\n\nqwen2.5:14b\n\nQuantization\n\nQ4_K_M. Not a column in this file. Taken from the same model in the throughput CSV.\n\nContext\n\n4096 from the file's settings note\n\nOffload\n\nFull offload from the file's settings note\n\nRuns\n\n5\n\nDecode, median (server counters)\n\n99.43 tok/s\n\nDecode, median (wall clock)\n\n98.59 tok/s\n\nEach run\n\n99.5 · 99.5 · 99.4 · 99.4 · 99.3 tok/s\n\nPrefill, short prompt, median\n\n3151.4 tok/s\n\nPrefill, long prompt\n\n5004 tok/s at 451 prompt tokens\n\nWhole-card memory, median\n\n10941 MiB\n\nLoad\n\n5.6 s\n\nDesktop baseline at bench start\n\n1642 MiB one reading for the whole capture\n\nDriver\n\nNot recorded in this file\n\nResident in GiB\n\n10.68 GiB Computed: 10,941 ÷ 1,024\n\nEnergy for this model, from its own power run\n\nA separate run of qwen2.5:14b on (UTC), stock power limit: 98.8 tok/s at 311.6 W board power, 3.154 J per token in the file. That is 3,154 J per 1,000 tokensComputed: 3.154 × 1,000. The power file records no context length, driver or runtime version, so read it as the model's energy at stock, not this row's.\n\ngpt-oss:20bMXFP4 · 8,192 context · LM Studio 0.4.21 · defaults\n\nMeasured at 12,310 MiB resident on with LM Studio 0.4.21. That figure is the whole card as nvidia-smi reported it, Windows desktop included.\n\nRan partly off the card\n\nLM Studio's default automatic GPU offload was in effect, at its default 8,192 context. The dataset attributes the lower decode rate to that setting. The normalized row is the same model with full offload. Runtime comparison dataset.\n\nResident, whole card\n\n12,310 MiB\n\nMeasured\n\nDecode\n\n186.26 tok/s\n\nMeasured\n\nLeft on the card\n\n3,993 MiB\n\nComputed: 16,303 − 12,310\n\nShare of the card\n\n75.5%\n\nComputed: 12,310 ÷ 16,303\n\nFull measured row\n\nCaptured\n\nfrom the file's name\n\nApp\n\nLM Studio 0.4.21\n\nEngine\n\nllama.cpp-win-x86_64-nvidia-cuda12-avx2@2.29.1\n\nCondition\n\ndefaults\n\nModel name in the file\n\ngpt-oss-20b\n\nQuantization\n\nMXFP4. Not a column in this file. Taken from the same model in the throughput CSV.\n\nMeasured at 13,157 MiB resident on with Ollama 0.32.1. That figure is the whole card as nvidia-smi reported it, Windows desktop included.\n\nResident, whole card\n\n13,157 MiB\n\nMeasured\n\nDecode\n\n97.02 tok/s\n\nMeasured\n\nLeft on the card\n\n3,146 MiB\n\nComputed: 16,303 − 13,157\n\nShare of the card\n\n80.7%\n\nComputed: 13,157 ÷ 16,303\n\nFull measured row\n\nCaptured (UTC)\n\nGPU\n\nNVIDIA GeForce RTX 5080\n\nDriver\n\n610.88\n\nRuntime\n\nOllama 0.32.1\n\nMethod\n\nllm-throughput-ollama-v1\n\nModel\n\nqwen2.5:14b digest 7cdf5a0187d5\n\nParameters\n\n14.8B\n\nQuantization\n\nQ4_K_M\n\nContext (num_ctx)\n\n16384\n\nPrompt cell 1\n\nThe row pinned on the map\n\nPrompt tokens\n\n488\n\nRepetitions\n\n3\n\nDecode, median\n\n97.02 tok/s\n\nPrefill, median\n\n5225.96 tok/s\n\nLoad, median\n\n131.3 ms\n\nWhole-card memory after the cell\n\n13157 MiB\n\nPrompt cell 2\n\nSame configuration, longer prompt\n\nPrompt tokens\n\n1674\n\nRepetitions\n\n3\n\nDecode, median\n\n94.58 tok/s\n\nPrefill, median\n\n4864.67 tok/s\n\nLoad, median\n\n128.9 ms\n\nWhole-card memory after the cell\n\n13159 MiB\n\nResident in GiB\n\n12.85 GiB Computed: 13,157 ÷ 1,024\n\nEnergy for this model, from its own power run\n\nA separate run of qwen2.5:14b on (UTC), stock power limit: 98.8 tok/s at 311.6 W board power, 3.154 J per token in the file. That is 3,154 J per 1,000 tokensComputed: 3.154 × 1,000. The power file records no context length, driver or runtime version, so read it as the model's energy at stock, not this row's.\n\nMeasured at 13,381 MiB resident on with Ollama 0.32.1. That figure is the whole card as nvidia-smi reported it, Windows desktop included.\n\nA re-capture of the same model. The dataset's change log notes it ran below the 2026-08-12 rows on a card several hours into continuous bench work. Throughput dataset, change log.\n\nResident, whole card\n\n13,381 MiB\n\nMeasured\n\nDecode\n\n181.50 tok/s\n\nMeasured\n\nLeft on the card\n\n2,922 MiB\n\nComputed: 16,303 − 13,381\n\nShare of the card\n\n82.1%\n\nComputed: 13,381 ÷ 16,303\n\nFull measured row\n\nCaptured (UTC)\n\nGPU\n\nNVIDIA GeForce RTX 5080\n\nDriver\n\n610.88\n\nRuntime\n\nOllama 0.32.1\n\nMethod\n\nllm-throughput-ollama-v1\n\nModel\n\ngpt-oss:20b digest 17052f91a42e\n\nParameters\n\n20.9B\n\nQuantization\n\nMXFP4\n\nContext (num_ctx)\n\n4096\n\nPrompt cell 1\n\nThe row pinned on the map\n\nPrompt tokens\n\n503\n\nRepetitions\n\n3\n\nDecode, median\n\n181.5 tok/s\n\nPrefill, median\n\n9607.49 tok/s\n\nLoad, median\n\n383.7 ms\n\nWhole-card memory after the cell\n\n13381 MiB\n\nPrompt cell 2\n\nSame configuration, longer prompt\n\nPrompt tokens\n\n1360\n\nRepetitions\n\n3\n\nDecode, median\n\n178.37 tok/s\n\nPrefill, median\n\n11001.37 tok/s\n\nLoad, median\n\n375.9 ms\n\nWhole-card memory after the cell\n\n13385 MiB\n\nResident in GiB\n\n13.07 GiB Computed: 13,381 ÷ 1,024\n\nEnergy for this model, from its own power run\n\nA separate run of gpt-oss:20b on (UTC), stock power limit: 194.4 tok/s at 292.4 W board power, 1.504 J per token in the file. That is 1,504 J per 1,000 tokensComputed: 1.504 × 1,000. The power file records no context length, driver or runtime version, so read it as the model's energy at stock, not this row's.\n\nMeasured at 13,570 MiB resident on with Ollama 0.32.15. That figure is the whole card as nvidia-smi reported it, Windows desktop included.\n\nResident, whole card\n\n13,570 MiB\n\nMeasured\n\nDecode\n\n213.54 tok/s\n\nMeasured\n\nLeft on the card\n\n2,733 MiB\n\nComputed: 16,303 − 13,570\n\nShare of the card\n\n83.2%\n\nComputed: 13,570 ÷ 16,303\n\nFull measured row\n\nCaptured\n\nfrom the file's name\n\nApp\n\nollama 0.32.15\n\nEngine\n\nvendored llama.cpp 9d77fa172\n\nCondition\n\ndefaults\n\nModel name in the file\n\ngpt-oss:20b\n\nQuantization\n\nMXFP4. Not a column in this file. Taken from the same model in the throughput CSV.\n\nContext\n\n4096\n\nRuns\n\n5\n\nDecode, median (server counters)\n\n213.54 tok/s\n\nDecode, median (wall clock)\n\n212.03 tok/s\n\nEach run\n\n213.22 · 213.64 · 213.48 · 213.54 · 213.76 tok/s\n\nPrefill, long prompt, median\n\n9239.4 tok/s at 475 prompt tokens\n\nTime to first token\n\nEmpty in this file\n\nWhole-card memory, median\n\n13570 MiB\n\nWhole-card memory before the load\n\n1053 MiB\n\nAdded by loading the model\n\n12,517 MiB Computed: 13,570 − 1,053\n\nLoad\n\n18.56 s\n\nCold first request\n\n23.67 s\n\nApp RAM while loaded\n\n2138.8 MB\n\nDriver\n\nNot recorded in this file\n\nResident in GiB\n\n13.25 GiB Computed: 13,570 ÷ 1,024\n\nEnergy for this model, from its own power run\n\nA separate run of gpt-oss:20b on (UTC), stock power limit: 194.4 tok/s at 292.4 W board power, 1.504 J per token in the file. That is 1,504 J per 1,000 tokensComputed: 1.504 × 1,000. The power file records no context length, driver or runtime version, so read it as the model's energy at stock, not this row's.\n\nMeasured at 13,570 MiB resident on with Ollama 0.32.15. That figure is the whole card as nvidia-smi reported it, Windows desktop included.\n\nResident, whole card\n\n13,570 MiB\n\nMeasured\n\nDecode\n\n213.65 tok/s\n\nMeasured\n\nLeft on the card\n\n2,733 MiB\n\nComputed: 16,303 − 13,570\n\nShare of the card\n\n83.2%\n\nComputed: 13,570 ÷ 16,303\n\nFull measured row\n\nCaptured\n\nfrom the file's name\n\nApp\n\nollama 0.32.15\n\nEngine\n\nvendored llama.cpp 9d77fa172\n\nCondition\n\nnormalized\n\nModel name in the file\n\ngpt-oss:20b\n\nQuantization\n\nMXFP4. Not a column in this file. Taken from the same model in the throughput CSV.\n\nContext\n\n4096\n\nRuns\n\n5\n\nDecode, median (server counters)\n\n213.65 tok/s\n\nDecode, median (wall clock)\n\n211.8 tok/s\n\nEach run\n\n213.62 · 213.65 · 213.68 · 213.25 · 213.73 tok/s\n\nPrefill, long prompt, median\n\n9187.1 tok/s at 474 prompt tokens\n\nTime to first token\n\nEmpty in this file\n\nWhole-card memory, median\n\n13570 MiB\n\nWhole-card memory before the load\n\n1053 MiB\n\nAdded by loading the model\n\n12,517 MiB Computed: 13,570 − 1,053\n\nLoad\n\n8.05 s\n\nCold first request\n\n8.36 s\n\nApp RAM while loaded\n\n2152.1 MB\n\nDriver\n\nNot recorded in this file\n\nResident in GiB\n\n13.25 GiB Computed: 13,570 ÷ 1,024\n\nEnergy for this model, from its own power run\n\nA separate run of gpt-oss:20b on (UTC), stock power limit: 194.4 tok/s at 292.4 W board power, 1.504 J per token in the file. That is 1,504 J per 1,000 tokensComputed: 1.504 × 1,000. The power file records no context length, driver or runtime version, so read it as the model's energy at stock, not this row's.\n\nMeasured at 13,683 MiB resident on with Ollama 0.32.1. That figure is the whole card as nvidia-smi reported it, Windows desktop included.\n\nA re-capture of the same model. The dataset's change log notes it ran below the 2026-08-12 rows on a card several hours into continuous bench work. Throughput dataset, change log.\n\nResident, whole card\n\n13,683 MiB\n\nMeasured\n\nDecode\n\n183.00 tok/s\n\nMeasured\n\nLeft on the card\n\n2,620 MiB\n\nComputed: 16,303 − 13,683\n\nShare of the card\n\n83.9%\n\nComputed: 13,683 ÷ 16,303\n\nFull measured row\n\nCaptured (UTC)\n\nGPU\n\nNVIDIA GeForce RTX 5080\n\nDriver\n\n610.88\n\nRuntime\n\nOllama 0.32.1\n\nMethod\n\nllm-throughput-ollama-v1\n\nModel\n\ngpt-oss:20b digest 17052f91a42e\n\nParameters\n\n20.9B\n\nQuantization\n\nMXFP4\n\nContext (num_ctx)\n\n16384\n\nPrompt cell 1\n\nThe row pinned on the map\n\nPrompt tokens\n\n503\n\nRepetitions\n\n3\n\nDecode, median\n\n183.0 tok/s\n\nPrefill, median\n\n9092.22 tok/s\n\nLoad, median\n\n347.7 ms\n\nWhole-card memory after the cell\n\n13683 MiB\n\nPrompt cell 2\n\nSame configuration, longer prompt\n\nPrompt tokens\n\n1360\n\nRepetitions\n\n3\n\nDecode, median\n\n178.34 tok/s\n\nPrefill, median\n\n10328.62 tok/s\n\nLoad, median\n\n345.3 ms\n\nWhole-card memory after the cell\n\n13683 MiB\n\nResident in GiB\n\n13.36 GiB Computed: 13,683 ÷ 1,024\n\nEnergy for this model, from its own power run\n\nA separate run of gpt-oss:20b on (UTC), stock power limit: 194.4 tok/s at 292.4 W board power, 1.504 J per token in the file. That is 1,504 J per 1,000 tokensComputed: 1.504 × 1,000. The power file records no context length, driver or runtime version, so read it as the model's energy at stock, not this row's.\n\nMeasured at 14,067 MiB resident on with Ollama 0.32.1. That figure is the whole card as nvidia-smi reported it, Windows desktop included.\n\nResident, whole card\n\n14,067 MiB\n\nMeasured\n\nDecode\n\n190.14 tok/s\n\nMeasured\n\nLeft on the card\n\n2,236 MiB\n\nComputed: 16,303 − 14,067\n\nShare of the card\n\n86.3%\n\nComputed: 14,067 ÷ 16,303\n\nFull measured row\n\nCaptured (UTC)\n\nGPU\n\nNVIDIA GeForce RTX 5080\n\nDriver\n\n610.88\n\nRuntime\n\nOllama 0.32.1\n\nMethod\n\nllm-throughput-ollama-v1\n\nModel\n\ngpt-oss:20b digest 17052f91a42e\n\nParameters\n\n20.9B\n\nQuantization\n\nMXFP4\n\nContext (num_ctx)\n\n4096\n\nPrompt cell 1\n\nThe row pinned on the map\n\nPrompt tokens\n\n503\n\nRepetitions\n\n3\n\nDecode, median\n\n190.14 tok/s\n\nPrefill, median\n\n9078.28 tok/s\n\nLoad, median\n\n297.7 ms\n\nWhole-card memory after the cell\n\n14067 MiB\n\nPrompt cell 2\n\nSame configuration, longer prompt\n\nPrompt tokens\n\n1658\n\nRepetitions\n\n3\n\nDecode, median\n\n187.27 tok/s\n\nPrefill, median\n\n10464.66 tok/s\n\nLoad, median\n\n279.5 ms\n\nWhole-card memory after the cell\n\n14060 MiB\n\nResident in GiB\n\n13.74 GiB Computed: 14,067 ÷ 1,024\n\nEnergy for this model, from its own power run\n\nA separate run of gpt-oss:20b on (UTC), stock power limit: 194.4 tok/s at 292.4 W board power, 1.504 J per token in the file. That is 1,504 J per 1,000 tokensComputed: 1.504 × 1,000. The power file records no context length, driver or runtime version, so read it as the model's energy at stock, not this row's.\n\nMeasured at 14,152 MiB resident on with Ollama 0.32.1. That figure is the whole card as nvidia-smi reported it, Windows desktop included.\n\nResident, whole card\n\n14,152 MiB\n\nMeasured\n\nDecode\n\n195.79 tok/s\n\nMeasured\n\nLeft on the card\n\n2,151 MiB\n\nComputed: 16,303 − 14,152\n\nShare of the card\n\n86.8%\n\nComputed: 14,152 ÷ 16,303\n\nFull measured row\n\nCaptured\n\nfrom the file's footer notes\n\nRuntime\n\nollama 0.32.1\n\nModel name in the file\n\ngpt-oss:20b\n\nQuantization\n\nMXFP4. Not a column in this file. Taken from the same model in the throughput CSV.\n\nContext\n\n4096 from the file's settings note\n\nOffload\n\nFull offload from the file's settings note\n\nRuns\n\n5\n\nDecode, median (server counters)\n\n195.79 tok/s\n\nDecode, median (wall clock)\n\n194.59 tok/s\n\nEach run\n\n195.8 · 195.7 · 196.3 · 196.1 · 195.7 tok/s\n\nPrefill, short prompt, median\n\n3744.3 tok/s\n\nPrefill, long prompt\n\nEmpty in this file\n\nWhole-card memory, median\n\n14152 MiB\n\nLoad\n\n8.6 s\n\nDesktop baseline at bench start\n\n1642 MiB one reading for the whole capture\n\nDriver\n\nNot recorded in this file\n\nResident in GiB\n\n13.82 GiB Computed: 14,152 ÷ 1,024\n\nEnergy for this model, from its own power run\n\nA separate run of gpt-oss:20b on (UTC), stock power limit: 194.4 tok/s at 292.4 W board power, 1.504 J per token in the file. That is 1,504 J per 1,000 tokensComputed: 1.504 × 1,000. The power file records no context length, driver or runtime version, so read it as the model's energy at stock, not this row's.\n\nMeasured at 14,355 MiB resident on with Ollama 0.32.1. That figure is the whole card as nvidia-smi reported it, Windows desktop included.\n\nResident, whole card\n\n14,355 MiB\n\nMeasured\n\nDecode\n\n188.63 tok/s\n\nMeasured\n\nLeft on the card\n\n1,948 MiB\n\nComputed: 16,303 − 14,355\n\nShare of the card\n\n88.1%\n\nComputed: 14,355 ÷ 16,303\n\nFull measured row\n\nCaptured (UTC)\n\nGPU\n\nNVIDIA GeForce RTX 5080\n\nDriver\n\n610.88\n\nRuntime\n\nOllama 0.32.1\n\nMethod\n\nllm-throughput-ollama-v1\n\nModel\n\ngpt-oss:20b digest 17052f91a42e\n\nParameters\n\n20.9B\n\nQuantization\n\nMXFP4\n\nContext (num_ctx)\n\n16384\n\nPrompt cell 1\n\nThe row pinned on the map\n\nPrompt tokens\n\n503\n\nRepetitions\n\n3\n\nDecode, median\n\n188.63 tok/s\n\nPrefill, median\n\n9286.61 tok/s\n\nLoad, median\n\n299.2 ms\n\nWhole-card memory after the cell\n\n14355 MiB\n\nPrompt cell 2\n\nSame configuration, longer prompt\n\nPrompt tokens\n\n1658\n\nRepetitions\n\n3\n\nDecode, median\n\n187.73 tok/s\n\nPrefill, median\n\n10554.32 tok/s\n\nLoad, median\n\n280.2 ms\n\nWhole-card memory after the cell\n\n14355 MiB\n\nResident in GiB\n\n14.02 GiB Computed: 14,355 ÷ 1,024\n\nEnergy for this model, from its own power run\n\nA separate run of gpt-oss:20b on (UTC), stock power limit: 194.4 tok/s at 292.4 W board power, 1.504 J per token in the file. That is 1,504 J per 1,000 tokensComputed: 1.504 × 1,000. The power file records no context length, driver or runtime version, so read it as the model's energy at stock, not this row's.\n\nMeasured at 14,997 MiB resident on with Ollama 0.32.1. That figure is the whole card as nvidia-smi reported it, Windows desktop included.\n\nMeasured at a 4,096 context only. The dataset reports that an 8K attempt spilled and decode collapsed, but no row for that attempt was published, so it has no pin here. Throughput dataset, coding models note.\n\nMeasured at 15,003 MiB resident on with Ollama 0.32.1. That figure is the whole card as nvidia-smi reported it, Windows desktop included.\n\nRan partly off the card\n\nThe model file is larger than the card, so Ollama ran it partially offloaded with part of it in system RAM. The dataset warns that its decode rate depends on how much system RAM was free when it loaded. Throughput dataset, coding models note.\n\nMeasured at 15,084 MiB resident on with Ollama 0.32.15. That figure is the whole card as nvidia-smi reported it, Windows desktop included.\n\nRan partly off the card\n\nThe capture's own ollama ps line reads 22%/78% CPU/GPU at context 131,072, so part of this model ran from system RAM.\n\nOllama's app-level context slider was set to 256k, which this model clamps to 131,072. The dataset describes the result as the KV cache spilling to the CPU. Runtime comparison dataset.\n\nResident, whole card\n\n15,084 MiB\n\nMeasured\n\nDecode\n\n102.68 tok/s\n\nMeasured\n\nLeft on the card\n\n1,219 MiB\n\nComputed: 16,303 − 15,084\n\nShare of the card\n\n92.5%\n\nComputed: 15,084 ÷ 16,303\n\nFull measured row\n\nCaptured (UTC)\n\nGPU\n\nNVIDIA GeForce RTX 5080\n\nDriver\n\n610.88\n\nRuntime\n\nOllama 0.32.15\n\nCondition\n\ndefaults\n\nSchema\n\ntechfuelhq:lmstudio-vs-ollama:1\n\nModel\n\nllama3.2:3b\n\nQuantization\n\nQ4_K_M. Not a field in this file. Taken from the same model in the throughput CSV.\n\nContext\n\n131072 from the capture's ollama ps line\n\nProcessor split\n\n22% CPU / 78% GPU from the capture's ollama ps line\n\nSize reported by ollama ps\n\n18 GB\n\nRuns\n\n5\n\nDecode, median (server counters)\n\n102.68 tok/s\n\nDecode, median (wall clock)\n\n101.58 tok/s\n\nEach run\n\n97.57 · 99.85 · 103.59 · 102.68 · 103.21 tok/s\n\nPrefill, long prompt, median\n\n9998.8 tok/s at 434 prompt tokens\n\nWhole-card memory, median\n\n15084 MiB\n\nWhole-card memory before the load\n\n1067 MiB\n\nAdded by loading the model\n\n14,017 MiB Computed: 15,084 − 1,067\n\nLoad\n\n4.02 s\n\nCold first request\n\n4.2 s\n\nApp RAM while loaded\n\n4745.7 MB\n\nResident in GiB\n\n14.73 GiB Computed: 15,084 ÷ 1,024\n\nEnergy for this model, from its own power run\n\nA separate run of llama3.2:3b on (UTC), stock power limit: 318.3 tok/s at 265.2 W board power, 0.833 J per token in the file. That is 833 J per 1,000 tokensComputed: 0.833 × 1,000. The power file records no context length, driver or runtime version, so read it as the model's energy at stock, not this row's.\n\nMeasured at 15,101 MiB resident on with Ollama 0.32.1. That figure is the whole card as nvidia-smi reported it, Windows desktop included.\n\nRan partly off the card\n\nThe model file is larger than the card, so Ollama ran it partially offloaded with part of it in system RAM. The dataset warns that its decode rate depends on how much system RAM was free when it loaded. Throughput dataset, coding models note.\n\nMeasured at 15,682 MiB resident on with Ollama 0.32.1. That figure is the whole card as nvidia-smi reported it, Windows desktop included.\n\nReading includes another model\n\ngpt-oss:20b was still loaded from the cell before this one, so this reading is the whole card with two models on it, not Llama 3.2 3B alone. The 16,384-context row from the same capture was read after the larger model had been evicted. Throughput dataset, VRAM note.\n\nResident, whole card\n\n15,682 MiB\n\nMeasured\n\nDecode\n\n311.96 tok/s\n\nMeasured\n\nLeft on the card\n\n621 MiB\n\nComputed: 16,303 − 15,682\n\nShare of the card\n\n96.2%\n\nComputed: 15,682 ÷ 16,303\n\nFull measured row\n\nCaptured (UTC)\n\nGPU\n\nNVIDIA GeForce RTX 5080\n\nDriver\n\n610.88\n\nRuntime\n\nOllama 0.32.1\n\nMethod\n\nllm-throughput-ollama-v1\n\nModel\n\nllama3.2:3b digest a80c4f17acd5\n\nParameters\n\n3.2B\n\nQuantization\n\nQ4_K_M\n\nContext (num_ctx)\n\n4096\n\nPrompt cell 1\n\nThe row pinned on the map\n\nPrompt tokens\n\n471\n\nRepetitions\n\n3\n\nDecode, median\n\n311.96 tok/s\n\nPrefill, median\n\n18196.57 tok/s\n\nLoad, median\n\n162.8 ms\n\nWhole-card memory after the cell\n\n15682 MiB\n\nPrompt cell 2\n\nSame configuration, longer prompt\n\nPrompt tokens\n\n1657\n\nRepetitions\n\n3\n\nDecode, median\n\n298.21 tok/s\n\nPrefill, median\n\n17681.08 tok/s\n\nLoad, median\n\n156.2 ms\n\nWhole-card memory after the cell\n\n15682 MiB\n\nResident in GiB\n\n15.31 GiB Computed: 15,682 ÷ 1,024\n\nEnergy for this model, from its own power run\n\nA separate run of llama3.2:3b on (UTC), stock power limit: 318.3 tok/s at 265.2 W board power, 0.833 J per token in the file. That is 833 J per 1,000 tokensComputed: 0.833 × 1,000. The power file records no context length, driver or runtime version, so read it as the model's energy at stock, not this row's.\n\nNo measured configuration matches those filters. That combination was not on the bench, and I do not estimate here. For an estimate, use the LLM VRAM calculator.\n\nHow to read it\n\nThe map answers a narrow question: what did this model, at this context length, on this runtime version, take on a 16 GB card, and what speed came with it. Each record reads “measured at so many MiB resident on a date with a runtime version” because that is all a measurement can say. It is not a promise that the same model fits on your card today.\n\nThe line along the bottom is the card, from zero to the total it reports to nvidia-smi. A pin’s position across is the memory in use on the whole card while that model was loaded, Windows desktop included, and its height is the decode rate. Select a pin and the line fills to it in the measured colour. The hatched remainder is headroom, which I compute by subtracting.\n\nA dashed connector joins the same model, runtime and capture at two context lengths. Flat means memory grew and speed held; a drop means the model stopped fitting. A ring is a run recorded as partly off the card. A diamond is a reading taken while another model was still loaded, so it overstates that model.\n\nWhere the cliff shows up\n\nThe cliff is not a memory number. In these files, a model that sits whole on the card decodes at the same speed whether it uses a third of the memory or nearly all of it. Speed falls when part of the model no longer fits and runs from system RAM instead.\n\nEach row is the same model, runtime version and capture at two context lengths, and both ends are measured rows. Memory change is the second reading minus the first. Decode change is (second − first) ÷ first.\n\nModel and runtimeContextResident MiB, measuredMemory change, computedDecode tok/s, measuredDecode change, computed\n\nllama3.2:3bOllama 0.32.15, defaults · 131,072: ran partly off the card4,096 → 131,0723,760 → 15,084+11,324351.22 → 102.68−70.8%\n\nqwen3-coder:30bOllama 0.32.1 · 16,384: ran partly off the card4,096 → 16,38415,101 → 15,003−9861.89 → 51.62−16.6%\n\nThe table has two rows where raising the context cost speed, and both are runs that left the card. Llama 3.2 3B is the small model in this set, and a large enough context still pushed it over. Qwen3-Coder 30B never fit whole, so its memory reading barely moved and its speed did.\n\nEvery other pair held its decode rate within one percent while the context quadrupled. What they paid was memory, and how much depended on the model: the two 14B Qwen models paid the most, and gpt-oss:20b, the largest of them by parameter count, paid the least.\n\nWhat the files do not hold is a dense model caught in the act of falling off. The throughput dataset says Devstral 24B collapsed when I tried it at an 8K context, but that attempt was never written to the CSV, so there is no pin for it and I have not drawn one.\n\nEnergy per thousand tokens\n\nllama3.2:3bQ4_K_M · named “Llama 3.2 3B” in the file · run logged UTC\n\n833 J per 1,000 tokens Computed: 0.833 J per token × 1,000\n\nDecode, median of 3\n\n318.3 tok/s\n\nBoard power, median while loaded\n\n265.2 W\n\nTokens per joule, as filed\n\n1.2\n\nJoules per token, as filed\n\n0.833\n\nMore from the power file\n\nWatt-hours per 1,000 tokens\n\n0.231 Wh Computed: 231.0 Wh per million ÷ 1,000\n\nDecode, slowest and fastest run\n\n315.5 to 318.5 tok/s\n\nBoard power, peak\n\n307.78 W\n\nMemory in use, median\n\n3,590 MiB\n\nGPU temperature, median\n\n49.0 °C\n\nGPU load, median\n\n82.0%\n\nTelemetry samples while loaded\n\n9\n\nqwen2.5:7bQ4_K_M · named “Qwen 2.5 7B” in the file · run logged UTC\n\n1,886 J per 1,000 tokens Computed: 1.886 J per token × 1,000\n\nDecode, median of 3\n\n182.2 tok/s\n\nBoard power, median while loaded\n\n343.6 W\n\nTokens per joule, as filed\n\n0.53\n\nJoules per token, as filed\n\n1.886\n\nMore from the power file\n\nWatt-hours per 1,000 tokens\n\n0.524 Wh Computed: 524.0 Wh per million ÷ 1,000\n\nDecode, slowest and fastest run\n\n182.1 to 182.3 tok/s\n\nBoard power, peak\n\n346.08 W\n\nMemory in use, median\n\n5,665 MiB\n\nGPU temperature, median\n\n52.0 °C\n\nGPU load, median\n\n88.0%\n\nTelemetry samples while loaded\n\n14\n\nqwen2.5:14bQ4_K_M · named “Qwen 2.5 14B” in the file · run logged UTC\n\n3,154 J per 1,000 tokens Computed: 3.154 J per token × 1,000\n\nDecode, median of 3\n\n98.8 tok/s\n\nBoard power, median while loaded\n\n311.6 W\n\nTokens per joule, as filed\n\n0.317\n\nJoules per token, as filed\n\n3.154\n\nMore from the power file\n\nWatt-hours per 1,000 tokens\n\n0.876 Wh Computed: 876.0 Wh per million ÷ 1,000\n\nDecode, slowest and fastest run\n\n98.6 to 98.8 tok/s\n\nBoard power, peak\n\n319.69 W\n\nMemory in use, median\n\n10,171 MiB\n\nGPU temperature, median\n\n50.0 °C\n\nGPU load, median\n\n93.0%\n\nTelemetry samples while loaded\n\n27\n\ngpt-oss:20bMXFP4 · named “gpt-oss 20B” in the file · run logged UTC\n\n1,504 J per 1,000 tokens Computed: 1.504 J per token × 1,000\n\nDecode, median of 3\n\n194.4 tok/s\n\nBoard power, median while loaded\n\n292.4 W\n\nTokens per joule, as filed\n\n0.665\n\nJoules per token, as filed\n\n1.504\n\nMore from the power file\n\nWatt-hours per 1,000 tokens\n\n0.418 Wh Computed: 418.0 Wh per million ÷ 1,000\n\nDecode, slowest and fastest run\n\n194.3 to 194.5 tok/s\n\nBoard power, peak\n\n294.86 W\n\nMemory in use, median\n\n13,381 MiB\n\nGPU temperature, median\n\n51.0 °C\n\nGPU load, median\n\n84.0%\n\nTelemetry samples while loaded\n\n18\n\nThese four runs are a separate capture from the map: one short prompt, three iterations per model, board power sampled once a second and counted only while the GPU was at least half loaded. They were taken at the card’s stock 448 W power limit, which none of the runs reached. Board power is the graphics card alone, not the wall.\n\nThe power file records no context length, driver or runtime version, which is why these runs are not pinned on the map. Its dataset page gives the runtime as Ollama 0.32.1.\n\nThere is no undervolt or power-limit comparison here because I have not measured one for language models. My undervolt database holds one row for this card and it is a stock 3DMark baseline. Method and the idle-power finding are on the power efficiency dataset.\n\nVersions on the bench, and how stale they are\n\nDriver and runtime versions per capture, exactly as each file records them. Where a file is silent, the cell says so.\n\nCapturedFileGPU driverRuntime versionsRows used here\n\nllm-server-compare-2026-08-26-lmstudio.csvNot recorded in this fileLM Studio 0.4.21, Ollama 0.32.15Ollama 0.32.15: vendored llama.cpp 9d77fa172LM Studio 0.4.21: llama.cpp-win-x86_64-nvidia-cuda12-avx2@2.29.112 configurations\n\nThose are the versions that were installed on the capture dates. Drivers and runtimes have both moved since, and I have not re-run this matrix on newer ones. Read every pin as true for its date and versions, and expect a current install to differ.\n\nThe second table is why I say that plainly. Two captures six days apart put the same three models on two Ollama versions, and all three decoded faster on the later one. The runtime comparison dataset notes that the earlier capture had less control over background load, so not all of that gap is the version. Either way, a few percent between runtimes does not survive an update.\n\nWhat the data covers\n\nModel tags (8)\n\nllama3.2:3b 3.2B · Q4_K_M 9 configurations\n\nqwen2.5-coder:7b 7.6B · Q4_K_M 2 configurations\n\nqwen2.5:7b 7.6B · Q4_K_M 2 configurations\n\nqwen2.5-coder:14b 14.8B · Q4_K_M 2 configurations\n\nqwen2.5:14b 14.8B · Q4_K_M 8 configurations\n\ngpt-oss:20b 20.9B · MXFP4 10 configurations\n\ndevstral:24b 23.6B · Q4_K_M 1 configuration\n\nqwen3-coder:30b 30.5B · Q4_K_M 2 configurations\n\nQuantizations (2)\n\nMXFP410\n\nQ4_K_M26\n\nNo model was measured at more than one quantization.\n\nOne quantization per model. I cannot show the same model at three quantizations stepping down the card, because each model was measured at one.\n\nNo dense model past the cliff. The runs that left the card are a 3B model at an extreme context, a mixture-of-experts model that never fit, and one LM Studio default that offloaded part of a model that otherwise fits. A dense model that fits at one context and spills at the next is the row this map most needs.\n\nA gap between 16,384 and 131,072 context. One run sits at the far end. Nothing was measured at 32K or 64K.\n\nEnergy at stock only. No undervolt, no reduced power limit, and no energy figure for a run that spilled.\n\nOne card. Another 16 GB card, or this one with a different desktop load, will read differently.\n\nMethod, exactly\n\nI did not run anything new for this page. It reads files I had already published and puts them on one axis.\n\nThe throughput rows come from my capture script. For each model, context and prompt size it calls Ollama’s /api/generate three times at temperature 0, seed 42 and a 256-token output, with a unique prefix on every prompt so each run pays a full prefill. Decode and prefill rates are Ollama’s own eval_count over eval_duration and prompt_eval_count over prompt_eval_duration, and the CSV keeps the median of the three. After each cell the script asks nvidia-smi for memory.used and stores it as vram_resident_mib. That one reading is the pin’s position.\n\nThat reading is the whole card. It includes the desktop, and it includes any model Ollama had not yet unloaded, which is how one Llama 3.2 3B row came to read 15,682 MiB. The runtime comparison captures used a different script that took the median memory reading across runs and, in the later capture, also read the card before each load.\n\nEach throughput configuration was measured with two prompt sizes. The pin is the first of the two, the shorter prompt, and both are in the record. The model names differ between files, so I join them on the name with punctuation and a trailing quantization tag removed. Only the throughput CSV has a quantization column; where another file’s row shows a quantization, the record says it was taken from the same model there.\n\nThe card total, 16,303 MiB, is the nvidia-smi total written into the raw capture file from 2026-08-26. Headroom, share of the card, GiB, the memory a load added, the changes between two rows and the per-thousand-token energy are computed when the site is built, from the measured values beside them, and each one is shown with its formula. Your browser does no arithmetic on them.\n\nDownload the files\n\nrtx-5080-llm-throughput.csv34 rows. Decode, prefill, load time and whole-card memory for each model, context and prompt size on Ollama. Raw capture JSON and the capture script are listed in its manifest.\n\nllm-server-compare-2026-08.csv6 rows. Ollama beside llama.cpp on three models, with its settings and capture date in the footer notes.\n\nllm-server-compare-2026-08-26-lmstudio.csv12 rows. Ollama beside LM Studio on three models, at default and normalized settings, with the memory reading before each load.\n\nAll of it is CC BY 4.0. Attribute as “TechFuelHQ RTX 5080 16 GB VRAM cliff map” with a link to this page. To cite one configuration, open its row in the list and copy its link; the address ends in that row’s own id.", "url": "https://wpnews.pro/news/the-16-gb-cliff-measured-llm-memory-and-speed-on-an-rtx-5080", "canonical_source": "https://techfuelhq.com/data/rtx-5080-16gb-vram-cliff/", "published_at": "2026-11-17 17:00:00+00:00", "updated_at": "2026-11-17 21:13:02.136091+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure"], "entities": ["Nvidia", "RTX 5080", "llama3.2:3b", "Ollama", "llama.cpp"], "also_reported_by": [], "takeaway": "On a 16 GB RTX 5080, llama3.2:3b Q4_K_M used 3,760 MiB of 16,303 MiB and decoded at 351.22 tok/s at 4,096 context but fell to 102.68 tok/s at 131,072 context.", "recommendation": {"advice": "Budget VRAM for context growth: llama3.2:3b Q4_K_M held 351.22 tok/s at 4,096 context but dropped to 102.68 tok/s at 131,072 on a 16 GB RTX 5080.", "playbook": "inference-serving", "playbook_url": "https://wpnews.pro/playbooks/inference-serving"}, "alternates": {"html": "https://wpnews.pro/news/the-16-gb-cliff-measured-llm-memory-and-speed-on-an-rtx-5080", "markdown": "https://wpnews.pro/news/the-16-gb-cliff-measured-llm-memory-and-speed-on-an-rtx-5080.md", "text": "https://wpnews.pro/news/the-16-gb-cliff-measured-llm-memory-and-speed-on-an-rtx-5080.txt", "jsonld": "https://wpnews.pro/news/the-16-gb-cliff-measured-llm-memory-and-speed-on-an-rtx-5080.jsonld"}}