{"slug": "a-straight-line-predicts-how-fast-my-laptop-generates-llm-tokens", "title": "A Straight Line Predicts How Fast My Laptop Generates LLM Tokens", "summary": "A blogger measured time per output token (TPOT) for six Qwen2.5, Qwen3 and Gemma 3 models on a MacBook Pro with an M2 Pro chip and 16 GB of RAM, finding a linear relationship between model size and TPOT that implies an effective memory bandwidth of 163 GB/s and a fixed per-run cost of 4.03 ms, against the laptop's 200 GB/s peak bandwidth. The fit predicted TPOT for Qwen3 1.7B in float32 (6.88 GB) within 0.2% (46.2 ms predicted vs. 46.1 ms measured), with all points within 1.6 ms of the line; the largest deviations were Qwen3 0.6B in bfloat16 (12.3% error) and Gemma 3 270M in float32 (15.6% error). The result matters because decode-phase LLM inference is bounded by memory bandwidth, so model size divided by effective bandwidth sets the floor on token generation speed.", "body_md": "[My previous post](https://bryananthonio.com/blog/one-reason-input-tokens-are-cheaper-than-output-tokens/) ended on a cliffhanger. For the model Qwen2.5-0.5B, I found that the time per output token (TPOT) was roughly 9 ms for different prompt sizes. But I didn’t have an explanation for why.\n\nI’ve since learned that in the decode phase for large language model (LLM) inference the primary bottleneck is memory bandwidth, the rate at which the GPU reads LLM model weights from RAM. To explore this, I measured the TPOT of six models of varying sizes on my laptop.\n\n## Contents[#](#contents)\n\n## The Memory Bandwidth Bottleneck[#](#the-memory-bandwidth-bottleneck)\n\nIn LLM inference, the decode phase produces one token at a time. Each time, the weights have to be streamed from RAM to the GPU chip, where the next token is computed.\n\nIn other words, to generate tokens, you’d have to re-read the model weights times. This is because the GPU chip has a limited amount of memory, on the order of kilobytes or megabytes depending on the hardware, so it can’t hold all of the weights at once.\n\nMemory bandwidth dictates the speed limit of this process:\n\nIn other words, **the lowest possible TPOT is given by the model size divided by the memory bandwidth of our system.**\n\nFor example, the weights for Qwen2.5-0.5B take ~988 MB of storage. As a result, on my MacBook Pro with the M2 Pro chip (16 GB) and a peak memory bandwidth of 200 GB/s, I can expect the lowest possible TPOT for this model to be 4.94 ms.\n\nWe can characterize the relationship between model size and TPOT using a linear model\n\nin which model size will be our independent variable and TPOT will be the dependent variable.\n\nThe slope of this line captures the effective memory bandwidth. Although the peak bandwidth of my laptop is 200 GB/s, we can’t expect to use all of it. This is partly because memory bandwidth is shared by the CPU, GPU, and laptop display. Thus, LLM inference gets whatever bandwidth is left over, which we can define as the effective bandwidth.\n\nIn addition, the vertical offset of this line measures system overhead, which is treated as a fixed cost regardless of model size. I’d guess this fixed cost might include things like the time it takes for the GPU to start doing work, among other things.\n\n## TPOT vs Model Size[#](#tpot-vs-model-size)\n\nUsing models in the Qwen2.5, Qwen3, and Gemma 3 families, I measured TPOT vs model size on my MacBook Pro. I ran each of these in both `bfloat16` and `float32` precisions to have more data points in the plot.\n\nThe dashed line here represents the best-case scenario where there’s no fixed cost and the system uses the 200 GB/s peak bandwidth.\n\nI was able to measure an effective bandwidth of 163 GB/s and a fixed cost of 4.03 ms. I redid this linear fit without the result from the largest LLM, Qwen3 1.7B in float32 (6.88 GB), to check how well it captures the trend. In this case, the line predicted the TPOT for it within 0.2% (46.2 ms vs. 46.1 ms measured).\n\nIn general, all points are within 1.6 ms of the line. The models that deviated the most from the trend were Qwen3 0.6B (bfloat16) and Gemma 3 270M (float32), which exhibited errors of 12.3% and 15.6%, respectively. The model exhibiting the most agreement was Qwen3 1.7B (float32), which had an error of 0.1%.\n\nAs you can see in the table below, the general trend is that as the model size increases, the TPOT also increases as you’d expect.\n\n| Model | Weights (GB) | Measured TPOT (ms) | Predicted TPOT (ms) | Fraction of time spent reading weights | \n|---|---|---|---|---|\n| Gemma 3 270M, bfloat16 | 0.54 | 7.09 | 7.31 | 45% | \n| Qwen2.5 0.5B, bfloat16 | 0.99 | 8.99 | 10.08 | 60% | \n| Gemma 3 270M, float32 | 1.07 | 9.17 | 10.59 | 62% | \n| Qwen3 0.6B, bfloat16 | 1.19 | 12.92 | 11.33 | 64% | \n| Qwen2.5 0.5B, float32 | 1.98 | 14.89 | 16.12 | 75% | \n| Gemma 3 1B, bfloat16 | 2.00 | 17.61 | 16.27 | 75% | \n| Qwen3 0.6B, float32 | 2.38 | 19.81 | 18.62 | 78% | \n| Qwen2.5 1.5B, bfloat16 | 3.09 | 22.12 | 22.92 | 82% | \n| Qwen3 1.7B, bfloat16 | 3.44 | 26.20 | 25.09 | 84% | \n| Gemma 3 1B, float32 | 4.00 | 28.81 | 28.51 | 86% | \n| Qwen2.5 1.5B, float32 | 6.17 | 41.07 | 41.81 | 90% | \n| Qwen3 1.7B, float32 | 6.88 | 46.10 | 46.14 | 91% | \n\nFor each model, I calculated the fraction of time spent using two steps:\n\n1. Estimate the time spent reading weights by taking the model size and dividing that by the effective bandwidth (163 GB/s).\n2. Divide this read time by the predicted TPOT.\n\nWe can see that system overhead takes up a larger fraction of the time needed to generate a new token for smaller models. For the smallest model my laptop spends roughly 45% of time reading weights, whereas for the largest model it spends 91%.\n\n## How I Measured This[#](#how-i-measured-this)\n\nFor each model, I measured TPOT using a similar [`ttft_tpot` function from my previous post](https://bryananthonio.com/blog/one-reason-input-tokens-are-cheaper-than-output-tokens/#measuring-the-prefill-and-decoding-phases) with the [MPS backend in PyTorch](https://docs.pytorch.org/docs/stable/notes/mps.html), which allows PyTorch to run code on my MacBook’s integrated GPU.\n\n``` python\nimport statistics\nimport time\nimport torch\n\ndef timed_generate(model, ids, n):\n    torch.mps.synchronize()  # wait for any earlier GPU work to finish\n    t0 = time.perf_counter()\n    model.generate(ids, min_new_tokens=n, max_new_tokens=n, do_sample=False)\n    torch.mps.synchronize()  # wait for this call's GPU work to finish\n    return time.perf_counter() - t0\n\ndef ttft_tpot(model, ids, num_samples=5):\n    timed_generate(model, ids, 1)  # warm-up\n    ttft = statistics.median(timed_generate(model, ids, 1) for _ in range(num_samples))\n    total = statistics.median(timed_generate(model, ids, 101) for _ in range(num_samples))\n    return ttft, (total - ttft) / 100\n```\n\nThen I looped over the models:\n\n``` python\nimport gc\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\nmodels = [\n    (\"google/gemma-3-270m-it\", torch.bfloat16),\n    (\"google/gemma-3-270m-it\", torch.float32),\n    (\"Qwen/Qwen2.5-0.5B-Instruct\", torch.bfloat16),\n    (\"Qwen/Qwen2.5-0.5B-Instruct\", torch.float32),\n    (\"google/gemma-3-1b-it\", torch.bfloat16),\n    (\"google/gemma-3-1b-it\", torch.float32),\n    (\"Qwen/Qwen2.5-1.5B-Instruct\", torch.bfloat16),\n    (\"Qwen/Qwen2.5-1.5B-Instruct\", torch.float32),\n    (\"Qwen/Qwen3-0.6B\", torch.bfloat16),\n    (\"Qwen/Qwen3-0.6B\", torch.float32),\n    (\"Qwen/Qwen3-1.7B\", torch.bfloat16),\n    (\"Qwen/Qwen3-1.7B\", torch.float32),\n]\n\ntext = open(\"input.txt\").read()\nrows = []  # (size in GB, TPOT in seconds)\n\nfor name, dtype in models:\n    model = AutoModelForCausalLM.from_pretrained(name, dtype=dtype).to(\"mps\").eval()\n    tok = AutoTokenizer.from_pretrained(name)\n    input_ids = tok(text[:20_000], return_tensors=\"pt\").input_ids.to(\"mps\")\n\n    size_gb = sum(p.numel() * p.element_size() for p in model.parameters()) / 1e9\n    _, tpot = ttft_tpot(model, input_ids[:, :128])\n    rows.append((size_gb, tpot))\n\n    del model\n    gc.collect()\n    torch.mps.empty_cache()\n```\n\nFor each model, I used the first 128 tokens of the [Tiny Shakespeare text](https://github.com/karpathy/char-rnn/blob/master/data/tinyshakespeare/input.txt) as the input prompt.\n\nAfter gathering the data, doing the linear fit is as simple as:\n\n``` python\nimport numpy as np\n\ngb = [r[0] for r in rows]\nms = [r[1] * 1000 for r in rows]  # seconds -> ms\nslope, intercept = np.polyfit(gb, ms, 1)  # ms per GB, ms\n```\n\n## Caveats of This Measurement[#](#caveats-of-this-measurement)\n\nThis was just a quick experiment to get a feel for the numbers, and not a rigorous benchmark. With that being said, there are a few flaws to my approach I should call out.\n\nOne peculiar aspect of the data is that a linear fit on just the Gemma 3 270M data points (bfloat16 and float32) implies a memory bandwidth of about 258 GB/s, which is far above the spec limit of 200 GB/s. I was able to reproduce similar results across multiple runs, which suggests to me that this isn’t due to noise.\n\nI believe the simple conclusion is that fixed cost isn’t always the same for each model. This is true even when considering the same model in either bfloat16 or float32. I don’t yet fully know what the underlying cause is but I suspect it may have something to do with the model architecture.\n\nAnother caveat is that I didn’t run this using an optimized inference server, such as [vLLM](https://vllm.ai) or [llama.cpp](https://github.com/ggml-org/llama.cpp). I’d expect a more optimized setup would lower the fixed cost measured here but might not improve the bandwidth utilization as much given that there’s probably not much headroom to do better. It might be interesting to try this experiment again with [`mlx-lm`, which offers optimized kernels for Apple silicon](https://github.com/ml-explore/mlx-lm), instead of PyTorch.\n\n## What I Took Away From This[#](#what-i-took-away-from-this)\n\nNow, let’s get back to my original question: for Qwen2.5-0.5B why is TPOT ~9 ms? The linear fit predicts that for this model my laptop takes roughly 6.05 ms to read the weights plus a fixed cost of 4.03 ms, giving us a total of 10.08 ms. This is 12% higher than the 8.99 ms figure I measured but it does paint a picture of how time is spent generating each token.\n\nI’ll admit I was surprised to see how well the data fit the expected linear trend. But now that I understand the mechanism behind this it all makes sense.", "url": "https://wpnews.pro/news/a-straight-line-predicts-how-fast-my-laptop-generates-llm-tokens", "canonical_source": "https://bryananthonio.com/blog/straight-line-predicts-how-fast-my-laptop-generates-llm-tokens/", "published_at": "2026-11-15 15:03:31+00:00", "updated_at": "2026-11-15 15:23:50.153322+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "machine-learning"], "entities": ["Qwen2.5", "Qwen3", "Gemma 3", "MacBook Pro", "M2 Pro", "Qwen2.5-0.5B", "Qwen3 1.7B", "Gemma 3 270M"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/a-straight-line-predicts-how-fast-my-laptop-generates-llm-tokens", "markdown": "https://wpnews.pro/news/a-straight-line-predicts-how-fast-my-laptop-generates-llm-tokens.md", "text": "https://wpnews.pro/news/a-straight-line-predicts-how-fast-my-laptop-generates-llm-tokens.txt", "jsonld": "https://wpnews.pro/news/a-straight-line-predicts-how-fast-my-laptop-generates-llm-tokens.jsonld"}}