{"slug": "42x-faster-prompt-lookup-drafting-in-llama-cpp", "title": "42x faster prompt lookup drafting in llama.cpp", "summary": "A developer writing as jadidbourbaki reports making prompt lookup decoding drafting in llama.cpp up to 42x faster while using up to 2.6x less memory, through performance optimizations based on work by Daniel Lemire and Martin Ankerl. The post details llama.cpp's three n-gram caches — context, dynamic, and static — and the scoring formula that drafts the next k tokens by selecting the highest-scoring token y* subject to thresholds a_n and p_n. Prompt lookup decoding, also called n-gram speculation, is supported by inference engines including llama.cpp and vLLM and by Hugging Face's transformers library.", "body_md": "[[homepage](https://jadidbourbaki.github.io/)]\n        [[github](https://github.com/jadidbourbaki)]\n        [[twitter](https://x.com/jadidbourbaki)]\n        \n\n<sub>This article was originally published on 2026-09-26.</sub>\n\n**TL;DR** I make drafting for prompt lookup decoding in llama.cpp up to 42x faster while using up to 2.6x less memory through\na set of simple performance optimizations largely based on the work of \n[Daniel Lemire](https://github.com/lemire) and [Martin Ankerl](https://github.com/martinus).\n\n    Many popular inference engines including llama.cpp and vllm, and machine learning libraries\n    such as hugging face's transformers library, support \n    [prompt lookup decoding](https://github.com/apoorvumang/prompt-lookup-decoding)\n    (also called [n-gram speculation](https://x.com/joao_gante/status/1747322413006643259))\n    for faster token generation. Prompt lookup decoding is technically a special case of speculative decoding \n    that uses a really stupid draft model, an n-gram model. When prompt lookup decoding is used, \n    the inference engine drafts the next $k$ tokens using the following rule.\n\nLet $x_1, \\ldots, x_t$ be the current tokens of a model. An n-gram is a sequence of $n$ consecutive tokens. For example, a 3-gram would be $(x_1, x_2, x_3)$ or $(x_2, x_3, x_4)$ or, in general, $(x_i, x_{i+1}, x_{i+2})$ for any $i \\in \\{1, \\ldots, t-2\\}$. Now an n-gram model is a probabilistic model that predicts the next token based on the previous $n - 1$ tokens. The idea is extremely simple. You first select some corpus of text and parse it into n-grams. You then count the frequency of each n-gram. When your n-gram model needs to predict the next token after a sequence of $n-1$ tokens, you make the n-gram model select the token that most frequently follows that sequence of $n-1$ tokens in your corpus.\n\nllama.cpp maintains three types of *n-gram caches*. Let $\\eta$ be any n-gram and $y$ \n    be any token. An n-gram cache \n    is a data structure that stores $c(\\eta, y)$, i.e., the count of how many times the token $y$ follows the n-gram $\\eta$, \n    for all n-grams $\\eta$ and all tokens $y$ in a given corpus and vocabulary. The three n-gram caches used \n    by llama.cpp are the context cache, the dynamic cache, and the static cache.\n\n    llama.cpp's context cache stores n-grams of sizes 1 to 4 for the current tokens $x_1, \\ldots, x_t$ being \n    processed by the model. The context cache is updated as the model generates new tokens. The\n    dynamic cache stores the counts of n-grams from previous runs of the model, e.g., any earlier\n    conversations. Finally, the static cache stores n-grams of size 2 from a static text corpus, built\n    with `llama-lookup-create`.\n    I denote the context, dynamic, and static caches by $c_{\\text{ctx}}$, $c_{\\text{dyn}}$, and $c_{\\text{st}}$, respectively.\n\nllama.cpp drafts a new token using its n-gram caches in the following way. Let $X_n = (x_{t-n+1}, \\ldots, x_t)$ be the previous $n$ tokens processed by the model. For all tokens $y$ in the vocabulary, llama.cpp computes a score using the formula\n\n$$s_n^{f}(y) = f(X_n, y) \\cdot w(y) \\,\\, \\text{where} \\,\\, w(y) = \\begin{cases} 100 \\, c_{\\text{st}}(X_2, y) & \\text{if $c_{\\text{st}}(X_2, y) > 0$} \\\\ 1 & \\text{otherwise} \\end{cases}$$\nwhere $f$ is either the context cache $c_{\\text{ctx}}$ or the dynamic cache $c_{\\text{dyn}}$. Note that the weight $w(y)$ favors tokens that also agree with the static cache. Without a static cache, $w(y) = 1$ for every token. For each $n$, llama.cpp takes the highest-scoring token $y^* = \\arg\\max_y s_n^{f}(y)$. Let $F(X_n) = \\sum_y f(X_n, y)$ be the number of times $X_n$ appeared with a token after it. llama.cpp drafts $y^*$ based on two configurable thresholds $a_n$ and $p_n$ in the following way.\n\n$$F(X_n) \\ge a_n \\,\\, \\text{and} \\,\\, f(X_n, y^*) \\ge p_n \\, F(X_n)$$\nIn other words, $X_n$ must appear at least $a_n$ times and the token $y^*$ must have followed $X_n$ in at least a fraction $p_n$ of those occurrences for $y^*$ to be accepted as a draft token. As of release b11182 of llama.cpp, the thresholds are hard-coded as follows. For the context cache, $(a_1, a_2, a_3, a_4) = (2, 2, 1, 1)$ and $(p_1, p_2, p_3, p_4) = (0.66, 0.5, 0.5, 0.5)$. For the dynamic cache, $(a_1, a_2, a_3, a_4) = (4, 3, 2, 2)$ and $(p_1, p_2, p_3, p_4) = (0.75, 0.66, 0.66, 0.66)$. llama.cpp tries $n = 4, 3, 2, 1$ and drafts the first $y^*$ that passes the conditions above. It first scores with $c_{\\text{ctx}}$. It scores with $c_{\\text{dyn}}$ only when no candidate from $c_{\\text{ctx}}$ passes for any $n$. If no candidate from $c_{\\text{dyn}}$ passes either, llama.cpp falls back to relying only on the static cache (as opposed to only using it to reweight candidates in the other caches). As a side note, the static cache's thresholds in llama.cpp are the same values as the context cache's thresholds for the corresponding $n$, i.e., $n = 2$. Let $C_{\\text{st}}(X_2) = \\sum_y c_{\\text{st}}(X_2, y)$. llama.cpp takes the token $y$ with the largest $c_{\\text{st}}(X_2, y)$ and drafts it when $C_{\\text{st}}(X_2) \\ge a_2 = 2$ and $c_{\\text{st}}(X_2, y) \\ge p_2 \\, C_{\\text{st}}(X_2) = 0.5 \\, C_{\\text{st}}(X_2)$. If the static cache also fails, llama.cpp does not draft the next token.\n\n    llama.cpp's repository includes an example for prompt lookup decoding\n    [here](https://github.com/ggml-org/llama.cpp/tree/master/examples/lookup).\n    It includes two tools I use: `llama-lookup-create` for building a static cache from a corpus\n    and `llama-lookup-stats` for benchmarking prompt lookup decoding.\n    `llama-lookup-stats` essentially reads a file and treats the file's tokens\n    as the output of a model. It runs the drafting loop from llama.cpp over the simulated\n    \"model output\" (i.e. the file) and records how many drafted tokens match the file, the time it took\n    to draft the tokens, and the time it took to load the static ngram cache.\n\n    I build the static caches using `llama-lookup-create` with\n    [WikiText-103](https://arxiv.org/abs/1609.07843) and then I\n    replay the WikiText-103 test text through `llama-lookup-stats`. I borrowed\n    this evaluation method from the\n    [PR](https://github.com/ggml-org/llama.cpp/pull/5479)\n    by [@JohannesGaessler](https://github.com/JohannesGaessler) that added\n    the static n-gram cache to llama.cpp. Note that since I am not making any\n    algorithmic modifications to how prompt lookup decoding works in llama.cpp, the dataset\n    mainly matters for the acceptance rate, which my changes leave unchanged. Just to be safe, I make\n    sure my changes still have almost identical acceptance rates to the original implementation.\n    The important metrics here that actually change are 1) latency\n    per drafted token, 2) the load time of the static cache, and 3) the memory used by the static cache.\n\n    I also wanted to observe how the performance changes with different corpus sizes for the static \n    n-gram cache. So in addition to evaluating the full corpus of WikiText-103, which is about \n    541 MB, I also build static caches from the first 25, 50, 100, and 200 MB of the WikiText-103 training text.\n    A corpus size of 0 in the figures means I run without a static cache, which measures the context and\n    dynamic caches alone.\n    For all the results in this work, I am reporting the median of 3 runs with the error bars displaying the \n    min and the max value for the runs. Following the llama.cpp PR I linked \n    in the previous paragraph, I also benchmark assuming a model context side of 4096 tokens.\n    I run all my experiments on an Apple M4 Pro with 14 cores and 48 GB of memory.\n    All of my code and results are in this\n    [repository](https://github.com/jadidbourbaki/ngram-cache-bench).\n\nThe n-gram caches in llama.cpp are currently implemented as nested `std::unordered_map` s. An outer\nmap sends each n-gram to an inner map of the tokens that follow it and their counts.\nThis one is almost more of a bug fix than an optimization. I found that the inner maps were\nbeing copied unnecessarily in multiple places on every drafting step.\nI created this simple [PR](https://github.com/jadidbourbaki/llama.cpp/pull/2)\nto read them by reference instead.\nThis immediately made drafting 4.5x to 25.6x faster depending on the size of the\ncorpus (see figure below). The latency is the average time spent\ndrafting per drafted token.\n\nllama.cpp implements an n-gram cache as a map of maps.\n\n```\ntypedef std::unordered_map<common_ngram, common_ngram_cache_part,\n        common_ngram_hash_function> common_ngram_cache;\n```\n\nThe outer map,`common_ngram_cache`, maps each n-gram to an inner map. The inner \nmap, a `common_ngram_cache_part`, map stores the counts of each token in the vocabulary\nthat follows the given n-gram. As an example, if\n\"of the\" is followed by \"city\" 6 times, \"war\" 3 times, and \"year\" once, the n-gram cache looks like this.\n\n``` php\ncommon_ngram_cache\n  (\"of\", \"the\")  ->  common_ngram_cache_part { \"city\": 6, \"war\": 3, \"year\": 1 }\n  (\"in\", \"the\")  ->  common_ngram_cache_part { ... }\n  ....\n```\n\nllama.cpp currently implements both the outer and inner maps as an `std::unordered_map`.\n   However, the standard library's implementation of `std::unordered_map` is \n   [famously slow](https://stackoverflow.com/a/42588384) \n   because it uses chaining for collision resolution with linked lists as its buckets \n   which is cache unfriendly. There are many great alternatives here such as Google's \n   [Swiss Tables](https://abseil.io/about/design/swisstables) (which \n   were also [recently added](https://go.dev/blog/swisstable) to Golang)\n   and [Martin Ankerl's](https://github.com/martinus) [unordered_dense maps](https://github.com/martinus/unordered_dense).\n   I decided to go with `ankerl::unordered_dense` because 1) I really like its design and \n   performance, and 2) it is less of an annoyance than trying \n   to add all of abseil as a dependency to llama.cpp.\n\nMy change is in this [PR](https://github.com/jadidbourbaki/llama.cpp/pull/5).\nThis makes 1) loading the static n-gram cache 1.41x to 1.65x faster depending on the size of the corpus, \n2) drafting a new token 1.02x to 1.13x faster, and 3) the static cache use 1.07x to 1.11x\nless memory. See the figures below. Note that I use the `segmented_map` variant \nof `ankerl::unordered_dense` instead of the default `map` variant.\nThe default `map` variant keeps all entries in one vector that doubles as it fills. \nWhen I experimented with the `map` variant on the full 541 MB corpus, \nthe final doubling of the vectors caused the static cache to use 1.16x more memory than the baseline.\nNote that the baseline here is my previous PR where I removed\nthe unnecessary map copying. The `segmented_map` variant avoids this issue by growing the map in \nsegments of 4096 bytes allowing for lower peak memory usage.\n\n In the previous section, I only replaced the outer map with an `ankerl::unordered_dense::segmented_map`. \n The inner map is still a sad old `std::unordered_map`. Notice that most \n n-grams have very few followers which makes having a `std::unordered_map` \n or any kind of hash map for each inner map quite wasteful in terms of memory. Doing \n some street fighting math on the static cache created from WikiText-103, you can \n get the following CDF. \n\n    The important observation is that since 64% of the 2-grams used for drafting the static n-gram cache \n    have only one follower, simply using an `std::vector` is much more memory efficient than \n    maintaining hash maps for the inner map. The distribution is pretty heavy tailed though. A few frequent \n    2-grams are following by thousands of distinct tokens from the vocabulary which is why the figure \n    goes all the way to $10^4$ before the tail tapers off to $\\to 1$. So simply using a regular \n    `std::vector` would blow up the search latency for the n-grams at the tail. So \n    I used a sorted `std::vector` instead to keep the search still $\\mathcal{O}(\\log n)$ for the n-grams at the tail.\n\nMy first version simply used `std::lower_bound` with the sorted `std::vector`.\nHowever, this approach slightly reduced the drafting speed making it only 0.89x as fast as \nthe baseline (the baseline being the previous optimization where I replaced the outer map with an \n`ankerl::unordered_dense::segmented_map`). Most of the extra time goes into searching\nthe followers of frequent 2-grams, which can have thousands of entries. The loop of\n`std::lower_bound` in libc++, simplified to our vector of (token, count) pairs, looks like this.\n\n``` js\nconst value_type * first = pairs;\nsize_t len = n;\nwhile (len != 0) {                 // the loop ends when len reaches 0\n    const size_t half = len / 2;\n    const value_type * mid = first + half;\n    if (mid->first < token) {      // compares against an entry read from memory\n        first = mid + 1;\n        len -= half + 1;           // the new len depends on that entry\n    } else {\n        len = half;\n    }\n}\nreturn first - pairs;\n```\n\nThe remaining length `len` depends on the result of each comparison, and so does the number\nof iterations. A search over 8 entries takes 3 or 4 iterations depending on the token. The CPU cannot\nevaluate `len != 0` until the entry of the current iteration arrives from memory, which is often a\ncache miss for a vector with thousands of entries. My version separates the length from the comparison.\n\n``` js\nconst value_type * base = pairs;\nwhile (n > 1) {                    // the loop ends when n reaches 1\n    const size_t half = n / 2;\n    base = base[half].first < token ? base + half : base;   // only base depends on the entry\n    n -= half;                     // n does not depend on any entry\n}\nreturn (base - pairs) + (base->first < token);\n```\n\nHere `n` shrinks by the same amount whatever the comparison returns. A search over 8 entries\nalways takes 3 iterations, with `n` going from 8 to 4 to 2 to 1. The CPU can evaluate\n`n > 1` without waiting for any entry, so it can move on to the search for the next candidate token\nwhile the reads of the current search are still in flight.\n\nMy change is in this [PR](https://github.com/jadidbourbaki/llama.cpp/pull/10). Compared to the flat hash map,\nit makes drafting 2.09x faster without a static cache and 1.19x to 1.25x faster when a static cache is used. The PR \nalso reduces by peak memory by as much as 1.97x. Loading the static cache takes about the same time.\n\nDaniel Lemire recently published an optimized implementation of an immutable map from strings to 64-bit integers\ncalled [constmap](https://github.com/lemire/fastconstmap) which is built on top of\n[binary fuse filters](https://arxiv.org/abs/2201.01174).\n\nSince llama.cpp never changes the static cache after loading it, this is a perfect use case for a constmap. I replaced the outer map of the static cache with a verified constmap. My implementation packs the 2-grams in the static n-gram cache into a contiguous array of (token, count) pairs. Using my earlier example (if \"of the\" is followed by \"city\" 6 times, \"war\" 3 times, and \"year\" once), the constmap stores the following.\n\n```\npairs\n  ...\n  [1000]  (\"city\", 6)\n  [1001]  (\"war\", 3)\n  [1002]  (\"year\", 1)\n  ...\n\nconstmap\n  (\"of\", \"the\")  ->  (1000, 3)\n  (\"in\", \"the\")  ->  ...\n```\n\nThe followers of (\"of\", \"the\") start at position 1000 of the array. There are 3 of them. The constmap\nstores the position 1000 and the count 3 together as one 64-bit value. The position takes the high 40 bits\nof the value and the count takes the low 24 bits. The static cache file stores a small header, the array of pairs, and the\nserialized constmap one after another. Loading reads the whole file into one buffer and opens the constmap inside that buffer with\n`fcm_verified_constmap_view`. A lookup queries the constmap once and returns a pointer into the\narray and the number of pairs.\n\n``` js\ncommon_ngram_cache_static_part common_ngram_cache_static_find(\n        const common_ngram_cache_static & nc_static,\n        const common_ngram & ngram) {\n    const char * key = reinterpret_cast<const char *>(ngram.tokens);\n    const uint64_t value = fcm_verified_constmap_lookup(\n        nc_static.map.get(), key, STATIC_KEY_SIZE);\n    if (value == FCM_NOT_FOUND) {\n        return {};\n    }\n    const uint64_t position = value >> STATIC_LEN_BITS;\n    const size_t   count    = value & STATIC_LEN_MASK;\n    return { nc_static.entries + position, count };\n}\n```\n\nSince the pairs themselves, e.g. (\"city\", 6), (\"war\", 3), (\"year\", 1), have the same layout as the sorted vectors of the previous section, the implementation still uses fixed-length binary search to find the count of a candidate token.\n\nMy change is in this [PR](https://github.com/jadidbourbaki/llama.cpp/pull/7).\nCompared to the sorted vectors, it makes loading the static cache 6.32x to 16.12x faster, from 3.76 s to\n0.23 s with the 541 MB corpus. The static cache now takes about as much memory as its file, 463 MB for a\n467 MB file. Peak memory drops by up to 1.30x, from 1.71 GB to 1.31 GB with the 541 MB corpus. Drafting with\na static cache is 1.06x to 1.20x faster. The acceptance rate is identical to the sorted vectors on every corpus.\n\nIf you would like to cite this work, please use the following bibtex. Thank you.\n\n```\n@misc{tirmazi2026promptlookup,\n  author       = {Hayder Tirmazi},\n  title        = {42x Faster Prompt Lookup Drafting in {llama.cpp}},\n  year         = {2026},\n  month        = sep,\n  howpublished = {\\url{https://jadidbourbaki.github.io/blog/prompt-lookup-llama-cpp/}}\n}\n```\n\n", "url": "https://wpnews.pro/news/42x-faster-prompt-lookup-drafting-in-llama-cpp", "canonical_source": "https://jadidbourbaki.github.io/blog/prompt-lookup-llama-cpp/", "published_at": "2026-09-26 19:57:24+00:00", "updated_at": "2026-09-26 20:31:21.382561+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "mlops", "ai-tools"], "entities": ["llama.cpp", "vLLM", "Hugging Face", "transformers", "Daniel Lemire", "Martin Ankerl", "jadidbourbaki"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/42x-faster-prompt-lookup-drafting-in-llama-cpp", "markdown": "https://wpnews.pro/news/42x-faster-prompt-lookup-drafting-in-llama-cpp.md", "text": "https://wpnews.pro/news/42x-faster-prompt-lookup-drafting-in-llama-cpp.txt", "jsonld": "https://wpnews.pro/news/42x-faster-prompt-lookup-drafting-in-llama-cpp.jsonld"}}