{"slug": "i-got-9-9-lower-ttft-on-a-real-android-phone-by-reusing-llama-cpp-kv-state", "title": "I Got 9.9 Lower TTFT on a Real Android Phone by Reusing llama.cpp KV State", "summary": "A developer built EdgeSync-LLM, a mechanism that reuses KV cache state from llama.cpp to avoid recomputing shared prefixes during local LLM inference. On a real Android ARM64 phone, the approach achieved a 9.9× lower time-to-first-token (TTFT) on cache hits, dropping from 4828 ms to 486 ms. The project emphasizes correctness validation, as an earlier raw-tensor implementation produced a fake 8.8× speedup by silently dropping context.", "body_md": "Local LLM inference has an expensive habit:\n\nIt recomputes prefixes it has already seen.\n\nA system prompt.\n\nA reused RAG document.\n\nA few-shot block.\n\nA long static context.\n\nIf the prefix is identical, why pay the prefill cost again?\n\nThat's the problem I explored with EdgeSync-LLM.\n\nThe idea\n\nThe mechanism is simple:\n\nPrompt = shared prefix + new suffix\n\nOn the first request, EdgeSync prefills the prefix and captures its KV cache state.\n\nOn the next request sharing that exact prefix, it restores the state and decodes only the new suffix.\n\nNo llama.cpp fork.\n\nNo patch.\n\nThe current validated path uses the public:\n\nllama_state_seq_get_data\n\nand\n\nllama_state_seq_set_data\n\nAPIs.\n\nMeasured on a real Android ARM64 phone\n\nModel:\n\nQwen2.5-0.5B-Instruct Q4_K_M\n\nShared prefix:\n\n123 tokens\n\n40 requests. 4 threads. Release build.\n\nPath\n\nMean TTFT\n\np50\n\np95\n\nCold\n\n4828 ms\n\n4752 ms\n\n5297 ms\n\nKV state reuse\n\n486 ms\n\n476 ms\n\n569 ms\n\n9.9× lower TTFT on cache hits.\n\nThe warm path was approximately:\n\n363 ms to decode the 10-token suffix\n\n123 ms to restore the state blob\n\nFragment size: 1.64 MB\n\nI also measured the same mechanism on x86-64.\n\nCold mean TTFT: 1395 ms\n\nWarm mean TTFT: 185 ms\n\nThat's 7.5× on cache hits.\n\nBut I almost published a fake 8.8× speedup\n\nThis was the most important part of the project.\n\nMy first implementation directly copied raw K/V tensors.\n\nIt was fast.\n\nVery fast.\n\nThe benchmark reported an 8.8× speedup.\n\nThere was one problem.\n\nIt was wrong.\n\nllama.cpp tracks more than the K/V tensor values. Cache cells also have position and sequence metadata used to construct the attention mask.\n\nCopying tensor values without restoring that bookkeeping produced an inert fragment.\n\nThe model skipped prefix computation...\n\n...but attention could not actually see the restored prefix.\n\n14 of 24 cache hits reproduced, token for token, the output of a generation with no prefix at all.\n\nThe “speedup” was dropped context.\n\nSo I discarded it.\n\nTiming is not enough\n\nA broken cache can be fast.\n\nThat's why EdgeSync now runs two correctness checks.\n\nFirst:\n\nWarm output must match cold output for the same prompt.\n\nResults:\n\nARM64: 28/28\n\nx86-64: 24/24\n\nSecond:\n\nThe warm output is compared with a generation from the suffix alone, without the prefix.\n\nIf both outputs match, the injected fragment may be doing nothing.\n\nResults:\n\nARM64: 0/28 matches\n\nx86-64: 0/24 matches\n\nThis second control is what caught the broken raw-tensor implementation.\n\nA latency benchmark alone would not have caught it.\n\nWhat I learned\n\nThere is a broader lesson here for LLM infrastructure benchmarking:\n\nSkipping computation is not proof that state reuse works.\n\nIf a cache implementation silently drops context, it can look extremely fast.\n\nCorrectness must be part of the benchmark.\n\nNot a separate test.\n\nNot an assumption.\n\nPart of the measurement itself.\n\nWhat EdgeSync does not claim\n\nI deliberately marked several parts of the project as unvalidated or unsound.\n\nSemantic approximate prefix reuse is not proven.\n\nReusing KV state between merely “similar” prefixes is incorrect with the current mechanism.\n\nCross-engine reuse does not work with opaque llama.cpp sequence-state blobs.\n\nPer-layer striding cannot preserve output.\n\nThe Android JNI bridge and several cache management components still need end-to-end validation.\n\nThe on-device ARM64 sampling is also thin and needs to be rerun with more timed repeats.\n\nI would rather publish a smaller result that can be falsified than a larger claim that cannot survive reproduction.\n\nWhat's next\n\nThe next steps are:\n\nRerun ARM64 with multiple timed repeats.\n\nCompare complete generated strings instead of one token.\n\nBenchmark HNSW against an exact radix-tree lookup.\n\nMeasure state restoration cost as prefix length increases.\n\nRemove or redefine the current PARTIAL-hit mechanism.\n\nThe code, benchmark methodology and measured manifests are public.\n\nEdgeSync-LLM:\n\n[https://github.com/bossandboss/EdgeSync-LLM](https://github.com/bossandboss/EdgeSync-LLM)\n\nIf you work on llama.cpp, local LLM inference, KV cache management or edge AI, I would especially value attempts to reproduce or break the benchmark.\n\nFinding a flaw is more useful to me than a GitHub star.", "url": "https://wpnews.pro/news/i-got-9-9-lower-ttft-on-a-real-android-phone-by-reusing-llama-cpp-kv-state", "canonical_source": "https://dev.to/bossandboss/i-got-99x-lower-ttft-on-a-real-android-phone-by-reusing-llamacpp-kv-state-1ngi", "published_at": "2026-07-11 21:17:57+00:00", "updated_at": "2026-07-11 21:44:22.201212+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "developer-tools"], "entities": ["EdgeSync-LLM", "llama.cpp", "Qwen2.5-0.5B-Instruct", "Android", "ARM64"], "alternates": {"html": "https://wpnews.pro/news/i-got-9-9-lower-ttft-on-a-real-android-phone-by-reusing-llama-cpp-kv-state", "markdown": "https://wpnews.pro/news/i-got-9-9-lower-ttft-on-a-real-android-phone-by-reusing-llama-cpp-kv-state.md", "text": "https://wpnews.pro/news/i-got-9-9-lower-ttft-on-a-real-android-phone-by-reusing-llama-cpp-kv-state.txt", "jsonld": "https://wpnews.pro/news/i-got-9-9-lower-ttft-on-a-real-android-phone-by-reusing-llama-cpp-kv-state.jsonld"}}