{"slug": "hexcore-low-latency-paged-kv-cache-allocator-in-c-20-and-cuda", "title": "HexCore: Low-Latency Paged KV Cache Allocator in C++20 and CUDA", "summary": "HexCore, a low-latency paged KV cache allocator for LLM inference written in C++20 and CUDA, has been released under the Apache License 2.0 by Rasuljanov Muhammadali. The CPU-side allocator and related components pass 176 unit tests, including a real concurrent 2M-item producer/consumer test verified data-race-free under ThreadSanitizer, and coverage-guided fuzzing completed ~1.3M executions with zero crashes. However, the GPU kernels remain unverified due to lack of CUDA toolkit in the development environment, and the project is positioned as a component in the same space as vLLM's PagedAttention and FlashAttention.", "body_md": "**Author:** Rasuljanov Muhammadali\n**License:** [Apache License 2.0](/password162156/hexcore-llm/blob/main/LICENSE)\n\nPaged, quantized (FP16/INT4/INT2) KV-cache attention kernels for LLM inference — a component in the same space as vLLM's PagedAttention, FlashAttention, or TensorRT-LLM's KV-cache management.\n\nThis README exists to be honest about exactly what has and hasn't been verified, because the gap between \"code exists\" and \"code is safe to deploy\" is where production incidents come from.\n\n| Component | Verified how |\n|---|---|\n`hexcore::host::KVCacheConfig` / `PagedKVCache` (CPU allocator) |\n65 unit tests (g++, C++20), 100% passing |\n`hexcore::host` exception hierarchy (`HexcoreOOMException` , `HexcoreInvalidBlockException` ) + noexcept `try_*` status-code API |\n36 unit tests (`test_exceptions_and_metrics.cpp` ), 100% passing |\n`hexcore::host::MetricsCollector` (cache hit rate, active block gauge, allocation latency) |\n22 unit tests (`test_metrics.cpp` ) + integration coverage wired into the allocator, 100% passing |\n`hexcore::host::LockFreeRingBuffer` + `SpeculativeAllocator` (predictive prefetch) |\n21 unit tests including a REAL concurrent 2M-item producer/consumer test, verified data-race-free under ThreadSanitizer |\n`hexcore::host::FaultTolerantKVCache` (canary/checksum detection + migration) |\n19 unit tests; real measured CPU-side migration latency (p50 ~0.2us, see `benchmarks/README.md` ) |\n| Adversarial testing (ASan, UBSan, TSan, fuzzing, collision search) | Found and fixed 2 real bugs (signed integer overflow, inconsistent exception type) + confirmed several documented limitations empirically. Full report: `ADVERSARIAL_TESTING_REPORT.md` |\n| Coverage-guided fuzzing (libFuzzer) | ~1.3M executions, coverage plateaued at 656 edges, zero crashes -- see `fuzz/` |\n`pip install` packaging |\nVerified end-to-end: built a real sdist, installed it in total isolation (original source hidden), ran the full test suite against only the installed package. Full report: `PACKAGING.md` -- note: publish under `hexcore-llm` , not `hexcore` , which is already taken on PyPI by an unrelated project |\n`hexcore.host` Python bindings (allocator, exceptions, metrics) |\npybind11 module, built via `setup.py` , 13 pytest tests passing |\n| CPU allocator performance | Real microbenchmark numbers in `benchmarks/README.md` |\n| GPU/CPU layout math agreement | Independently re-derived and cross-checked on the host side (`test_layout_consistency.cpp` ) |\n| CI (host-only jobs) | `.github/workflows/ci.yml` , manually dry-run end-to-end during development, including the TSan race check |\n\n| Component | Why not verified | What to do before trusting it |\n|---|---|---|\n`src/cuda/*.cu` kernels (the actual attention math) |\nNo GPU/CUDA toolkit in the dev sandbox | Build with the CMake project on a CUDA machine, run `ctest` |\n`include/hexcore/cuda/persistent_kernel.cuh` (checkpoint-and-yield decode kernel) |\nSame, plus needs real device occupancy queries this sandbox can't run | Compile with `nvcc --ptxas-options=-v` to get real register/occupancy numbers, then profile the actual time-slice budget against measured relaunch latency before trusting `kTimeSliceCycleBudget` |\n`tests/cuda/test_layout_cross_check.cu` |\nSame | `ctest -R cuda_host_layout_cross_check` |\n`tests/test_phase_2_2.cpp` (original correctness suite) |\nSame | `ctest -R phase_2_2_correctness` |\n`hexcore/cuda/torch_ext.cpp` (PyTorch bindings) |\nNo CUDA/libtorch available | See `hexcore/cuda/BUILD_AND_TEST.md` |\n`benchmarks/cuda/bench_paged_attention.cu` |\nSame | See `benchmarks/README.md` |\n| Hopper (sm_90) TMA path | Flagged as unwired even in the original code | Not started |\n`.github/workflows/ci.yml` 's `cuda-tests` job |\nNo GPU runner available | Disabled (`if: false` ) until provisioned and verified |\n\n**Practical implication:** if you're evaluating HexCore, assume the GPU\nkernels are an unverified first draft, even though the code looks\ncomplete. The CPU-side scaffolding around them (allocator, layout\nsafety checks, Python bindings, CI) is solid; the core attention math\nis not yet proven correct on real hardware.\n\n```\ninclude/hexcore/cuda/     -- GPU kernel headers (layout math, attention kernel interface,\n                              persistent_kernel.cuh -- checkpoint-and-yield design, UNVERIFIED)\ninclude/hexcore/host/     -- CPU allocator headers (no CUDA dependency):\n                              kv_cache_config.hpp        -- layout math (bytes/block, etc.)\n                              paged_kv_cache.hpp          -- the allocator itself\n                              exceptions.hpp              -- HexcoreOOMException, HexcoreInvalidBlockException, try_* status codes\n                              metrics.hpp                  -- MetricsCollector (hit rate, active blocks, alloc latency)\n                              speculative_prefetch.hpp     -- lock-free ring buffer + predictive prefetch allocator\n                              fault_tolerance.hpp          -- canary/checksum corruption detection + sequence migration\nsrc/cuda/                 -- GPU kernel implementations\ntests/host/                -- CPU-only tests (run anywhere)\ntests/cuda/                -- GPU-dependent tests (need CUDA toolkit)\nbenchmarks/host/           -- CPU allocator microbenchmarks (real numbers in benchmarks/README.md)\nbenchmarks/cuda/           -- GPU kernel benchmark harness (unverified)\npython/                    -- REMOVED as of this pass; see PACKAGING.md for why.\n                              The Python package now lives at repo root as\n                              hexcore/ (importable as `import hexcore` once\n                              installed), with setup.py/pyproject.toml/\n                              MANIFEST.in also at root -- required for\n                              `pip install .` (and eventually `pip install\n                              HexCore`) to actually work. See PACKAGING.md\n                              for a real, verified end-to-end proof.\nhexcore/                    -- The pip-installable Python package (host verified, cuda unverified)\ntests/python/               -- pytest suite for hexcore.host (13/13 passing, verified against a REAL pip install)\n.github/workflows/         -- CI (host job verified via manual dry-run, GPU job disabled)\n```\n\nA later round of requests asked for \"zero-latency predictive allocation,\" a persistent kernel that \"bypasses the TDR watchdog,\" and \"<1 microsecond\" fault recovery. Those exact framings aren't physically achievable (negative latency isn't a thing; disabling the OS's GPU watchdog is a stability regression, not an optimization; a sub-microsecond bound can't honestly cover GPU-side data movement that wasn't measured). What got built instead, and what's real about each:\n\n— a genuinely lock-free SPSC ring buffer (verified race-free under ThreadSanitizer with a real 2-million-item concurrent test) backing a one-block-ahead predictive allocator. It`speculative_prefetch.hpp`\n\n*hides*allocation latency on correct predictions (O(1) hit path) and*pays*a real, measured, counted cost on mispredictions — see`confirmed_hits()`\n\n/`mispredicted()`\n\n. No claim of negative latency.— a checkpoint-and-yield design (the real technique behind \"persistent kernels\" in production inference engines): keep KV data hot in shared memory across decode steps, but voluntarily return control to the host well within the TDR window and immediately relaunch. This keeps the watchdog's safety net intact instead of defeating it.`persistent_kernel.cuh`\n\n**Unverified**— no CUDA toolkit/GPU in this environment; treat as a documented design to build and profile, not working code.— FNV-1a checksums for corruption`fault_tolerance.hpp`\n\n*detection on access*(not real-time interception, which isn't possible in software) and a migration path built on the existing allocator API. Real measured numbers: CPU-side bookkeeping alone is sub-microsecond at p50 on this sandbox's hardware — but that explicitly excludes the GPU-side KV data copy a real migration would also need, which is bandwidth-bound and wasn't measured (no GPU here). See`benchmarks/README.md`\n\nfor the numbers and the caveat in full.\n\n`PagedKVCache`\n\nnever crashes on OOM or a bad `block_id`\n\n— every failure\npath either throws a typed exception or returns a status code,\ndepending which method you call:\n\n```\n// Throwing API (default) -- catch by specific type or by the common base:\ntry {\n    cache.grow_sequence_or_throw(seq_id, new_len);\n} catch (const hexcore::host::HexcoreOOMException& e) {\n    // preempt/evict and retry\n}\n\n// noexcept status-code API -- for hot loops / code that can't use exceptions:\nauto status = cache.try_grow_sequence(seq_id, new_len);\nif (status == hexcore::host::HexcoreStatus::kOutOfMemory) { /* ... */ }\n```\n\nAttach a `MetricsCollector`\n\n(optional, `nullptr`\n\nby default, zero\noverhead when absent) to get live cache hit rate, active block count,\nand allocation latency stats, thread-safe via atomics:\n\n```\nhexcore::host::MetricsCollector metrics;\nhexcore::host::PagedKVCache cache(config, num_phys_blocks, &metrics);\n// ... use cache normally ...\nstd::cout << metrics.to_string(); // e.g. for a log line\ng++ -std=c++20 -Iinclude tests/host/test_layout_consistency.cpp -o t1 && ./t1\ng++ -std=c++20 -Iinclude tests/host/test_paged_kv_cache.cpp -o t2 && ./t2\npip install .  # once published: pip install hexcore-llm (see PACKAGING.md\n                # for why not bare \"hexcore\" -- that name is already taken)\ncd /tmp && python -m pytest /path/to/hexcore_phase22/tests/python  # run from elsewhere so pytest\n                                                                     # doesn't shadow the installed\n                                                                     # package with the local source dir\ncmake -B build -DCMAKE_BUILD_TYPE=Release\ncmake --build build -j\ncd build && ctest --output-on-failure\n```\n\nLicensed under the [Apache License, Version 2.0](/password162156/hexcore-llm/blob/main/LICENSE) — the same\nlicense used by TensorFlow, PyTorch, and Kubernetes, chosen\nspecifically because it includes an explicit patent grant (Section 3),\nwhich is what lets companies adopt the code without individually\nnegotiating patent risk. Copyright © 2026 Rasuljanov Muhammadali. See\n[NOTICE](/password162156/hexcore-llm/blob/main/NOTICE) for the required attribution notice.", "url": "https://wpnews.pro/news/hexcore-low-latency-paged-kv-cache-allocator-in-c-20-and-cuda", "canonical_source": "https://github.com/password162156/hexcore-llm", "published_at": "2026-07-31 06:39:24+00:00", "updated_at": "2026-07-31 06:52:27.080829+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-infrastructure", "ai-research"], "entities": ["HexCore", "Rasuljanov Muhammadali", "vLLM", "PagedAttention", "FlashAttention", "TensorRT-LLM", "PyPI", "Apache License 2.0"], "alternates": {"html": "https://wpnews.pro/news/hexcore-low-latency-paged-kv-cache-allocator-in-c-20-and-cuda", "markdown": "https://wpnews.pro/news/hexcore-low-latency-paged-kv-cache-allocator-in-c-20-and-cuda.md", "text": "https://wpnews.pro/news/hexcore-low-latency-paged-kv-cache-allocator-in-c-20-and-cuda.txt", "jsonld": "https://wpnews.pro/news/hexcore-low-latency-paged-kv-cache-allocator-in-c-20-and-cuda.jsonld"}}