# HexCore: Low-Latency Paged KV Cache Allocator in C++20 and CUDA > Source: > Published: 2026-07-31 06:39:24+00:00 **Author:** Rasuljanov Muhammadali **License:** [Apache License 2.0](/password162156/hexcore-llm/blob/main/LICENSE) Paged, quantized (FP16/INT4/INT2) KV-cache attention kernels for LLM inference — a component in the same space as vLLM's PagedAttention, FlashAttention, or TensorRT-LLM's KV-cache management. This README exists to be honest about exactly what has and hasn't been verified, because the gap between "code exists" and "code is safe to deploy" is where production incidents come from. | Component | Verified how | |---|---| `hexcore::host::KVCacheConfig` / `PagedKVCache` (CPU allocator) | 65 unit tests (g++, C++20), 100% passing | `hexcore::host` exception hierarchy (`HexcoreOOMException` , `HexcoreInvalidBlockException` ) + noexcept `try_*` status-code API | 36 unit tests (`test_exceptions_and_metrics.cpp` ), 100% passing | `hexcore::host::MetricsCollector` (cache hit rate, active block gauge, allocation latency) | 22 unit tests (`test_metrics.cpp` ) + integration coverage wired into the allocator, 100% passing | `hexcore::host::LockFreeRingBuffer` + `SpeculativeAllocator` (predictive prefetch) | 21 unit tests including a REAL concurrent 2M-item producer/consumer test, verified data-race-free under ThreadSanitizer | `hexcore::host::FaultTolerantKVCache` (canary/checksum detection + migration) | 19 unit tests; real measured CPU-side migration latency (p50 ~0.2us, see `benchmarks/README.md` ) | | Adversarial testing (ASan, UBSan, TSan, fuzzing, collision search) | Found and fixed 2 real bugs (signed integer overflow, inconsistent exception type) + confirmed several documented limitations empirically. Full report: `ADVERSARIAL_TESTING_REPORT.md` | | Coverage-guided fuzzing (libFuzzer) | ~1.3M executions, coverage plateaued at 656 edges, zero crashes -- see `fuzz/` | `pip install` packaging | Verified end-to-end: built a real sdist, installed it in total isolation (original source hidden), ran the full test suite against only the installed package. Full report: `PACKAGING.md` -- note: publish under `hexcore-llm` , not `hexcore` , which is already taken on PyPI by an unrelated project | `hexcore.host` Python bindings (allocator, exceptions, metrics) | pybind11 module, built via `setup.py` , 13 pytest tests passing | | CPU allocator performance | Real microbenchmark numbers in `benchmarks/README.md` | | GPU/CPU layout math agreement | Independently re-derived and cross-checked on the host side (`test_layout_consistency.cpp` ) | | CI (host-only jobs) | `.github/workflows/ci.yml` , manually dry-run end-to-end during development, including the TSan race check | | Component | Why not verified | What to do before trusting it | |---|---|---| `src/cuda/*.cu` kernels (the actual attention math) | No GPU/CUDA toolkit in the dev sandbox | Build with the CMake project on a CUDA machine, run `ctest` | `include/hexcore/cuda/persistent_kernel.cuh` (checkpoint-and-yield decode kernel) | Same, plus needs real device occupancy queries this sandbox can't run | Compile with `nvcc --ptxas-options=-v` to get real register/occupancy numbers, then profile the actual time-slice budget against measured relaunch latency before trusting `kTimeSliceCycleBudget` | `tests/cuda/test_layout_cross_check.cu` | Same | `ctest -R cuda_host_layout_cross_check` | `tests/test_phase_2_2.cpp` (original correctness suite) | Same | `ctest -R phase_2_2_correctness` | `hexcore/cuda/torch_ext.cpp` (PyTorch bindings) | No CUDA/libtorch available | See `hexcore/cuda/BUILD_AND_TEST.md` | `benchmarks/cuda/bench_paged_attention.cu` | Same | See `benchmarks/README.md` | | Hopper (sm_90) TMA path | Flagged as unwired even in the original code | Not started | `.github/workflows/ci.yml` 's `cuda-tests` job | No GPU runner available | Disabled (`if: false` ) until provisioned and verified | **Practical implication:** if you're evaluating HexCore, assume the GPU kernels are an unverified first draft, even though the code looks complete. The CPU-side scaffolding around them (allocator, layout safety checks, Python bindings, CI) is solid; the core attention math is not yet proven correct on real hardware. ``` include/hexcore/cuda/ -- GPU kernel headers (layout math, attention kernel interface, persistent_kernel.cuh -- checkpoint-and-yield design, UNVERIFIED) include/hexcore/host/ -- CPU allocator headers (no CUDA dependency): kv_cache_config.hpp -- layout math (bytes/block, etc.) paged_kv_cache.hpp -- the allocator itself exceptions.hpp -- HexcoreOOMException, HexcoreInvalidBlockException, try_* status codes metrics.hpp -- MetricsCollector (hit rate, active blocks, alloc latency) speculative_prefetch.hpp -- lock-free ring buffer + predictive prefetch allocator fault_tolerance.hpp -- canary/checksum corruption detection + sequence migration src/cuda/ -- GPU kernel implementations tests/host/ -- CPU-only tests (run anywhere) tests/cuda/ -- GPU-dependent tests (need CUDA toolkit) benchmarks/host/ -- CPU allocator microbenchmarks (real numbers in benchmarks/README.md) benchmarks/cuda/ -- GPU kernel benchmark harness (unverified) python/ -- REMOVED as of this pass; see PACKAGING.md for why. The Python package now lives at repo root as hexcore/ (importable as `import hexcore` once installed), with setup.py/pyproject.toml/ MANIFEST.in also at root -- required for `pip install .` (and eventually `pip install HexCore`) to actually work. See PACKAGING.md for a real, verified end-to-end proof. hexcore/ -- The pip-installable Python package (host verified, cuda unverified) tests/python/ -- pytest suite for hexcore.host (13/13 passing, verified against a REAL pip install) .github/workflows/ -- CI (host job verified via manual dry-run, GPU job disabled) ``` A later round of requests asked for "zero-latency predictive allocation," a persistent kernel that "bypasses the TDR watchdog," and "<1 microsecond" fault recovery. Those exact framings aren't physically achievable (negative latency isn't a thing; disabling the OS's GPU watchdog is a stability regression, not an optimization; a sub-microsecond bound can't honestly cover GPU-side data movement that wasn't measured). What got built instead, and what's real about each: — a genuinely lock-free SPSC ring buffer (verified race-free under ThreadSanitizer with a real 2-million-item concurrent test) backing a one-block-ahead predictive allocator. It`speculative_prefetch.hpp` *hides*allocation latency on correct predictions (O(1) hit path) and*pays*a real, measured, counted cost on mispredictions — see`confirmed_hits()` /`mispredicted()` . No claim of negative latency.— a checkpoint-and-yield design (the real technique behind "persistent kernels" in production inference engines): keep KV data hot in shared memory across decode steps, but voluntarily return control to the host well within the TDR window and immediately relaunch. This keeps the watchdog's safety net intact instead of defeating it.`persistent_kernel.cuh` **Unverified**— no CUDA toolkit/GPU in this environment; treat as a documented design to build and profile, not working code.— FNV-1a checksums for corruption`fault_tolerance.hpp` *detection on access*(not real-time interception, which isn't possible in software) and a migration path built on the existing allocator API. Real measured numbers: CPU-side bookkeeping alone is sub-microsecond at p50 on this sandbox's hardware — but that explicitly excludes the GPU-side KV data copy a real migration would also need, which is bandwidth-bound and wasn't measured (no GPU here). See`benchmarks/README.md` for the numbers and the caveat in full. `PagedKVCache` never crashes on OOM or a bad `block_id` — every failure path either throws a typed exception or returns a status code, depending which method you call: ``` // Throwing API (default) -- catch by specific type or by the common base: try { cache.grow_sequence_or_throw(seq_id, new_len); } catch (const hexcore::host::HexcoreOOMException& e) { // preempt/evict and retry } // noexcept status-code API -- for hot loops / code that can't use exceptions: auto status = cache.try_grow_sequence(seq_id, new_len); if (status == hexcore::host::HexcoreStatus::kOutOfMemory) { /* ... */ } ``` Attach a `MetricsCollector` (optional, `nullptr` by default, zero overhead when absent) to get live cache hit rate, active block count, and allocation latency stats, thread-safe via atomics: ``` hexcore::host::MetricsCollector metrics; hexcore::host::PagedKVCache cache(config, num_phys_blocks, &metrics); // ... use cache normally ... std::cout << metrics.to_string(); // e.g. for a log line g++ -std=c++20 -Iinclude tests/host/test_layout_consistency.cpp -o t1 && ./t1 g++ -std=c++20 -Iinclude tests/host/test_paged_kv_cache.cpp -o t2 && ./t2 pip install . # once published: pip install hexcore-llm (see PACKAGING.md # for why not bare "hexcore" -- that name is already taken) cd /tmp && python -m pytest /path/to/hexcore_phase22/tests/python # run from elsewhere so pytest # doesn't shadow the installed # package with the local source dir cmake -B build -DCMAKE_BUILD_TYPE=Release cmake --build build -j cd build && ctest --output-on-failure ``` Licensed under the [Apache License, Version 2.0](/password162156/hexcore-llm/blob/main/LICENSE) — the same license used by TensorFlow, PyTorch, and Kubernetes, chosen specifically because it includes an explicit patent grant (Section 3), which is what lets companies adopt the code without individually negotiating patent risk. Copyright © 2026 Rasuljanov Muhammadali. See [NOTICE](/password162156/hexcore-llm/blob/main/NOTICE) for the required attribution notice.