HexCore: Low-Latency Paged KV Cache Allocator in C++20 and CUDA

Author: Rasuljanov Muhammadali License: Apache License 2.0

Paged, quantized (FP16/INT4/INT2) KV-cache attention kernels for LLM inference — a component in the same space as vLLM's PagedAttention, FlashAttention, or TensorRT-LLM's KV-cache management.

This README exists to be honest about exactly what has and hasn't been verified, because the gap between "code exists" and "code is safe to deploy" is where production incidents come from.

Practical implication: if you're evaluating HexCore, assume the GPU kernels are an unverified first draft, even though the code looks complete. The CPU-side scaffolding around them (allocator, layout safety checks, Python bindings, CI) is solid; the core attention math is not yet proven correct on real hardware.

include/hexcore/cuda/     -- GPU kernel headers (layout math, attention kernel interface,
                              persistent_kernel.cuh -- checkpoint-and-yield design, UNVERIFIED)
include/hexcore/host/     -- CPU allocator headers (no CUDA dependency):
                              kv_cache_config.hpp        -- layout math (bytes/block, etc.)
                              paged_kv_cache.hpp          -- the allocator itself
                              exceptions.hpp              -- HexcoreOOMException, HexcoreInvalidBlockException, try_* status codes
                              metrics.hpp                  -- MetricsCollector (hit rate, active blocks, alloc latency)
                              speculative_prefetch.hpp     -- lock-free ring buffer + predictive prefetch allocator
                              fault_tolerance.hpp          -- canary/checksum corruption detection + sequence migration
src/cuda/                 -- GPU kernel implementations
tests/host/                -- CPU-only tests (run anywhere)
tests/cuda/                -- GPU-dependent tests (need CUDA toolkit)
benchmarks/host/           -- CPU allocator microbenchmarks (real numbers in benchmarks/README.md)
benchmarks/cuda/           -- GPU kernel benchmark harness (unverified)
python/                    -- REMOVED as of this pass; see PACKAGING.md for why.
                              The Python package now lives at repo root as
                              hexcore/ (importable as `import hexcore` once
                              installed), with setup.py/pyproject.toml/
                              MANIFEST.in also at root -- required for
                              `pip install .` (and eventually `pip install
                              HexCore`) to actually work. See PACKAGING.md
                              for a real, verified end-to-end proof.
hexcore/                    -- The pip-installable Python package (host verified, cuda unverified)
tests/python/               -- pytest suite for hexcore.host (13/13 passing, verified against a REAL pip install)
.github/workflows/         -- CI (host job verified via manual dry-run, GPU job disabled)

A later round of requests asked for "zero-latency predictive allocation," a persistent kernel that "bypasses the TDR watchdog," and "<1 microsecond" fault recovery. Those exact framings aren't physically achievable (negative latency isn't a thing; disabling the OS's GPU watchdog is a stability regression, not an optimization; a sub-microsecond bound can't honestly cover GPU-side data movement that wasn't measured). What got built instead, and what's real about each:

— a genuinely lock-free SPSC ring buffer (verified race-free under ThreadSanitizer with a real 2-million-item concurrent test) backing a one-block-ahead predictive allocator. Itspeculative_prefetch.hpp

hidesallocation latency on correct predictions (O(1) hit path) andpaysa real, measured, counted cost on mispredictions — seeconfirmed_hits()

/mispredicted()

. No claim of negative latency.— a checkpoint-and-yield design (the real technique behind "persistent kernels" in production inference engines): keep KV data hot in shared memory across decode steps, but voluntarily return control to the host well within the TDR window and immediately relaunch. This keeps the watchdog's safety net intact instead of defeating it.persistent_kernel.cuh

Unverified— no CUDA toolkit/GPU in this environment; treat as a documented design to build and profile, not working code.— FNV-1a checksums for corruptionfault_tolerance.hpp

detection on access(not real-time interception, which isn't possible in software) and a migration path built on the existing allocator API. Real measured numbers: CPU-side bookkeeping alone is sub-microsecond at p50 on this sandbox's hardware — but that explicitly excludes the GPU-side KV data copy a real migration would also need, which is bandwidth-bound and wasn't measured (no GPU here). Seebenchmarks/README.md

for the numbers and the caveat in full.

PagedKVCache

never crashes on OOM or a bad block_id

— every failure path either throws a typed exception or returns a status code, depending which method you call:

// Throwing API (default) -- catch by specific type or by the common base:
try {
    cache.grow_sequence_or_throw(seq_id, new_len);
} catch (const hexcore::host::HexcoreOOMException& e) {
    // preempt/evict and retry
}

// noexcept status-code API -- for hot loops / code that can't use exceptions:
auto status = cache.try_grow_sequence(seq_id, new_len);
if (status == hexcore::host::HexcoreStatus::kOutOfMemory) { /* ... */ }

Attach a MetricsCollector

(optional, nullptr

by default, zero overhead when absent) to get live cache hit rate, active block count, and allocation latency stats, thread-safe via atomics:

hexcore::host::MetricsCollector metrics;
hexcore::host::PagedKVCache cache(config, num_phys_blocks, &metrics);
// ... use cache normally ...
std::cout << metrics.to_string(); // e.g. for a log line
g++ -std=c++20 -Iinclude tests/host/test_layout_consistency.cpp -o t1 && ./t1
g++ -std=c++20 -Iinclude tests/host/test_paged_kv_cache.cpp -o t2 && ./t2
pip install .  # once published: pip install hexcore-llm (see PACKAGING.md
cd /tmp && python -m pytest /path/to/hexcore_phase22/tests/python  # run from elsewhere so pytest
cmake -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build -j
cd build && ctest --output-on-failure

Licensed under the Apache License, Version 2.0 — the same license used by TensorFlow, PyTorch, and Kubernetes, chosen specifically because it includes an explicit patent grant (Section 3), which is what lets companies adopt the code without individually negotiating patent risk. Copyright © 2026 Rasuljanov Muhammadali. See NOTICE for the required attribution notice.