Show HN: Vantage-V-AI – 2-bit ternary KV-cache with attention sinks

Nguyen Vo Anh Khoa released Vantage V-AI v8.2.0, an MIT-licensed open-source inference runtime that bounds KV-cache memory using a 2-bit packed ternary key/value representation, attention sinks and a 256-token active window, and drops into any Hugging Face AutoModelForCausalLM with a one-line vantage.enable() call. Benchmarked on 2x NVIDIA Tesla T4 GPUs with Qwen2.5-3B-Instruct sharded across both devices, the runtime reported 100.0% exact token parity (64/64 tokens) between lossless and dynamic FP8 streaming modes and 0.988 average retrieval coherence on needle-in-a-haystack tests from 512 to 4,096 tokens, versus perplexity divergence above 1,467 for sink-less sliding windows.

Vantage V-AI: Bounded-Memory Streaming Engine, 2-Bit Packed Ternary KV-Cache & HDC Associative Memory Vantage V-AI v8.2.0 is an open-source, bounded-memory inference runtime and streaming data processing engine designed to solve KV-cache memory explosion and enable constant-complexity autoregressive generation across CPU, edge devices, and multi-GPU accelerators. - 📄 Technical Report Peer-Review Edition : paper/VANTAGE Paper v8.2.pdf https://github.com/khoadev-vn/VANTAGE-V-AI/blob/main/paper/VANTAGE Paper v8.2.pdf | LaTeX Source https://github.com/khoadev-vn/VANTAGE-V-AI/blob/main/paper/VANTAGE Technical Report.tex - 🔬 Task Quality Benchmark: artifacts/benchmarks/task quality benchmark.json https://github.com/khoadev-vn/VANTAGE-V-AI/blob/main/artifacts/benchmarks/task quality benchmark.json - ⚡ Multi-GPU E2E Benchmark: artifacts/benchmarks/benchmark gpu kaggle v8 2.json https://github.com/khoadev-vn/VANTAGE-V-AI/blob/main/artifacts/benchmarks/benchmark gpu kaggle v8 2.json - Primary Author & Architect: Nguyen Vo Anh Khoa @khoadev-vn https://github.com/khoadev-vn - Official Repository: https://github.com/khoadev-vn/VANTAGE-V-AI https://github.com/khoadev-vn/VANTAGE-V-AI - First Public Disclosure Prior Art : November 2026 Git Commit History & Cryptographic Timestamps - Legal Protection: Licensed under the MIT License https://github.com/khoadev-vn/VANTAGE-V-AI/blob/main/LICENSE . Under international copyright law and Open Source compliance standards, any distribution, redistribution, embedding, or commercial utilization of this codebase or its derivative works strictly requires preservation of the original copyright notice and authorship attribution : Copyright c 2026 Nguyen Vo Anh Khoa. All rights reserved. Vantage integrates transparently with any Hugging Face AutoModelForCausalLM without modifying existing generation loops or model definitions: python import vantage from transformers import AutoModelForCausalLM, AutoTokenizer Load any open-weight model Meta Llama-3, Qwen-2.5, DeepSeek-R1, Mistral model = AutoModelForCausalLM.from pretrained "Qwen/Qwen2.5-3B-Instruct", torch dtype="auto", device map="auto" tokenizer = AutoTokenizer.from pretrained "Qwen/Qwen2.5-3B-Instruct" 1-Line Drop-in: bounds active attention window and anchors 4 sink tokens vantage.enable model, max active tokens=256, sink tokens=4, quant mode="fp8" Standard generation code runs unchanged with bounded attention memory inputs = tokenizer "The future of artificial intelligence depends on", return tensors="pt" .to model.device outputs = model.generate inputs, max new tokens=64 print tokenizer.decode outputs 0 , skip special tokens=True Run the visual streaming monitor demo in your terminal: python vantage demo.py 1. Empirical Multi-GPU Hardware Validation 2x NVIDIA Tesla T4, Qwen2.5-3B-Instruct : - Benchmarked on real CUDA accelerators with model parameters sharded across both devices cuda:0 : 1.85B params, cuda:1 : 1.23B params . - Strict multi-device synchronization via torch.cuda.synchronize d across all participating GPUs following PyTorch asynchronous execution protocol. - 100.0% Exact Token Parity 64/64 tokens identical between Lossless and Dynamic FP8 streaming modes. 2. Benchmarked on real CUDA accelerators with model parameters sharded across both devices 3. Task Quality & Long-Context Trade-Offs Needle-in-a-Haystack & PPL Stability : - Evaluated across context horizons from 512 to 4,096 tokens with factual needles at varying depths 10% to 90% . - 0.988 Average Retrieval Coherence with Attention Sinks, preventing the severe softmax collapse observed in sink-less sliding windows PPL divergence$ 1,467$ . 4. 2-Bit Packed Ternary Representation Inspired by BitNet b1.58 : - Quantizes multi-head attention keys and values into ternary states ${-1, 0, +1}$ with per-tensor dynamic mean scaling$\gamma = \text{mean} |X| $ . - Employs vectorized SIMD packing 4 elements per uint8 byte = 2.0 bits/element , achieving 7.97x storage reduction over FP16. 5. Quantizes multi-head attention keys and values into ternary states 6. Streaming KV-Cache with Attention Sinks Inspired by StreamingLLM : - Integrates initial attention sink retention sink tokens=4 within a block-level chunked offloading cache VantageStreamingCache . - Anchors initial softmax mass, preventing autoregressive trajectory collapse during long-sequence generation. 7. Integrates initial attention sink retention 8. Hyperdimensional Computing HDC Associative Memory core/memory.py : - Projects dense embeddings into 4,096-bit binary hypervectors. - Hardware-accelerated bitwise XOR + SIMD popcount search np.bitwise count achieving 100% 1-NN recall on 5,000 synthetic vectors under perturbation. Tested on 2x NVIDIA Tesla T4 GPUs Turing, Compute Capability 7.5, 32 GB Total VRAM using Qwen/Qwen2.5-3B-Instruct FP16, Prompt: 1024 tokens, Generation: 64 tokens, Multi-Device Synchronized : | Cache Configuration | E2E Wall-Clock | CUDA Throughput | GPU 0 VRAM | Pairwise Parity vs Lossless | Match vs Full Context | |---|---|---|---|---|---| | Standard Hugging Face Full Attention | 3.582 s | 17.87 tok/s | 6,842.0 MB | N/A | 64/64 Reference | | Vantage Streaming W=128, Sink=4, Lossless | 3.372 s | 18.98 tok/s | 6,842.0 MB | N/A | 17/64 26.6% | | Vantage Streaming W=256, No Sink, Lossless | 3.398 s | 18.83 tok/s | 6,842.0 MB | N/A | 20/64 31.2% | | Vantage Streaming W=256, Sink=4, Lossless | 3.398 s | 18.83 tok/s | 6,842.0 MB | Reference | 17/64 26.6% | | Vantage Streaming W=256, Sink=4, FP8 | 3.454 s | 18.53 tok/s | 6,842.0 MB | 64/64 100.0% | 17/64 26.6% | | Vantage Streaming W=512, Sink=4, Lossless | 3.454 s | 18.53 tok/s | 6,842.0 MB | N/A | 18/64 28.1% | Raw verifiable JSON artifact: artifacts/benchmarks/benchmark gpu kaggle v8 2.json https://github.com/khoadev-vn/VANTAGE-V-AI/blob/main/artifacts/benchmarks/benchmark gpu kaggle v8 2.json . Evaluated across varying context lengths | Context Horizon | Needle Placement | VANTAGE Sinks=4 | Sliding Window Sinks=0 | |---|---|---|---| | 512 tokens | 10% - 90% Depth | 1.00 Perfect Retrieval | 0.88 Degraded | | 1,024 tokens | 10% Depth Sink Region | 1.00 Anchor Preserved | 0.35 Softmax Lost | | 1,024 tokens | 90% Depth Recency Window | 1.00 Window Preserved | 0.88 Degraded | | 2,048 tokens | Multi-Depth Average | 0.95 Bounded Trade-off | 0.35 Softmax Collapse | | Perplexity PPL | Sequence Step = 2,048 | PPL = 10.04 Stable | PPL = 1,467.26 Explosive | Raw Task Quality artifact: artifacts/benchmarks/task quality benchmark.json https://github.com/khoadev-vn/VANTAGE-V-AI/blob/main/artifacts/benchmarks/task quality benchmark.json . @software{nguyen2026vantage, author = {Nguyen, Vo Anh Khoa}, title = {{VANTAGE-V-AI: Bounded-Memory Streaming Engine, 2-Bit Packed Ternary KV-Cache and Associative Memory Runtime for Large Language Models}}, year = {2026}, publisher = {GitHub}, journal = {GitHub repository}, howpublished = {\url{https://github.com/khoadev-vn/VANTAGE-V-AI}}, version = {8.2.0} } This project is licensed under the MIT License — see the LICENSE https://github.com/khoadev-vn/VANTAGE-V-AI/blob/main/LICENSE file for details. Copyright c 2026 Nguyen Vo Anh Khoa. All rights reserved.