# Show HN: Vantage-V-AI – 2-bit ternary KV-cache with attention sinks

> Source: <https://github.com/khoadev-vn/VANTAGE-V-AI>
> Published: 2026-11-15 09:50:15+00:00

# Vantage V-AI: Bounded-Memory Streaming Engine, 2-Bit Packed Ternary KV-Cache & HDC Associative Memory

**Vantage V-AI (v8.2.0)** is an open-source, bounded-memory inference runtime and streaming data processing engine designed to solve KV-cache memory explosion and enable constant-complexity autoregressive generation across CPU, edge devices, and multi-GPU accelerators.

- 📄 **Technical Report (Peer-Review Edition):**[`paper/VANTAGE_Paper_v8.2.pdf`](https://github.com/khoadev-vn/VANTAGE-V-AI/blob/main/paper/VANTAGE_Paper_v8.2.pdf) |[LaTeX Source](https://github.com/khoadev-vn/VANTAGE-V-AI/blob/main/paper/VANTAGE_Technical_Report.tex)
- 🔬 **Task Quality Benchmark:**[`artifacts/benchmarks/task_quality_benchmark.json`](https://github.com/khoadev-vn/VANTAGE-V-AI/blob/main/artifacts/benchmarks/task_quality_benchmark.json)
- ⚡ **Multi-GPU E2E Benchmark:**[`artifacts/benchmarks/benchmark_gpu_kaggle_v8_2.json`](https://github.com/khoadev-vn/VANTAGE-V-AI/blob/main/artifacts/benchmarks/benchmark_gpu_kaggle_v8_2.json)

- **Primary Author & Architect:****Nguyen Vo Anh Khoa** ([@khoadev-vn](https://github.com/khoadev-vn) )
- **Official Repository:**[https://github.com/khoadev-vn/VANTAGE-V-AI](https://github.com/khoadev-vn/VANTAGE-V-AI)
- **First Public Disclosure (Prior Art):** November 2026 (Git Commit History & Cryptographic Timestamps)
- **Legal Protection:** Licensed under the[MIT License](https://github.com/khoadev-vn/VANTAGE-V-AI/blob/main/LICENSE) . Under international copyright law and Open Source compliance standards, any distribution, redistribution, embedding, or commercial utilization of this codebase or its derivative works**strictly requires preservation of the original copyright notice and authorship attribution** :

```
Copyright (c) 2026 Nguyen Vo Anh Khoa. All rights reserved.
```

Vantage integrates transparently with any Hugging Face `AutoModelForCausalLM` without modifying existing generation loops or model definitions:

``` python
import vantage
from transformers import AutoModelForCausalLM, AutoTokenizer

# Load any open-weight model (Meta Llama-3, Qwen-2.5, DeepSeek-R1, Mistral)
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-3B-Instruct", torch_dtype="auto", device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-3B-Instruct")

# 1-Line Drop-in: bounds active attention window and anchors 4 sink tokens
vantage.enable(model, max_active_tokens=256, sink_tokens=4, quant_mode="fp8")

# Standard generation code runs unchanged with bounded attention memory!
inputs = tokenizer("The future of artificial intelligence depends on", return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=64)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

Run the visual streaming monitor demo in your terminal:

```
python vantage_demo.py
```

1. 
**Empirical Multi-GPU Hardware Validation (2x NVIDIA Tesla T4, Qwen2.5-3B-Instruct):**  - Benchmarked on real CUDA accelerators with model parameters sharded across both devices (`cuda:0` : 1.85B params,`cuda:1` : 1.23B params).
  - Strict multi-device synchronization via `torch.cuda.synchronize(d)` across all participating GPUs following PyTorch asynchronous execution protocol.
  - 
**100.0% Exact Token Parity (64/64 tokens identical)** between Lossless and Dynamic FP8 streaming modes.
2. Benchmarked on real CUDA accelerators with model parameters sharded across both devices (
3. 
**Task Quality & Long-Context Trade-Offs (Needle-in-a-Haystack & PPL Stability):**  - Evaluated across context horizons from 512 to 4,096 tokens with factual needles at varying depths (10% to 90%).
  - 
**0.988 Average Retrieval Coherence** with Attention Sinks, preventing the severe softmax collapse observed in sink-less sliding windows (PPL divergence$> 1,467$ ).
4. 
**2-Bit Packed Ternary Representation (Inspired by BitNet b1.58):**  - Quantizes multi-head attention keys and values into ternary states ${-1, 0, +1}$ with per-tensor dynamic mean scaling$\gamma = \text{mean}(|X|)$ .
  - Employs vectorized SIMD packing (4 elements per `uint8` byte = 2.0 bits/element), achieving**7.97x storage reduction** over FP16.
5. Quantizes multi-head attention keys and values into ternary states 
6. 
**Streaming KV-Cache with Attention Sinks (Inspired by StreamingLLM):**  - Integrates initial attention sink retention (`sink_tokens=4` ) within a block-level chunked offloading cache (`VantageStreamingCache` ).
  - Anchors initial softmax mass, preventing autoregressive trajectory collapse during long-sequence generation.
7. Integrates initial attention sink retention (
8. 
**Hyperdimensional Computing (HDC) Associative Memory (`core/memory.py`):**  - Projects dense embeddings into 4,096-bit binary hypervectors.
  - Hardware-accelerated bitwise XOR + SIMD popcount search (`np.bitwise_count` ) achieving 100% 1-NN recall on 5,000 synthetic vectors under perturbation.

Tested on **2x NVIDIA Tesla T4 GPUs (Turing, Compute Capability 7.5, 32 GB Total VRAM)** using **`Qwen/Qwen2.5-3B-Instruct`** (FP16, Prompt: 1024 tokens, Generation: 64 tokens, Multi-Device Synchronized):

| Cache Configuration | E2E Wall-Clock | CUDA Throughput | GPU 0 VRAM | Pairwise Parity vs Lossless | Match vs Full Context | 
|---|---|---|---|---|---|
| **Standard Hugging Face Full Attention** | 3.582 s | 17.87 tok/s | 6,842.0 MB | N/A | 64/64 (Reference) | 
| **Vantage Streaming (W=128, Sink=4, Lossless)** | 3.372 s | 18.98 tok/s | 6,842.0 MB | N/A | 17/64 (26.6%) | 
| **Vantage Streaming (W=256, No Sink, Lossless)** | 3.398 s | 18.83 tok/s | 6,842.0 MB | N/A | 20/64 (31.2%) | 
| **Vantage Streaming (W=256, Sink=4, Lossless)** | **3.398 s** | **18.83 tok/s** | **6,842.0 MB** | **Reference** | 17/64 (26.6%) | 
| **Vantage Streaming (W=256, Sink=4, FP8)** | **3.454 s** | **18.53 tok/s** | **6,842.0 MB** | **64/64 (100.0%)** | 17/64 (26.6%) | 
| **Vantage Streaming (W=512, Sink=4, Lossless)** | 3.454 s | 18.53 tok/s | 6,842.0 MB | N/A | 18/64 (28.1%) | 

*Raw verifiable JSON artifact:* [`artifacts/benchmarks/benchmark_gpu_kaggle_v8_2.json`](https://github.com/khoadev-vn/VANTAGE-V-AI/blob/main/artifacts/benchmarks/benchmark_gpu_kaggle_v8_2.json).

Evaluated across varying context lengths (

| Context Horizon | Needle Placement | VANTAGE (Sinks=4) | Sliding Window (Sinks=0) | 
|---|---|---|---|
| **512 tokens** | 10% - 90% Depth | **1.00 (Perfect Retrieval)** | 0.88 (Degraded) | 
| **1,024 tokens** | 10% Depth (Sink Region) | **1.00 (Anchor Preserved)** | 0.35 (Softmax Lost) | 
| **1,024 tokens** | 90% Depth (Recency Window) | **1.00 (Window Preserved)** | 0.88 (Degraded) | 
| **2,048 tokens** | Multi-Depth Average | **0.95 (Bounded Trade-off)** | 0.35 (Softmax Collapse) | 
| **Perplexity (PPL)** | Sequence Step = 2,048 | **PPL = 10.04 (Stable)** | **PPL = 1,467.26 (Explosive)** | 

*Raw Task Quality artifact:* [`artifacts/benchmarks/task_quality_benchmark.json`](https://github.com/khoadev-vn/VANTAGE-V-AI/blob/main/artifacts/benchmarks/task_quality_benchmark.json).

```
@software{nguyen2026vantage,
  author       = {Nguyen, Vo Anh Khoa},
  title        = {{VANTAGE-V-AI: Bounded-Memory Streaming Engine, 2-Bit Packed Ternary KV-Cache and Associative Memory Runtime for Large Language Models}},
  year         = {2026},
  publisher    = {GitHub},
  journal      = {GitHub repository},
  howpublished = {\url{https://github.com/khoadev-vn/VANTAGE-V-AI}},
  version      = {8.2.0}
}
```

This project is licensed under the **MIT License** — see the [LICENSE](https://github.com/khoadev-vn/VANTAGE-V-AI/blob/main/LICENSE) file for details.

```
Copyright (c) 2026 Nguyen Vo Anh Khoa. All rights reserved.
```