Don't use cosine similarity carelessly We fixed it this way

VectorPrism, an open-source retrieval engine from insightitsGit, claims to cut multi-vector storage costs to a single 1024-dimensional vector per chunk by multiplexing six specialized relevance subspaces plus a 16-float control header, using HNSW for stage-1 and intent-gated rescoring for stage-2. In a calibrated hard_adversarial finance pack, dense retrieval achieved only 7.1% Recall@10 and missed 13 of 14 queries, while the multi-channel approach recovered all 13 missed queries at 100% (13/13) with ~1.8 ms latency at 1000-doc scale, though the authors caution the results are not generalizable to all corpora.

Positional Subspace Multiplexing PSM & Intent-Gated 2-Stage Retrieval Engine for High-Scale RAG. One contiguous 1024dtensor. Six independently trained relevance subspaces. Stage-1 HNSW + Stage-2 intent-gated rescoring. Baseline vector-DB storage cost — not 6× multi-vector inflation. Interactive demo · · Benchmarks /insightitsGit/VectorPrism/blob/main/BENCHMARKS.md · Pilot guide /insightitsGit/VectorPrism/blob/main/PILOT.md Technical report /insightitsGit/VectorPrism/blob/main/demos/finance demo/TECHNICAL REPORT.md Keywords: pgvector multi-vector cost reduction, Intent-gated RAG retrieval engine, Causal retrieval for enterprise RAG, Positional subspace multiplexing vector search, Reduce hallucinations in root-cause RAG, VectorPrism, HNSW Scope honesty: the table below is from our calibrated hard adversarial finance pack dense is designed to miss . It is not a claim that every public corpus shows the same Miss@10. For partner corpora use scripts/corpus recovery audit.py /insightitsGit/VectorPrism/blob/main/scripts/corpus recovery audit.py — see . /insightitsGit/VectorPrism/blob/main/demos/external audit demos/external audit/ Dense fails on purpose in that pack . Multi-channel recovers those misses. | Metric | Result | |---|---| | Dense R@10 | 7.1% | | Dense Miss@10 | 13/14 93% | | Multi z-score recovered@10 | 13/13 100% | | RRF recovered@10 conservative | 10–11/13 77–85% | | Auto-graph recovered@10 | 11/13 85% | | 1000-doc scale recovered@10 | 13/13 ~1.8 ms | Full tables, caveats, and reproduce commands → BENCHMARKS.md Interactive query comparison dense vs multi → demo site https://insightitsgit.github.io/VectorPrism/ Raw JSON/MD artifacts → demos/finance demo/results/ Enterprise RAG is stuck between two bad defaults: Flat cosine over a single embedding — semantically “close” neighbors that are causally wrong, taxonomically wrong, or temporally expired. Teams call them funny neighbors ; production calls them hallucination fuel . Multi-vector indexing one ANN index per representation — better signal, but 500%–1,000% storage and query fan-out on pgvector / Qdrant bills. VectorPrism multiplexes six specialized representation subspaces plus a 16-float Control Header into a single 1024-dimensional contiguous buffer per chunk: | Constraint | VectorPrism answer | |---|---| | Storage | 1× vector footprint one vector 1024 / named full tensor | | Stage 1 | HNSW only on the 368d dense core slice | | Stage 2 | In-RAM zero-copy slice scoring with intent weights | | Early exit | Header filters epistemic truth , anchor dist , model version before heavy math | | Latency target | < 15ms end-to-end search SLA see benchmarks | Philosophically grounded channel design. Engineering-grounded memory contract. Production path for pgvector and Qdrant. Keywords: causal retrieval, incident log RAG, DevOps root-cause analysis, “why did the service fail” When on-call asks “Why did Server X crash at 3 AM?” , cosine-only RAG returns symptom-adjacent text. VectorPrism’s Time ODE & Directional Causality slice 896:1024 is trained with an asymmetric bilinear score q^{\top} M c PSMRetrievalEngine.causal score . Intent routing up-weights the causal channel on “why / cause / reason” queries so Stage-2 rescoring prefers cause→effect order , not merely lexical neighbors. Keywords: hyperbolic embeddings RAG, taxonomy search, medical ontology retrieval, legal hierarchy search Parent–child trees distort badly in Euclidean space. The Hyperbolic Taxonomy Porphyry slice 640:768 lives in a Poincaré ball norm < 1 and is scored with Poincaré distance in Stage 2. Hierarchy intents “category”, “parent”, “type of”, “tree” shift IntentClassifier weights toward hyperbolic structure for medical, legal, and product taxonomies. Keywords: bitemporal retrieval, compliance RAG, healthcare audit trail, finance document expiry filter Before Stage-2 matrix math, the 16d Control Header Manifest 0:16 exposes O 1 metadata: - Epistemic truth score soft by default; hard filter opt-in after ECE calibration - Identity anchor distance OOD / injection-risk gate in Stage 1 - Exact int64 timestamp packed in the header for audit / future filters — not applied as a Stage-1 SQL/Qdrant predicate today - Model version for safe re-ingest after retrains applied as a Stage-1 filter; search defaults to the checkpoint’s model version Stage 1 rejects low-truth, high-anchor-distance, or wrong- model version chunks before rescoring. Keywords: multi-vector RAG cost reduction, pgvector HNSW, Qdrant named vectors, high-scale vector search Instead of six ANN indexes, VectorPrism stores one 1024d tensor . Stage 1 indexes only the generated 368d dense core slice . Stage 2 pulls the full tensor for the top-~100 candidates and rescored slices in RAM. AI SaaS platforms keep multi-signal retrieval without multi-vector sticker shock. Ground truth: PSMTensorContract / VectorPrismTensorContract in tensor contract.py /insightitsGit/VectorPrism/blob/main/tensor contract.py . 1024-d VectorPrism Tensor float32 ┌──────────────────────────────────────────────────────────────────────────┐ │ 0 .. 15 16d Control Header Manifest │ │ 16 .. 383 368d Dense Semantic Core Hume / Wittgenstein │ │ 384 .. 511 128d Relational Group Algebra Aristotle / Al-Khwarizmi │ │ 512 .. 639 128d Disentangled Latent Space Jabir │ │ 640 .. 767 128d Hyperbolic Taxonomy Porphyry │ │ 768 .. 895 128d Identity Consistency Ibn Sina │ │ 896 ..1023 128d Time ODE & Causality Mulla Sadra / Spinoza │ └──────────────────────────────────────────────────────────────────────────┘ ▲ Stage-1 HNSW indexes ONLY dense core 16:384 → 368 dims | Inclusive range | Code slice start:end | Dims | Channel | Role | |---|---|---|---|---| 0000..0015 | HEADER 0:16 | 16 | Control Header Manifest | Bitmask, truth, anchor dist, timestamp, model version | 0016..0383 | DENSE CORE 16:384 | 368 | Dense Semantic Core | L2-normalized cosine space; Stage-1 ANN | 0384..0511 | RELATIONAL 384:512 | 128 | Relational Group Algebra | Train: TransE S+R\approx O ; serve today: L2 proximity -|q {\mathrm{rel}}-c {\mathrm{rel}}| no query-time relation id yet | 0512..0639 | DISENTANGLED 512:640 | 128 | Disentangled Latent Jabir | VIB latent z | 0640..0767 | HYPERBOLIC 640:768 | 128 | Hyperbolic Taxonomy Porphyry | Poincaré ball | 0768..0895 | IDENTITY 768:896 | 128 | Identity Consistency Ibn Sina | Distance-to-frozen v 0 ; Stage-1 gate only | 0896..1023 | CAUSAL TIME 896:1024 | 128 | Time ODE & Causality Spinoza | Scored as q^{\top} M c | Header sub-layout exact packing via PSMTensorContract.pack header / unpack header : | Slot | Field | Encoding | |---|---|---| 0 | Channel bitmask | uint32 ↔ float32 bit reinterpret | 1 | Epistemic truth | float32 in 0, 1 | 2 | Identity anchor distance | float32 | 3:5 | Bitemporal timestamp | int64 ↔ 2×float32 bit reinterpret | 5 | Model version | uint32 ↔ float32 bit reinterpret | 6:16 | Reserved | zero-filled | From PyPI pip install "vectorprism all " Or from git latest main / full adversarial packs git clone https://github.com/insightitsGit/VectorPrism.git cd VectorPrism python -m venv .venv && source .venv/bin/activate Windows: .venv\Scripts\activate pip install -U pip pip install -e ". all " vectorprism version vectorprism pilot-check pytest test psm.py test phases.py -q The PyPI wheel ships schema.sql and data/ .example.jsonl enough for pilot-check / run-all-smoke . Full adversarial finance packs stay in git , not on PyPI. Publish / release: PUBLISH.md /insightitsGit/VectorPrism/blob/main/PUBLISH.md · External pilot: · Production: /insightitsGit/VectorPrism/blob/main/PILOT.md PILOT.md PRODUCTION.md Core deps: torch , numpy , scipy , scikit-learn . Optional extras: encoder , postgres , qdrant , dev , all . docker compose up -d db docker compose run --rm test docker compose run --rm finance-pg docker compose run --rm production-smoke - DB: localhost:5433 · DSN postgresql://vectorprism:vectorprism@localhost:5433/vectorprism - Results: demos/finance demo/results/ PRODUCTION RESULTS.md , eval, live search JSON - Full checklist: · Docker notes: PRODUCTION.md DOCKER.md Encode raw text with a frozen 768d encoder → MultiTaskProjectionAdapter → contiguous 1024d tensor matches ingestion adapter.py + ingest pipeline.py . python import time import torch import numpy as np from base encoder import SentenceTransformerEncoder from ingestion adapter import MultiTaskProjectionAdapter, VectorPrismProjectionAdapter from tensor contract import PSMTensorContract as C, VectorPrismTensorContract from losses import anchor distance score Frozen base encoder 768d + trainable 6-head adapter encoder = SentenceTransformerEncoder "sentence-transformers/all-mpnet-base-v2" adapter = MultiTaskProjectionAdapter base dim=768 alias: VectorPrismProjectionAdapter adapter.eval texts = "Cache eviction storm preceded the 3 AM outage on Server X." base = encoder.encode texts 1, 768 header = C.pack header bitmask=C.default channel bitmask {"dense": True, "identity": True, "causal": True} , epistemic truth=1.0, anchor distance=0.0, timestamp=int time.time , model version=1, header t = torch.from numpy header .unsqueeze 0 1, 16 with torch.no grad : tensor 1024d, raw = adapter base, header t Fill identity distance into header slot 2 dist = anchor distance score raw "identity" , adapter.identity anchor v0 out = tensor 1024d.cpu .numpy .astype np.float32 out 0, C.HDR ANCHOR.start = float dist 0 .item assert out.shape == 1, 1024 assert out 0, C.DENSE CORE.start:C.DENSE CORE.end .shape == 368, meta = C.unpack header out 0 print meta bitmask, epistemic truth, anchor distance, timestamp, model version Production shorthand upsert path : python from checkpointing import load checkpoint from db client import PgVectorClient or QdrantVectorClient from ingest pipeline import VectorPrismIngestPipeline, IngestDocument from base encoder import SentenceTransformerEncoder ckpt = load checkpoint "checkpoints/vectorprism.pt" encoder = SentenceTransformerEncoder "sentence-transformers/all-mpnet-base-v2" db = PgVectorClient "postgresql://user:pass@localhost:5432/vectorprism" pipe = VectorPrismIngestPipeline encoder=encoder, adapter=ckpt "adapter" , db=db, model version=ckpt "model version" , enabled channels=ckpt.get "enabled channels" , pipe.upsert documents IngestDocument document id="inc-42", chunk text="Cache eviction preceded the outage." , Intent classification → HNSW on dense core → zero-copy slice rescoring PSMRetrievalEngine.search . python from checkpointing import load checkpoint from base encoder import SentenceTransformerEncoder from db client import PgVectorClient from ingest pipeline import VectorPrismIngestPipeline from retrieval engine import PSMRetrievalEngine, IntentClassifier, VectorPrismRetrievalEngine from tensor contract import PSMTensorContract as C ckpt = load checkpoint "checkpoints/vectorprism.pt" encoder = SentenceTransformerEncoder "sentence-transformers/all-mpnet-base-v2" db = PgVectorClient "postgresql://user:pass@localhost:5432/vectorprism" pipe = VectorPrismIngestPipeline encoder, ckpt "adapter" , db, model version=ckpt "model version" engine = PSMRetrievalEngine alias: VectorPrismRetrievalEngine db client=db, causal matrix=ckpt "causal matrix" , learned M for qᵀ M c hard truth filter=False, keep soft until ECE-calibrated query text = "Why did Server X crash at 3 AM?" query 1024d = pipe.encode query query text Optional: inspect intent weights dense, relational, disentangled, hyperbolic, causal w intent, filters = engine.classifier.classify query text print "w intent=", w intent, "filters=", filters hits = engine.search query 1024d, query text, top k=5 for h in hits: print h "document id" , h "final score" , h.get "chunk text", "" :120 Stage-2 scoring uses exact slices, e.g. causal: q c = query 1024d C.CAUSAL TIME.start:C.CAUSAL TIME.end s causal = engine.causal score q c, candidate causal matrix CLI equivalents: python train.py --channel dense --data data/dense pairs.example.jsonl \ --encoder sentence-transformers/all-mpnet-base-v2 --out checkpoints/vectorprism.pt python vectorprism.py ingest --checkpoint checkpoints/vectorprism.pt \ --documents data/documents.example.jsonl --backend pgvector --dsn "$VECTORPRISM PG DSN" python vectorprism.py search --checkpoint checkpoints/vectorprism.pt \ --query "Why did Server X crash at 3 AM?" --backend pgvector --dsn "$VECTORPRISM PG DSN" Exact DDL from schema.sql /insightitsGit/VectorPrism/blob/main/schema.sql : CREATE EXTENSION IF NOT EXISTS vector; CREATE TABLE IF NOT EXISTS psm document embeddings id UUID PRIMARY KEY DEFAULT gen random uuid , document id VARCHAR 255 NOT NULL UNIQUE, chunk text TEXT NOT NULL, -- Full 1024-Dimensional Composite Tensor Payload tensor 1024d vector 1024 NOT NULL, -- Generated Column for Stage 1 Dense Core Slice 16..383 368d -- pgvector subvector is 1-indexed: 17, 368 == zero-indexed 16:384 dense core slice vector 368 GENERATED ALWAYS AS subvector tensor 1024d, 17, 368 STORED, epistemic truth FLOAT NOT NULL DEFAULT 1.0, anchor dist FLOAT NOT NULL DEFAULT 0.0, valid timestamp BIGINT NOT NULL, model version INTEGER NOT NULL DEFAULT 0, created at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT TIMESTAMP ; CREATE INDEX IF NOT EXISTS idx psm dense core hnsw ON psm document embeddings USING hnsw dense core slice vector cosine ops WITH m = 16, ef construction = 64 ; CREATE INDEX IF NOT EXISTS idx psm epistemic truth ON psm document embeddings epistemic truth ; CREATE INDEX IF NOT EXISTS idx psm anchor dist ON psm document embeddings anchor dist ; CREATE INDEX IF NOT EXISTS idx psm model version ON psm document embeddings model version ; CREATE INDEX IF NOT EXISTS idx psm document id ON psm document embeddings document id ; Apply: psql "$VECTORPRISM PG DSN" -f schema.sql Matches QdrantVectorClient in db client.py /insightitsGit/VectorPrism/blob/main/db client.py : python from qdrant client import QdrantClient from qdrant client.http import models as qmodels client = QdrantClient url="http://localhost:6333" collection = "psm document embeddings" if not client.collection exists collection : client.create collection collection name=collection, vectors config={ Stage 1 — HNSW on dense core only "dense core slice": qmodels.VectorParams size=368, distance=qmodels.Distance.COSINE, hnsw config=qmodels.HnswConfigDiff m=16, ef construct=128 , , Stage 2 — full 1024d tensor flat / m=0, not a second ANN tax "full tensor": qmodels.VectorParams size=1024, distance=qmodels.Distance.COSINE, hnsw config=qmodels.HnswConfigDiff m=0 , , }, Payload fields used for Stage-1 filters: epistemic truth , anchor dist , model version . Payload also stores valid timestamp , chunk text , and document id timestamp is header metadata today; not a Stage-1 predicate . Targets enforced by the design and benchmark harness.py / live benchmark.py budgets: | Stage | Operation | Budget | |---|---|---| | Header | Bitmask / header unpack 0:16 via PSMTensorContract.unpack header | < 0.05 ms | | Stage 1 | HNSW coarse search on dense core slice 368d + header filters → top 100 | < 10 ms | | Stage 2 | RAM zero-copy slice rescoring z-score fuse × w intent | < 2 ms | E2E | Encode path excluded in pure Stage-2 harness; search SLA | < 15 ms | Stage-2 focused latency synthetic corpus, real PSMRetrievalEngine.search python benchmark harness.py End-to-end against a live backend ingest + encode query + search python vectorprism.py live-benchmark \ --checkpoint checkpoints/vectorprism.pt \ --documents data/documents.example.jsonl \ --backend memory --n-trials 20 --p95-budget-ms 15 Architecture path unchanged pillars : Text ─► Frozen 768d Encoder ─► MultiTaskProjectionAdapter ─► 1024d tensor │ ▼ pgvector / Qdrant dense HNSW + full tensor │ IntentClassifier ─► w intent + filters │ Stage 1: HNSW dense core slice + truth/anchor filters │ Stage 2: dense / rel / dis / hyp / causal scores → top-k Identity is Stage-1 gate only — not double-counted Channels are earned , not assumed. One channel at a time: python train.py --channel dense --data your pairs.jsonl \ --encoder sentence-transformers/all-mpnet-base-v2 --out checkpoints/vectorprism.pt python vectorprism.py eval --checkpoint checkpoints/vectorprism.pt \ --documents your docs.jsonl --eval your eval.jsonl \ --encoder sentence-transformers/all-mpnet-base-v2 Only after dense DoD: add causal / relational / hyperbolic / ... python train.py --channel causal --data your causal.jsonl \ --init checkpoints/vectorprism.pt --out checkpoints/vectorprism.pt See IMPLEMENTATION SPEC.md /insightitsGit/VectorPrism/blob/main/IMPLEMENTATION SPEC.md for phased Definitions of Done and the gap matrix. Building a regulated RAG stack, a multi-tenant AI SaaS retrieval plane, or a private compliance-aware knowledge system? Insight ITS works with enterprise architects on: - Custom multi-task adapter fine-tuning for your ontology / incident / audit corpora - Private compliance connectors bitemporal filters, calibrated epistemic truth, HITL review - Managed control planes for versioned re-ingest across pgvector & Qdrant fleets Talk to us - Email: insightits.info@gmail.com - Soft CTA: email subject RECOVER → insightits.info@gmail.com - GitHub Discussions: insightitsGit/VectorPrism https://github.com/insightitsGit/VectorPrism/discussions - Community: GitHub Discussions https://github.com/insightitsGit/VectorPrism/discussions no public Discord invite is published Apache License 2.0 — see LICENSE /insightitsGit/VectorPrism/blob/main/LICENSE or repository license metadata . If VectorPrism informs your research or production retrieval stack: @software{vectorprism2026, title = {VectorPrism: Positional Subspace Multiplexing for Intent-Gated Retrieval}, author = {Amin Parva}, year = {2026}, url = {https://github.com/insightitsGit/VectorPrism} } VectorPrism — six signals, one tensor, baseline storage cost, intent-gated speed. - Author: Amin Parva insightits.info@gmail.com mailto:insightits.info@gmail.com - Company: https://www.insightits.com https://www.insightits.com - GitHub: https://github.com/insightitsGit/VectorPrism https://github.com/insightitsGit/VectorPrism - PyPI: https://pypi.org/project/vectorprism/ https://pypi.org/project/vectorprism/ - Product page: https://www.insightits.com/products/vectorprism.html https://www.insightits.com/products/vectorprism.html