{"slug": "show-hn-hadamask-encrypt-rag-metadata-keep-exact-vector-search", "title": "Show HN: Hadamask – Encrypt RAG metadata, keep exact vector search", "summary": "A project called hadamask released alpha software that applies a keyed orthogonal transform to embeddings and AES-256-GCM encryption to metadata so vector databases can run exact cosine, dot-product and L2 similarity search on protected data, according to its Show HN post. The library works behind one API with Qdrant, Weaviate, Milvus, Chroma, pgvector, Elasticsearch, OpenSearch, Redis and MongoDB Atlas, plus LangChain and LlamaIndex, and keeps master keys wrapped by HashiCorp Vault, AWS KMS, Google Cloud KMS or Azure Key Vault. The authors state the embedding transform is \"obfuscation with measured leakage, not encryption\": in experiments reported in Cernica & Popescu, Applied Sciences 2026, an attacker holding only protected vectors and public same-distribution data inferred document topics with 81–87% accuracy across four classes, and the software has not had an external cryptographic audit.", "body_md": "**Client-side protection for vector databases.**\nKeyed orthogonal obfuscation for embeddings, AES-256-GCM for metadata. Similarity search stays exact.\n\n[Quick start](#quick-start) · [Security model](#security-model) · [Store API](#the-store-api) · [Databases](#supported-databases) · [Key management](#key-management) · [Frameworks](#langchain-and-llamaindex) · [Design](#design)\n\nVector databases need the geometry of your embeddings to search them, so the vectors usually sit in the database exactly as your embedding model produced them, next to the documents' metadata. Anyone who gets a snapshot, a backup or an over-privileged database account gets both.\n\n**hadamask** protects that data before it leaves your application:\n\n- **Embeddings** go through a secret orthogonal transform. Queries get the same transform,\nso the database computes exactly the same cosine, dot-product and L2 scores as on the\nclear vectors. Your index, your database and your recall stay the same.\n- **Metadata and document text** are encrypted field by field with AES-256-GCM, with a fresh\nnonce per value and length padding. An amount of`100.00` and one of`9,999,999.99` produce\nciphertexts of the same length.\n- **Filters still work.** Fields you declare`blind` get keyed equality tokens. Fields you\ndeclare`plaintext` stay in clear and support ranges.\n- **Keys stay with you.** It is a library, not a proxy or a service. The master key can stay\nwrapped by HashiCorp Vault, AWS KMS, Google Cloud KMS or Azure Key Vault.\n\nIt works with **Qdrant, Weaviate, Milvus, Chroma, pgvector, Elasticsearch, OpenSearch, Redis\nand MongoDB Atlas**, behind one API, and with **LangChain** and **LlamaIndex**.\n\nThe method and its leakage are analysed in\n[*What a Secret Rotation Buys: Exact Retrieval and Measured Leakage in Vector Databases*](https://doi.org/10.3390/app162010259)\n(Cernica & Popescu, *Applied Sciences* 2026).\n\nImportant\n\n**What it is not.** The embedding transform is **obfuscation with measured leakage, not\nencryption**. It hides the coordinate system, but the relative geometry of your corpus\n(which documents are similar to which) stays visible. In the paper's experiments, an\nattacker holding only the protected vectors and public data from the same distribution\ninferred document topics with **81–87% accuracy** (four classes). Metadata *is*\nencrypted. Read the [security model](#security-model) before deploying. hadamask is alpha\nsoftware and has not had an external cryptographic audit.\n\nNote\n\n**What \"exact\" means.** Scores are mathematically identical to those on the clear vectors\n(up to float32 rounding). With an approximate index such as HNSW, the database returns\nwhat that index would return on the clear data: recall against exhaustive search is the\nengine's own, not hadamask's. In the [benchmark](#benchmarks), recall and the top-10 sets\nmatch the unprotected collections.\n\n```\nflowchart LR\n    subgraph trusted[\"Your application (trusted)\"]\n        A[Documents and queries] --> B[Embedding model]\n        B --> C[\"hadamask<br/>vectors: v ↦ Q·v<br/>metadata: AES-256-GCM\"]\n        K[(\"Master key<br/>(Vault / KMS / HSM)\")] -.-> C\n    end\n    C -- \"transformed vectors<br/>ciphertexts, tokens\" --> D[(\"Vector database<br/>any of 9 engines\")]\n    D -- \"ids, scores,<br/>ciphertexts\" --> C\n    C --> E[Decrypted results]\n```\n\n1. Your application embeds a document as usual.\n2. hadamask pads the vector to a power-of-two dimension and applies the keyed orthogonal map\n`Q = H·D2·P·H·D1` , built from Walsh–Hadamard transforms, secret sign flips and a secret\npermutation. Because`Q` is orthogonal,`⟨Qu, Qv⟩ = ⟨u, v⟩` .\n3. Each metadata field is serialized, padded, and encrypted with AES-256-GCM, or tokenized or left in clear, depending on your policy.\n4. The database indexes and searches what it receives, with no plugin and no configuration change. Queries get the same transform, so rankings and scores are those of the clear data.\n5. Results are decrypted in your process.\n\nThe threat is an attacker who obtains a **copy of the database** (a snapshot, a backup, a\nleaked dump or an over-privileged account) but not your application or your key.\n\n| What is stored | Protection | What an attacker with a snapshot still learns | \n|---|---|---|\n| Encrypted fields (default for every field) | AES-256-GCM, fresh 96-bit nonce per value, field name and key id bound as associated data | That the field exists. Short values reveal nothing about their length; long values reveal a power-of-two size class, or nothing with `fixed_lengths` | \n| Blind fields | Encrypted as above, plus an HMAC-SHA256 equality token | Which records share a value, and how often each value occurs | \n| Plaintext fields | None. This is your explicit choice | Everything | \n| Embeddings | Keyed orthogonal transform | **All pairwise similarities** (the Gram matrix): clusters, duplicates, nearest-neighbour graph, spectrum. Not the coordinates themselves | \n\nWhat the paper measures about the embedding leakage:\n\n- Search is exact: exhaustive top-10 sets are identical to clear search across three BEIR datasets and three encoders. Five of six database engines return identical sets; the sixth (Chroma) differs only within its own re-indexing variability.\n- If an attacker knows enough plaintext–vector pairs (as many as the dimension), the map is recovered exactly.\n- Without any pairs, public embeddings from the same distribution still support topic inference: median four-way topic accuracy of 0.81–0.87 in the paper's experiments.\n\n**Not covered:** compromise of the application host or the key; attackers who can insert\nchosen documents and later read a snapshot; query and access-pattern leakage; server-side\nvectorizers (the database must never receive clear text or clear vectors).\n\nUse hadamask where the threat is data at rest in a database you do not fully control. It\nraises the bar from \"read the dump\" to \"mount an inference attack, with auxiliary data,\nagainst a geometry-only view\". If you need the corpus geometry itself hidden, you need a\ncryptographic protocol or trusted hardware. See [SECURITY.md](https://github.com/cernica/hadamask/blob/main/SECURITY.md).\n\nInstall hadamask with the extra for your database:\n\n```\npip install \"hadamask[pgvector]\"            # or qdrant, weaviate, milvus, chroma, ...\npip install \"hadamask[qdrant,langchain]\"    # several at once\npip install \"hadamask[all]\"                 # every database, framework and key manager\n```\n\n`pip install hadamask` alone installs only the core (vector transform and metadata\nencryption, which depend on `numpy` and `cryptography`). Each extra adds the client SDK\nit needs, so a pgvector deployment does not pull in a dozen other database SDKs. If an\nextra is missing, the import tells you which one to install.\n\n| Extra | Adds | \n|---|---|\n| `qdrant`` weaviate``milvus`` chroma``pgvector` | vector database clients | \n| `elasticsearch`` opensearch``redis`` mongodb` | search engine and database clients | \n| `langchain`` llamaindex` | RAG framework integrations | \n| `vault`` aws``gcp`` azure` (or`kms` for all four) | key managers | \n| `all` | everything above | \n\nPython 3.10 or newer.\n\nThis runs as is, with an in-memory Qdrant ([`examples/quickstart.py`](https://github.com/cernica/hadamask/blob/main/examples/quickstart.py)):\n\n``` python\nimport numpy as np\nfrom qdrant_client import QdrantClient, models\n\nfrom hadamask import MasterKey, MetadataPolicy, Protector\nfrom hadamask.adapters.qdrant import QdrantStore\n\n# One key per tenant. \"client\" can be filtered on, \"year\" stays in clear,\n# every other field (\"text\", \"amount\", ...) is encrypted.\nprotector = Protector(MasterKey.generate(\"demo1\"), MetadataPolicy(plaintext=[\"year\"], blind=[\"client\"]))\n\nclient = QdrantClient(\":memory:\")\nstore = QdrantStore(client, protector, payload_schema={\"year\": models.PayloadSchemaType.INTEGER})\nstore.create_collection(\"docs\", dim=384, metric=\"cosine\")        # stored as 512-d\n\nembeddings = np.random.default_rng(0).standard_normal((3, 384))  # use your embedding model\nstore.upsert(\n    \"docs\",\n    ids=[1, 2, 3],\n    vectors=embeddings,\n    payloads=[\n        {\"text\": \"Wire transfer of 1,250,000 EUR to RO49AAAA1B31007593840000\", \"client\": \"ACME\", \"year\": 2026},\n        {\"text\": \"Loan agreement: principal 480,000 EUR at 4.1% fixed\", \"client\": \"ACME\", \"year\": 2025},\n        {\"text\": \"Quarterly audit report for the treasury department\", \"client\": \"Globex\", \"year\": 2026},\n    ],\n)\n\nfor hit in store.search(\"docs\", embeddings[0], k=2, where={\"client\": \"ACME\"}):\n    print(hit.id, round(hit.score, 3), hit.payload[\"text\"])\n```\n\nWhat Qdrant stores looks like this (abridged):\n\n```\npayload: {'text': 'hm1:demo1:0Zh9ab4mBNDEKBEwmzkCVnWsvFpnz2...', 'client': 'hm1:demo1:LXXVS92j-4jE...',\n          '_hmb_client': 'v5z-4r66z7jduraiUjjnKg', 'year': 2026}\nvector:  [-0.007, 0.037, 0.047, 0.082, -0.035, 0.01, ...]   (512 dims)\n```\n\nIn production, load the key from a key manager instead of generating it\n([Key management](#key-management)).\n\nA 256-bit secret plus a public `key_id` that is stored next to every ciphertext. Every other\nsecret is derived from it with domain-separated HMAC-SHA256: the transform's signs and\npermutation, the AES key, and the blind-token key.\n\n```\nMasterKey.generate(\"prod-2026\")                  # new random key\nMasterKey.from_env()                             # HADAMASK_MASTER_KEY (base64), HADAMASK_KEY_ID\nKeyring.from_wrapped(wrapper, wrapped_string)    # unwrapped by Vault / KMS (see below)\n```\n\nUse one key per tenant: different keys give unrelated coordinate systems.\n\nEvery payload field falls into one of three classes:\n\n| Class | Declared as | Stored as | Filters | \n|---|---|---|---|\n| encrypted | default | `hm1:<key_id>:<base64url(nonce‖ciphertext‖tag)>` | none | \n| blind | `blind=[...]` | ciphertext, plus `_hmb_<field>` token(s) | `$eq $ne $in $nin` | \n| plaintext | `plaintext=[...]` | the value itself | all operators, including ranges | \n\n```\nMetadataPolicy(\n    plaintext=[\"year\", \"department\"],\n    blind=[\"client_id\", \"tags\"],         # list values get one token per element\n    min_padded_length=64,                # every short value pads to 64 bytes\n    fixed_lengths={\"text\": 8192},        # this field always pads to exactly 8 KiB\n)\n```\n\nKeep `blind` and `plaintext` as small as possible. A blind field with few possible values\n(status, country) is open to frequency analysis.\n\nHolds a key and a policy and does all the cryptography. Adapters never see keys.\n\n```\nprotector = Protector(key, policy, previous_keys=[old_key])\n\nprotector.encode(vectors)           # (n, d) -> (n, D) float32; documents and queries alike\nprotector.decode(vectors, dim=384)  # back to native vectors\nprotector.encrypt_payload(payload)  # -> stored dict\nprotector.decrypt_payload(stored)   # -> clear dict (also reads previous keys)\nprotector.stored_dim(384)           # 512: create collections with this size\n```\n\n|  | `transform=\"hadamard\"` (default) | `transform=\"dense\"` | \n|---|---|---|\n| Map | `H·D2·P·H·D1` (paper, Eq. 3), structured,`O(d log d)` | Dense Haar-uniform orthogonal matrix, `O(d²)` | \n| Stored dimension | Next power of two: 384 → 512, 1536 → **2048** | **Native** : 1536 stays 1536 | \n| Storage overhead (1536-d, measured) | +34% | +2–5% | \n| pgvector HNSW with 1536-d embeddings | **Not possible** (pgvector limit: 2000 dims) | Works | \n| Client memory | Negligible | `8·d²` bytes per dimension: 19 MB for 1536, 75 MB for 3072 | \n| Security analysis | Structured key family; the paper's Haar theorems do not apply directly | Exactly the Haar ideal of the paper's Section 3.4 | \n\n```\nprotector = Protector(key, policy, transform=\"dense\")\n```\n\n**What the two transforms are.**\n\n- `hadamard` is the construction from the paper.`H` is the Walsh–Hadamard matrix: a\nfixed, public matrix of ±1 entries that can be applied in`O(d log d)` . The secret is the\nsign vectors`D1` ,`D2` and the permutation`P` . That speed comes from structure: only\npower-of-two sizes work, and every key selects a map from a small, highly structured family.\n- `dense` is a full`d × d` orthogonal matrix with no structure at all. It is drawn*uniformly at random among all rotations and reflections* of`d` -dimensional space:\nhadamask fills a matrix with Gaussian numbers derived from your key, orthogonalizes it\n(QR), and fixes the column signs. Mathematicians call that uniform distribution the*Haar\nmeasure* , which is why the paper calls it the Haar ideal. This has nothing to do with the\nHaar*wavelet* , and no Hadamard matrix is involved. The same key always gives the same\nmatrix.\n\n**Which one to use.**\n\n| Your situation | Use | \n|---|---|\n| Embedding dimension is a power of two (256, 512, 1024, 4096) | `hadamard` : no padding needed, smallest memory footprint | \n| 384, 768, 1536, 3072 or any other dimension | `dense` : avoids padding (+33% storage for 1536 → 2048) | \n| pgvector with HNSW and more than 1024 dimensions | `dense` (with`hadamard` , 1536-d vectors become 2048-d, above pgvector's 2000-dimension HNSW limit) | \n| You want the paper's formal results for the ideal transform to apply to your data | `dense` | \n| Very large dimensions (8192+) or memory-constrained clients | `hadamard` (`dense` keeps a`d × d` matrix: 537 MB at 8192) | \n| You want exactly the construction evaluated experimentally in the paper | `hadamard` | \n\nPick once per collection: vectors written with one transform cannot be searched with the\nother. To switch, `copy_to` a new collection whose store uses a `Protector` with the other\ntransform.\n\n`dense` does not make the attacks measured in the paper harder: the geometry leaks the\nsame way, and `d` known pairs recover either map. What it changes is that the paper's\nresults for the ideal transform (a snapshot reveals exactly the Gram matrix and nothing\nmore) describe your stored data directly, with no structured family left that a future\nattack could exploit. It also removes padding.\n\nBoth preserve scores exactly, and both leak the same relative geometry (the Gram matrix).\nPadding to a larger dimension adds **no** security: the data still spans a `d`-dimensional\nsubspace, so its rank reveals `d`, and its spectrum is unchanged (paper, Proposition 1). For\n1536-d or 3072-d embeddings, or any pgvector deployment, use `dense`. A collection must be\nwritten and queried with the same transform.\n\nEvery adapter implements the same interface (`hadamask.adapters.base.ProtectedStore`).\n\n```\nstore.create_collection(\"docs\", dim=1536, metric=\"cosine\")   # \"cosine\" | \"dot\" | \"l2\"\nstore.register_collection(\"docs\", dim=1536, metric=\"cosine\") # for collections created elsewhere\nstore.drop_collection(\"docs\")\nstore.upsert(\"docs\", ids, vectors, payloads, batch_size=500)   # any size, split into batches\nstore.update_payload(\"docs\", id, {\"status\": \"closed\"})          # merge and re-encrypt, vector kept\nstore.update_payload(\"docs\", id, new_payload, merge=False)      # replace\nstore.delete(\"docs\", [id1, id2])\nstore.delete(\"docs\", where={\"year\": {\"$lt\": 2020}})\n```\n\n`upsert` replaces records with the same id. The lengths of `ids`, `vectors` and `payloads`\nmust match, otherwise it raises instead of silently dropping records.\n\n```\nhits = store.search(\n    \"docs\", query_vector, k=10,\n    where={\"client_id\": {\"$in\": [\"C-17\", \"C-18\"]}, \"year\": {\"$gte\": 2024}},\n    offset=10,                   # paging\n    score_threshold=0.75,        # minimum similarity (maximum distance for l2)\n    fields=[\"text\", \"client_id\"],# decrypt only these fields\n    with_vectors=True,           # native vectors on each hit\n    hnsw_ef=256,                 # engine-specific options, see below\n)\nhit.id, hit.score, hit.payload, hit.vector\n\nstore.search_many(\"docs\", query_vectors, k=5)     # one result list per query\n```\n\n`hit.score` means the same thing on every database, and equals the value computed on the\nclear vectors:\n\n| Metric | `score` | \n|---|---|\n| `cosine` | cosine similarity (higher is closer) | \n| `dot` | inner product (higher is closer) | \n| `l2` | Euclidean distance, not squared (lower is closer) | \n\n```\nstore.get(\"docs\", ids, with_vectors=True, fields=[\"year\"])\nstore.count(\"docs\", where={\"client_id\": \"C-17\"})\nfor record in store.scroll(\"docs\", where={\"year\": 2026}, batch_size=500, with_vectors=True):\n    ...\n```\n\nMongoDB-style. Several fields in one dict are combined with AND.\n\n| Operator | Example | Plaintext | Blind | \n|---|---|---|---|\n| equality | `{\"client_id\": \"C-17\"}` or`{\"$eq\": ...}` | ✓ | ✓ | \n| `$ne` | `{\"status\": {\"$ne\": \"closed\"}}` | ✓ | ✓ | \n| `$in` ,`$nin` | `{\"client_id\": {\"$in\": [\"C-17\", \"C-18\"]}}` | ✓ | ✓ | \n| `$gt` ,`$gte` ,`$lt` ,`$lte` | `{\"year\": {\"$gte\": 2024, \"$lt\": 2027}}` | ✓ | — | \n| `$and` ,`$or` | `{\"$or\": [{\"client_id\": \"C-17\"}, {\"year\": 2026}]}` | ✓ | ✓ | \n\n- Filters on encrypted-only fields, and ranges on blind fields, raise `ValueError` before\nanything is sent: the database cannot evaluate them.\n- `$ne` and`$nin` also match records that do not have the field, as in MongoDB.\n- On a list-valued blind field, equality means \"the list contains this value\".\n\n```\nstore.copy_to(\"docs\", target_store, \"docs_v2\", where=None, batch_size=500)\n```\n\nRecords are decrypted with this store's key and re-protected with the target's. Use it to\nmove between databases, or to rotate keys ([Key rotation](#key-rotation)).\n\n| Database | Class | Tested against | Search options | \n|---|---|---|---|\n| Qdrant | `QdrantStore` | 1.19, Qdrant Cloud | `hnsw_ef` ,`exact` , other`SearchParams` | \n| Weaviate | `WeaviateStore` | 1.33, Weaviate Cloud 1.40 | — | \n| Milvus | `MilvusStore` | 2.6 | `ef` ,`nprobe` ,`search_list` ,`radius` ,`range_filter` ,`level` | \n| Chroma | `ChromaStore` | 1.5 | — | \n| pgvector | `PgVectorStore` | PostgreSQL 17, pgvector 0.8 | `ef_search` ,`iterative_scan` | \n| Elasticsearch | `ElasticsearchStore` | 8.17 | `num_candidates` | \n| OpenSearch | `OpenSearchStore` | 2.19 | `num_candidates` ,`ef_search` | \n| Redis Stack | `RedisStore` | 7.4 | `ef_runtime` | \n| MongoDB Atlas Vector Search | `MongoStore` | 8.0 (atlas-local) | `num_candidates` ,`exact` | \n\n``` python\nfrom hadamask.adapters.qdrant import QdrantStore             # QdrantStore(client, protector, payload_schema=...)\nfrom hadamask.adapters.weaviate import WeaviateStore         # WeaviateStore(client, protector, index_type=\"hnsw\")\nfrom hadamask.adapters.milvus import MilvusStore             # MilvusStore(client, protector, index_type=\"HNSW\")\nfrom hadamask.adapters.chroma import ChromaStore             # ChromaStore(client, protector)\nfrom hadamask.adapters.pgvector import PgVectorStore         # PgVectorStore(\"postgresql://...\", protector, hnsw=True)\nfrom hadamask.adapters.elasticsearch import ElasticsearchStore\nfrom hadamask.adapters.opensearch import OpenSearchStore\nfrom hadamask.adapters.redis import RedisStore               # RedisStore(client, protector, numeric_fields=[\"year\"])\nfrom hadamask.adapters.mongodb import MongoStore             # MongoStore(client[\"db\"], protector, exact=True)\n```\n\nEvery adapter passes the same contract test against a real server. The test covers recall\nagainst exact search, every filter operator checked against reference semantics, counts,\npaging, thresholds, projections, scroll, payload updates, `copy_to` under a second key,\ndeletion by filter, and a check that **only ciphertext reaches the database**. A second test\nchecks that `score` equals the clear value for all three metrics on all nine engines. In\nthese tests, recall@10 is 1.0 on every engine.\n\nPer-database notes, including index choices, quantization, and limits such as Redis ranges\nneeding `numeric_fields`, are in [docs/databases.md](https://github.com/cernica/hadamask/blob/main/docs/databases.md).\n\nNote\n\nNever configure a server-side vectorizer (Weaviate `text2vec-*`, Chroma embedding\nfunctions, and so on): it would receive clear text. The Weaviate adapters refuse such\ncollections.\n\nIf your code already uses the official SDKs, the wrappers keep the same method names.\n\n``` python\nfrom hadamask.adapters.qdrant import ProtectedQdrantClient\n\nqdrant = ProtectedQdrantClient(QdrantClient(url), protector)\nqdrant.create_collection(\"docs\", vectors_config=models.VectorParams(size=1536, distance=models.Distance.COSINE))\nqdrant.upsert(\"docs\", points=[models.PointStruct(id=1, vector=embedding, payload={\"text\": \"...\", \"client\": \"ACME\"})])\nqdrant.query_points(\"docs\", query=query_embedding, limit=5,\n                    query_filter=models.Filter(must=[qdrant.match(\"client\", \"ACME\")]))\npython\nfrom hadamask.adapters.weaviate import ProtectedWeaviateCollection\n\ndocs = ProtectedWeaviateCollection(client.collections.get(\"Docs\"), protector, dim=1536)\ndocs.data.insert({\"text\": \"...\", \"client\": \"ACME\"}, vector=embedding)\ndocs.query.near_vector(query_embedding, limit=5, filters=docs.match(\"client\", \"ACME\"))\n```\n\nMethods that would read or write data without protection (`set_payload`, `upload_points`,\n`batch`, `query.hybrid`, `prefetch`, ...) raise instead of passing through. `.unprotected`\ngives explicit access to the raw client.\n\nOne wrapper covers every vector store these frameworks support.\n\n``` python\nfrom hadamask.integrations.langchain import ProtectedVectorStore\nfrom langchain_qdrant import QdrantVectorStore\n\nstore = ProtectedVectorStore(\n    build_inner=lambda emb: QdrantVectorStore(client, \"docs\", embedding=emb),\n    embedding=OpenAIEmbeddings(),        # your real embeddings\n    protector=protector,\n)\nstore.add_texts(texts, metadatas=metadatas)      # page_content and metadata encrypted\nretriever = store.as_retriever(search_kwargs={\"k\": 5, \"filter\": {\"client\": \"ACME\"}})\npython\nfrom hadamask.integrations.llamaindex import ProtectedVectorStore\nfrom llama_index.core import StorageContext, VectorStoreIndex\nfrom llama_index.vector_stores.qdrant import QdrantVectorStore\n\nstore = ProtectedVectorStore(QdrantVectorStore(client=client, collection_name=\"docs\"), protector, embed_dim=1536)\nindex = VectorStoreIndex.from_documents(docs, storage_context=StorageContext.from_defaults(vector_store=store))\n```\n\nBoth protect vectors, text, metadata, and (in LlamaIndex) relationship metadata, which\nLlamaIndex otherwise copies in clear. Clear query text is never sent to the database. If\nyou only want vector protection, wrap your embedding model instead:\n`ProtectedEmbeddings` (LangChain) or `ProtectedEmbedding` (LlamaIndex). Never combine the\ntwo, or the vectors are transformed twice.\n\nThe master key can stay **wrapped** by a key manager (envelope encryption). Only the\nwrapped string goes into configuration, and the clear key exists only in your application's\nmemory.\n\n```\nhadamask keygen --key-id prod-2026 --wrap aws-kms --aws-kms-key-id alias/hadamask --env\n# HADAMASK_WRAPPED_KEY=hmk1:aws-kms:prod-2026:AQICAHh...\npython\nimport os\nfrom hadamask.kms import Keyring\nfrom hadamask.kms.aws import AwsKmsKeyWrapper\n\nkeyring = Keyring.from_wrapped(AwsKmsKeyWrapper(\"alias/hadamask\"), os.environ[\"HADAMASK_WRAPPED_KEY\"])\nprotector = keyring.protector(MetadataPolicy(blind=[\"client_id\"]))\n```\n\n| Provider | Wrapper | Extra | \n|---|---|---|\n| HashiCorp Vault Transit | `hadamask.kms.vault.VaultTransitKeyWrapper(client, key_name)` | `vault` | \n| HashiCorp Vault KV v2 (key stored as a secret) | `hadamask.kms.vault.VaultKVKeyStore(client, path)` | `vault` | \n| AWS KMS | `hadamask.kms.aws.AwsKmsKeyWrapper(key_id)` | `aws` | \n| Google Cloud KMS | `hadamask.kms.gcp.GcpKmsKeyWrapper(key_name)` | `gcp` | \n| Azure Key Vault / Managed HSM | `hadamask.kms.azure.AzureKeyVaultKeyWrapper(key_url)` | `azure` | \n\nWrapped keys are bound to their `key_id`: relabelling a wrapped key, or using it with\nanother provider, fails. The application needs only encrypt/decrypt permission on one\nkey. Setup, minimum permissions and policies for each provider are in\n[docs/key-management.md](https://github.com/cernica/hadamask/blob/main/docs/key-management.md).\n\n1. Create a new key, for example `prod-2027` .\n2. Build a protector that writes with the new key and still reads the old one:\n`Keyring(current=new, previous=(old,)).protector(policy)` .\n3. Copy each collection to one protected with the new key, then switch reads to it:\n`old_store.copy_to(\"docs\", new_store, \"docs_2027\")` . This rewrites vectors,\nciphertexts and blind tokens.\n\nRotation cannot erase what an earlier snapshot already exposed, and persistent ids link old\nand new snapshots (paper, Section 6.2). Without moving records, vectors can also be\nre-mapped with `OrthogonalTransform.rekey_to` (`Q_new·Q_oldᵀ`).\n\n```\nhadamask keygen [--key-id ID] [--env]                       # new raw key (store it in a secret manager)\nhadamask keygen --wrap {local,vault-transit,aws-kms,gcp-kms,azure-kv} [provider options] [--env]\nhadamask info 1536                                          # 1536 -> stored dimension 2048\n```\n\n| Piece | Choice | \n|---|---|\n| Transform | `Q = H·D2·P·H·D1` (paper, Eq. 3).`H` is the normalized Walsh–Hadamard transform (FWHT, O(D log D)),`D1` /`D2` are sign vectors,`P` is a permutation | \n| Randomness | HMAC-SHA256 streams of the master key with separate labels; Fisher–Yates with rejection sampling for `P` | \n| Dimensions | Native `d` is zero-padded to the next power of two`D` ; queries are padded the same way | \n| Dense transform | `transform=\"dense\"` : sign-corrected QR of a Gaussian matrix drawn from a ChaCha20 keystream under a derived key; exactly Haar-uniform, no padding | \n| Metadata | AES-256-GCM, 96-bit random nonce per value; associated data = key id and field name | \n| Length hiding | ISO/IEC 7816-4 padding to `min_padded_length` (64 bytes), then powers of two, or a fixed size per field | \n| Blind tokens | HMAC-SHA256 under a derived key, truncated to 128 bits | \n| Wrapped keys | `hmk1:<provider>:<key_id>:<blob>` ; payload bound to`hadamask/master-key/v1/<key_id>` , also passed as AAD where the KMS supports it | \n| Validation | Non-finite vectors, mismatched batch lengths, unknown filter operators and unknown search options all raise | \n\nOptional noise padding (`Protector.transform(d).encode(docs, noise_norm=...)`, paper\nSection 3.5.5) is available for dot-product collections only.\n\nApproximate indexes return the same sets as clear search, up to the engine's own ANN and\nfloat32 behaviour (paper, Section 5.7). Some engines quantize by default (Elasticsearch\n`int8_hnsw`); the adapters choose unquantized indexes explicitly.\n\n100,000 real documents (DBpedia, OpenAI ada-002 embeddings, 1536-d) and 1,000 held-out\nqueries. Each row compares a hadamask-protected collection with an unprotected one on the\nsame database, with the same HNSW settings (`m=16`, `ef=128`), the same data, and one\nsingle-threaded client measuring end-to-end latency. Full tables, including p95/p99, ingest\ntime and memory, are in [benchmarks/results/RESULTS.md](https://github.com/cernica/hadamask/blob/main/benchmarks/results/RESULTS.md).\n\n|  | Qdrant, unprotected | Qdrant, hadamask `dense` | pgvector, unprotected | pgvector, hadamask `dense` | \n|---|---|---|---|---|\n| Recall@10 vs exact search | 0.991 | 0.991 | 0.995 | 0.995 | \n| Query latency p50 | 3.63 ms | 4.01 ms (1.11×) | 5.69 ms | 6.34 ms (1.11×) | \n| Throughput, one client | 260 q/s | 235 q/s (0.91×) | 178 q/s | 154 q/s (0.86×) | \n| Storage per 1M vectors | 13.2 GB | 13.8 GB (1.05×) | 17.0 GB | 17.4 GB (1.02×) | \n| Server CPU per 1M queries | 2,338 CPU-s | 2,245 CPU-s (0.96×) | 4,218 CPU-s | 4,466 CPU-s (1.06×) | \n\nWith the default `hadamard` transform, the 2048-d stored vectors cost +34% storage on\nQdrant. On pgvector they cannot be HNSW-indexed at all, so search falls back to an exact\nscan: 212 ms instead of 5.7 ms. On the client, the `dense` transform runs at about 77,000\nvectors/s in batches (0.17 ms for a single query). Encrypting or decrypting a payload\n(title and text) takes 5–7 µs; padding and the envelope grow the mean payload from 404 to\n838 bytes.\n\nCaveats: one machine (Apple Silicon laptop, databases in Docker); one run per\nconfiguration; HNSW graphs built from different coordinates differ slightly, so\nprotected and unprotected top-10 sets overlap at 0.99 rather than 1.00, with the same\nrecall. Reproduce with `python benchmarks/run.py` ([instructions](https://github.com/cernica/hadamask/blob/main/benchmarks/README.md)).\n\n```\ndocker compose up -d                 # 9 databases + Vault, LocalStack (KMS), Lowkey Vault (Azure)\npip install -e \".[dev]\"\npytest tests/test_core.py            # unit tests, no services\npytest -m integration                # every database and key manager\npytest -m cloud                      # Qdrant Cloud and Weaviate Cloud (QDRANT_CLOUD_URL, ... in env)\n```\n\n- Hybrid search: protected BM25 / sparse vectors\n- Named and multiple vectors per record in the store API\n- Async API\n- Reproducible leakage benchmarks on real embeddings (attack demos)\n- Benchmarks for the other seven databases, repeated runs\n- External security review\n\n```\n@article{cernica2026secretrotation,\n  title   = {What a Secret Rotation Buys: Exact Retrieval and Measured Leakage in Vector Databases},\n  author  = {Cernica, Ionu{\\c{t}} and Popescu, Nirvana},\n  journal = {Applied Sciences},\n  volume  = {16},\n  number  = {20},\n  pages   = {10259},\n  year    = {2026},\n  doi     = {10.3390/app162010259}\n}\n```\n\n[Apache-2.0](https://github.com/cernica/hadamask/blob/main/LICENSE). Security issues: see [SECURITY.md](https://github.com/cernica/hadamask/blob/main/SECURITY.md).", "url": "https://wpnews.pro/news/show-hn-hadamask-encrypt-rag-metadata-keep-exact-vector-search", "canonical_source": "https://github.com/cernica/hadamask", "published_at": "2026-11-10 15:30:50+00:00", "updated_at": "2026-11-10 15:54:27.654322+00:00", "lang": "en", "topics": ["ai-safety", "ai-infrastructure", "large-language-models", "ai-tools"], "entities": ["hadamask", "Qdrant", "Weaviate", "Milvus", "Chroma", "pgvector", "Elasticsearch", "LangChain"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/show-hn-hadamask-encrypt-rag-metadata-keep-exact-vector-search", "markdown": "https://wpnews.pro/news/show-hn-hadamask-encrypt-rag-metadata-keep-exact-vector-search.md", "text": "https://wpnews.pro/news/show-hn-hadamask-encrypt-rag-metadata-keep-exact-vector-search.txt", "jsonld": "https://wpnews.pro/news/show-hn-hadamask-encrypt-rag-metadata-keep-exact-vector-search.jsonld"}}