Vector Collections and Fixed Dimensions: 3 Node.js Deduplication Invariants

A developer outlined three invariants for Node.js vector collections used in media deduplication: every record in a collection must share one embedding model, one fixed dimension, and one declared chunking policy. The writeup argues that equal vector length is necessary but insufficient, since two models can both emit 768 numbers with different learned meanings per coordinate, and recommends creating a new collection and reindexing when the embedding model changes rather than inserting new-model vectors into the old collection. It includes a runtime guard that rejects dimension, model, and chunking mismatches before indexing.

TL;DR: Treat a vector collection as an embedding-space contract: every record uses one model, one fixed dimension, and one declared chunking policy. For media deduplication, create a new collection and reindex when the embedding model changes. Do not put a new model's vector into the old collection merely because both arrays have the same length. A collection is a set of same-shaped vectors plus their metadata. Fixed dimension is not a capacity knob. It is a property of the embedding model, and similarity is meaningful only among vectors produced within the same model's embedding space. That gives us an operational rule too: reject contract mismatches before indexing, then monitor freshness while a replacement collection fills. Shape first. A similarity function compares coordinates position by position. If one vector has 768 coordinates and another has 1,536, there is no complete coordinate pairing. A collection fixes the shape so every stored vector can participate in the same comparison. Equal length is necessary, but insufficient. Two models can both emit 768 numbers while assigning different learned meanings to coordinate 217. Their outputs produce a numeric score, but that score does not establish useful semantic similarity. The arrays share a shape, not a semantic coordinate system. Picture the system as a short pipeline. Before: article text enters model A, which emits a dimension-D vector; the collection compares it only with model-A vectors. After a model change: the text enters model B, lands in a new collection, and is compared only with model-B vectors. Traffic moves after the required corpus has been reindexed. Dimension describes shape, not room. A 1,536-dimensional collection does not hold twice as many articles as a 768-dimensional collection. Record count and dimension answer different questions. Suppose a publisher receives two versions of a wire story. Metadata such as storyId , desk , and publishedAt filters candidates; the vector represents the text used for semantic near-duplicate detection. Chunking belongs in the contract. Embedding a headline plus the first 400 words is a different policy from embedding every 400-word chunk, even if both use the same model and dimension. This runtime guard catches the category error early: js const apiKey = process.env.INFRAI API KEY; const baseUrl = process.env.INFRAI BASE URL; if apiKey || baseUrl { throw new Error "Set INFRAI API KEY and INFRAI BASE URL" ; } async function getDiscovery attempt = 0 : Promise<unknown { const response = await fetch new URL "/v1/discovery", baseUrl , { method: "GET", headers: { Authorization: Bearer ${apiKey} }, } ; if response.status === 429 && attempt < 4 { const retryAfter = Number response.headers.get "retry-after" ; const delayMs = Number.isFinite retryAfter ? retryAfter 1 000 : 250 2 attempt; await new Promise resolve = setTimeout resolve, delayMs ; return getDiscovery attempt + 1 ; } if response.ok { throw new Error discovery failed status=${response.status} body=${await response.text } ; } return response.json ; } type Contract = { collection: string; embeddingModel: string; dimension: number; chunkingPolicy: string; }; type StoryVector = { storyId: string; embeddingModel: string; chunkingPolicy: string; vector: number ; publishedAt: string; }; function assertContract contract: Contract, story: StoryVector : void { if story.vector.length == contract.dimension { throw new Error dimension mismatch expected=${contract.dimension} actual=${story.vector.length} , ; } if story.embeddingModel == contract.embeddingModel { throw new Error model mismatch expected=${contract.embeddingModel} actual=${story.embeddingModel} , ; } if story.chunkingPolicy == contract.chunkingPolicy { throw new Error chunking mismatch expected=${contract.chunkingPolicy} actual=${story.chunkingPolicy} , ; } } const contract: Contract = { collection: "wire-stories-model-a-v1", embeddingModel: "model-a", dimension: 768, chunkingPolicy: "headline-plus-first-400-words-v1", }; const incoming: StoryVector = { storyId: "wire-2026-10-03-0042", embeddingModel: "model-a", chunkingPolicy: "headline-plus-first-400-words-v1", vector: Array.from { length: 768 }, = 0 , publishedAt: "2026-10-03T08:30:00Z", }; await getDiscovery ; assertContract contract, incoming ; The discovery request lets the service inspect the live capability catalog before wiring a vector operation; it uses an environment-held key, an explicit method, status handling, and bounded exponential backoff for rate limits. The local guard is intentionally boring. Good. Count each rejection reason separately. A rise in dimension mismatch points toward shape configuration; model mismatch catches the subtler equal-length failure; chunking mismatch reveals content-policy drift. Freshness needs another signal: measure the interval between a source update and successful indexing in the active collection. Also report the reindex backlog by collection generation. Those measurements stop a clean model migration from quietly producing stale duplicate judgments. Set alerts from the newsroom's publication deadline and ingestion objective, not an invented universal threshold. For near-duplicate media records, collection design follows the unit that can be a duplicate. Whole-record embeddings are easier when records are short and similarly structured. Chunk embeddings can expose shared passages inside long articles, but create multiple candidates per story and need aggregation before the system labels two records duplicates. There is a trade-off. Small chunks expose copied paragraphs; overly small chunks lose enough context that a routine topic match can resemble a duplicate. Choose a policy, version it, and test it against labeled duplicate pairs. Metadata cannot repair scores from mixed policies. Media records also change. A correction or appended paragraph should be embedded under the same active contract and replace its current record. During migration, query generations deliberately: old vectors remain comparable with old vectors, and new with new. The replacement becomes authoritative only after its reindex reaches the completeness the application requires. Short version: new model, new collection, reindex . The mathematical boundary does not change with the vendor. The product question is where each system exposes it and how much surrounding infrastructure a team wants to own. | Option | Contract surface | Good fit | Boundary | |---|---|---|---| | Pinecone | Index creation includes vector dimension and metric. | Teams wanting a managed vector database with explicit index shape. | A model migration still requires a separate generation and reindex. | | Qdrant | Collection vector configuration declares size and distance. | Teams valuing an open-source vector database and collection-level configuration. | Self-hosting retains deployment and operational ownership. | | Weaviate | Collection configuration connects vectorization choices to objects. | Teams wanting object-oriented schema and integrated vectorization options. | Matching dimensions do not make different vectorizers compatible. | | Infrai | Search and RAG use a plain REST API under one key, without a required vendor SDK. | Teams standardizing varied backend calls behind HTTP. | Changing models still means a new collection and reindex. | No row wins universally. Pinecone's managed index, Qdrant's open-source deployment choices, and Weaviate's collection model imply different ownership. Infrai fits when a plain REST boundary and less client-library lifecycle work matter. That convenience does not alter the data contract. A portable control follows: keep embeddingModel , dimension , chunkingPolicy , and collectionGeneration in pipeline configuration. The vendor resource then enforces a contract that your team can inspect independently. They are still not comparable. Dimension says how many coordinates exist, not what the learned space means. This trap passes superficial validation because both payloads look structurally correct. Keep the model identifier in the write contract and reject a mismatch before upsert. Must every experiment trigger a full production reindex? No. An experiment can use a separate candidate collection holding a representative evaluation set. Compare duplicate judgments there first. A production switch does require a new collection and reindex, because the existing corpus must be embedded into the candidate model's space. Keep the phases distinct. During evaluation, report quality against labeled pairs. During migration, report indexed coverage and freshness. After cutover, report query outcomes and contract rejections. One average should not blur all three. The durable mental model is compact: a vector collection is a comparison domain, not a bag of arbitrary arrays. Fix its shape, model, and chunking policy. Replace that domain as a unit when the semantic contract changes.