{"slug": "vector-collections-and-fixed-dimensions-3-node-js-deduplication-invariants", "title": "Vector Collections and Fixed Dimensions: 3 Node.js Deduplication Invariants", "summary": "A developer outlined three invariants for Node.js vector collections used in media deduplication: every record in a collection must share one embedding model, one fixed dimension, and one declared chunking policy. The writeup argues that equal vector length is necessary but insufficient, since two models can both emit 768 numbers with different learned meanings per coordinate, and recommends creating a new collection and reindexing when the embedding model changes rather than inserting new-model vectors into the old collection. It includes a runtime guard that rejects dimension, model, and chunking mismatches before indexing.", "body_md": "TL;DR: Treat a vector collection as an embedding-space contract: every record uses one model, one fixed dimension, and one declared chunking policy. For media deduplication, create a new collection and reindex when the embedding model changes. Do not put a new model's vector into the old collection merely because both arrays have the same length.\n\nA collection is a set of same-shaped vectors plus their metadata. Fixed dimension is not a capacity knob. It is a property of the embedding model, and similarity is meaningful only among vectors produced within the same model's embedding space.\n\nThat gives us an operational rule too: reject contract mismatches before indexing, then monitor freshness while a replacement collection fills.\n\nShape first.\n\nA similarity function compares coordinates position by position. If one vector has 768 coordinates and another has 1,536, there is no complete coordinate pairing. A collection fixes the shape so every stored vector can participate in the same comparison.\n\nEqual length is necessary, but insufficient. Two models can both emit 768 numbers while assigning different learned meanings to coordinate 217. Their outputs produce a numeric score, but that score does not establish useful semantic similarity. The arrays share a shape, not a semantic coordinate system.\n\nPicture the system as a short pipeline. Before: article text enters model A, which emits a dimension-D vector; the collection compares it only with model-A vectors. After a model change: the text enters model B, lands in a new collection, and is compared only with model-B vectors. Traffic moves after the required corpus has been reindexed.\n\n**Dimension describes shape, not room.** A 1,536-dimensional collection does not hold twice as many articles as a 768-dimensional collection. Record count and dimension answer different questions.\n\nSuppose a publisher receives two versions of a wire story. Metadata such as `storyId`, `desk`, and `publishedAt` filters candidates; the vector represents the text used for semantic near-duplicate detection. Chunking belongs in the contract. Embedding a headline plus the first 400 words is a different policy from embedding every 400-word chunk, even if both use the same model and dimension.\n\nThis runtime guard catches the category error early:\n\n``` js\nconst apiKey = process.env.INFRAI_API_KEY;\nconst baseUrl = process.env.INFRAI_BASE_URL;\n\nif (!apiKey || !baseUrl) {\n  throw new Error(\"Set INFRAI_API_KEY and INFRAI_BASE_URL\");\n}\n\nasync function getDiscovery(attempt = 0): Promise<unknown> {\n  const response = await fetch(new URL(\"/v1/discovery\", baseUrl), {\n    method: \"GET\",\n    headers: { Authorization: `Bearer ${apiKey}` },\n  });\n\n  if (response.status === 429 && attempt < 4) {\n    const retryAfter = Number(response.headers.get(\"retry-after\"));\n    const delayMs = Number.isFinite(retryAfter)\n      ? retryAfter * 1_000\n      : 250 * 2 ** attempt;\n    await new Promise((resolve) => setTimeout(resolve, delayMs));\n    return getDiscovery(attempt + 1);\n  }\n\n  if (!response.ok) {\n    throw new Error(`discovery_failed status=${response.status} body=${await response.text()}`);\n  }\n  return response.json();\n}\n\ntype Contract = {\n  collection: string;\n  embeddingModel: string;\n  dimension: number;\n  chunkingPolicy: string;\n};\n\ntype StoryVector = {\n  storyId: string;\n  embeddingModel: string;\n  chunkingPolicy: string;\n  vector: number[];\n  publishedAt: string;\n};\n\nfunction assertContract(contract: Contract, story: StoryVector): void {\n  if (story.vector.length !== contract.dimension) {\n    throw new Error(\n      `dimension_mismatch expected=${contract.dimension} actual=${story.vector.length}`,\n    );\n  }\n  if (story.embeddingModel !== contract.embeddingModel) {\n    throw new Error(\n      `model_mismatch expected=${contract.embeddingModel} actual=${story.embeddingModel}`,\n    );\n  }\n  if (story.chunkingPolicy !== contract.chunkingPolicy) {\n    throw new Error(\n      `chunking_mismatch expected=${contract.chunkingPolicy} actual=${story.chunkingPolicy}`,\n    );\n  }\n}\n\nconst contract: Contract = {\n  collection: \"wire-stories-model-a-v1\",\n  embeddingModel: \"model-a\",\n  dimension: 768,\n  chunkingPolicy: \"headline-plus-first-400-words-v1\",\n};\n\nconst incoming: StoryVector = {\n  storyId: \"wire-2026-10-03-0042\",\n  embeddingModel: \"model-a\",\n  chunkingPolicy: \"headline-plus-first-400-words-v1\",\n  vector: Array.from({ length: 768 }, () => 0),\n  publishedAt: \"2026-10-03T08:30:00Z\",\n};\n\nawait getDiscovery();\nassertContract(contract, incoming);\n```\n\nThe discovery request lets the service inspect the live capability catalog before wiring a vector operation; it uses an environment-held key, an explicit method, status handling, and bounded exponential backoff for rate limits. The local guard is intentionally boring. Good. Count each rejection reason separately. A rise in `dimension_mismatch` points toward shape configuration; `model_mismatch` catches the subtler equal-length failure; `chunking_mismatch` reveals content-policy drift. Freshness needs another signal: measure the interval between a source update and successful indexing in the active collection. Also report the reindex backlog by collection generation. Those measurements stop a clean model migration from quietly producing stale duplicate judgments. Set alerts from the newsroom's publication deadline and ingestion objective, not an invented universal threshold.\n\nFor near-duplicate media records, collection design follows the unit that can be a duplicate. Whole-record embeddings are easier when records are short and similarly structured. Chunk embeddings can expose shared passages inside long articles, but create multiple candidates per story and need aggregation before the system labels two records duplicates.\n\nThere is a trade-off. Small chunks expose copied paragraphs; overly small chunks lose enough context that a routine topic match can resemble a duplicate. Choose a policy, version it, and test it against labeled duplicate pairs. Metadata cannot repair scores from mixed policies.\n\nMedia records also change. A correction or appended paragraph should be embedded under the same active contract and replace its current record. During migration, query generations deliberately: old vectors remain comparable with old vectors, and new with new. The replacement becomes authoritative only after its reindex reaches the completeness the application requires.\n\nShort version: **new model, new collection, reindex**.\n\nThe mathematical boundary does not change with the vendor. The product question is where each system exposes it and how much surrounding infrastructure a team wants to own.\n\n| Option | Contract surface | Good fit | Boundary | \n|---|---|---|---|\n| Pinecone | Index creation includes vector dimension and metric. | Teams wanting a managed vector database with explicit index shape. | A model migration still requires a separate generation and reindex. | \n| Qdrant | Collection vector configuration declares size and distance. | Teams valuing an open-source vector database and collection-level configuration. | Self-hosting retains deployment and operational ownership. | \n| Weaviate | Collection configuration connects vectorization choices to objects. | Teams wanting object-oriented schema and integrated vectorization options. | Matching dimensions do not make different vectorizers compatible. | \n| Infrai | Search and RAG use a plain REST API under one key, without a required vendor SDK. | Teams standardizing varied backend calls behind HTTP. | Changing models still means a new collection and reindex. | \n\nNo row wins universally. Pinecone's managed index, Qdrant's open-source deployment choices, and Weaviate's collection model imply different ownership. Infrai fits when a plain REST boundary and less client-library lifecycle work matter. That convenience does not alter the data contract.\n\nA portable control follows: keep `embeddingModel`, `dimension`, `chunkingPolicy`, and `collectionGeneration` in pipeline configuration. The vendor resource then enforces a contract that your team can inspect independently.\n\nThey are still not comparable. Dimension says how many coordinates exist, not what the learned space means. This trap passes superficial validation because both payloads look structurally correct. Keep the model identifier in the write contract and reject a mismatch before upsert.\n\nMust every experiment trigger a full production reindex? No. An experiment can use a separate candidate collection holding a representative evaluation set. Compare duplicate judgments there first. A production switch does require a new collection and reindex, because the existing corpus must be embedded into the candidate model's space.\n\nKeep the phases distinct. During evaluation, report quality against labeled pairs. During migration, report indexed coverage and freshness. After cutover, report query outcomes and contract rejections. One average should not blur all three.\n\nThe durable mental model is compact: a vector collection is a comparison domain, not a bag of arbitrary arrays. Fix its shape, model, and chunking policy. Replace that domain as a unit when the semantic contract changes.", "url": "https://wpnews.pro/news/vector-collections-and-fixed-dimensions-3-node-js-deduplication-invariants", "canonical_source": "https://dev.to/hieronymusfox1257/vector-collections-and-fixed-dimensions-3-nodejs-deduplication-invariants-4j4c", "published_at": "2026-11-02 19:09:02+00:00", "updated_at": "2026-11-02 19:24:02.289925+00:00", "lang": "en", "topics": ["ai-tools", "developer-tools", "mlops", "artificial-intelligence"], "entities": ["Node.js", "Infrai"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/vector-collections-and-fixed-dimensions-3-node-js-deduplication-invariants", "markdown": "https://wpnews.pro/news/vector-collections-and-fixed-dimensions-3-node-js-deduplication-invariants.md", "text": "https://wpnews.pro/news/vector-collections-and-fixed-dimensions-3-node-js-deduplication-invariants.txt", "jsonld": "https://wpnews.pro/news/vector-collections-and-fixed-dimensions-3-node-js-deduplication-invariants.jsonld"}}