{"slug": "the-silent-vector-contamination-bug-why-your-concurrent-embeddings-might-be-to", "title": "The Silent Vector Contamination Bug: Why Your Concurrent Embeddings Might Be Lying to You", "summary": "A developer discovered a silent race condition in OpenVINO's AsyncInferQueue that caused concurrent embedding requests to return vectors belonging to other inputs. The bug, which passed standard unit tests, was caught by a cosine-similarity contamination test that verified vector identity under high concurrency.", "body_md": "TL;DR:If you run concurrent inference (e.g., via OpenVINO`AsyncInferQueue`\n\nor custom threading) for text/code embeddings, your tests might show`0 exceptions`\n\nand`0 errors`\n\n, while silently returning embeddings belonging tootherinputs in the batch. Here is how we caught a subtle race condition using a cosine-similarity contamination test.\n\nWe use OpenVINO with an INT8 quantized [E5-small](https://huggingface.co/keisuke-miyako/multilingual-e5-small-onnx-int8) model for in-process code embedding. To maximize throughput on multi-core CPUs, we set up an asynchronous inference queue (`AsyncInferQueue`\n\n) with `jobs=4`\n\n.\n\nIn standard unit testing, everything looked pristine:\n\n`None`\n\nvalues or zero-filled tensors were returnedHowever, during end-to-end RAG retrieval tests, we noticed weird semantic anomalies: searching for authentication logic would occasionally return chunks related to database migrations or UI components with unreasonably high confidence.\n\nThe root cause was a subtle race condition in callback/userdata mapping inside the async wrapper.\n\nBecause the inputs were processed concurrently across multiple execution streams, a shared user-data context wasn't strictly isolated per inference request. When Request A (`auth.py`\n\n) and Request B (`payment.py`\n\n) were scheduled back-to-back:\n\nStandard assertion tests like `assert output_vector is not None`\n\nor `assert output_vector.shape == (384,)`\n\npassed 100% of the time. The pipeline was silently corrupting the vector store.\n\n```\n# Before fix: shared results dict across concurrent calls\nself._ov_results = {}\n\ndef _callback(request, userdata):\n    # BUG: userdata is a global index (0, 1, 2, ...)\n    # Two concurrent calls reuse the same indices!\n    self._ov_results[userdata] = request.get_tensor().data\n```\n\nThe problem: `userdata`\n\nwas a simple integer counter (`0, 1, 2, ...`\n\n) that reset between `embed_batch`\n\ncalls. When two calls overlapped, they wrote to the same dictionary keys.\n\nTo catch this reliably in CI, we wrote an explicit **cross-contamination test** designed for concurrent embedding queues.\n\n``` python\nimport asyncio\nimport numpy as np\nimport pytest\n\ndef cosine_similarity(a: np.ndarray, b: np.ndarray) -> float:\n    \"\"\"Calculate cosine similarity between two vectors.\"\"\"\n    return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))\n\n@pytest.mark.asyncio\nasync def test_async_embedder_no_cross_contamination(embedder):\n    \"\"\"Verify that concurrent embedding doesn't cross-contaminate vectors.\"\"\"\n\n    # 1. Semantically distinct inputs\n    samples = {\n        \"auth\": \"def authenticate_user(username, password_hash): return verify_jwt(token)\",\n        \"sql\": \"SELECT u.id, u.email FROM users u JOIN orders o ON u.id = o.user_id WHERE o.status = 'active'\",\n        \"html\": \"<div class='flex-container'><span id='user-profile'>Profile View</span></div>\",\n        \"rust\": \"pub fn allocate_buffer(size: usize) -> Result<Vec<u8>, MemoryError> { Vec::with_capacity(size) }\"\n    }\n\n    # 2. Sequential baseline (ground truth)\n    baseline_vectors = {}\n    for key, text in samples.items():\n        baseline_vectors[key] = await embedder.embed_single_sync(text)\n\n    # 3. High-concurrency stress test with randomized queue order\n    async_tasks = []\n    keys_order = list(samples.keys()) * 10  # 40 concurrent requests\n    np.random.shuffle(keys_order)\n\n    for key in keys_order:\n        async_tasks.append(embedder.embed_async(samples[key]))\n\n    async_results = await asyncio.gather(*async_tasks)\n\n    # 4. Verify identity & cross-isolation via Cosine Similarity\n    for expected_key, async_vec in zip(keys_order, async_results):\n        # Self-similarity against ground truth must be ~1.0\n        self_sim = cosine_similarity(async_vec, baseline_vectors[expected_key])\n        assert self_sim > 0.98, (\n            f\"Contamination detected! Vector for '{expected_key}' drifted \"\n            f\"(sim={self_sim:.2f}, expected >0.98)\"\n        )\n\n        # Cross-similarity against distinct inputs must remain low\n        for other_key, other_vec in baseline_vectors.items():\n            if other_key != expected_key:\n                cross_sim = cosine_similarity(async_vec, other_vec)\n                assert cross_sim < 0.6, (\n                    f\"Cross-talk detected between '{expected_key}' and \"\n                    f\"'{other_key}' (sim={cross_sim:.2f}, expected <0.6)\"\n                )\n```\n\nRunning this test on the **unpatched** queue revealed the contamination:\n\n| Metric | Unpatched | Patched | Expected |\n|---|---|---|---|\n| Self-similarity (auth↔auth) |\n0.34 ❌ |\n0.99 ✅ | >0.98 |\n| Cross-similarity (auth↔sql) |\n0.98 ❌ |\n0.32 ✅ | <0.6 |\n| Exceptions thrown | 0 | 0 | 0 |\n| Zero tensors | 0 | 0 | 0 |\n\nThe unpatched queue showed **0 exceptions** while vectors were completely swapped.\n\nThe fix was simple once we understood the problem:\n\n``` python\n# After fix: isolated results per call\nself._ov_results = {}\n\ndef _callback(request, userdata):\n    # FIX: userdata is now (index, local_results_dict)\n    # Each embed_batch call creates its own dict\n    index, local_dict = userdata\n    local_dict[index] = request.get_tensor().data\n```\n\nEach `embed_batch`\n\ncall now creates its own isolated dictionary. The callback writes to the call-specific dict, not a shared global. No locks needed — complete isolation by design.\n\n** 0 exceptions ≠ Correctness.** Silent data corruption doesn't throw errors.\n\n**Valid shape ≠ Valid embedding.** A `(384,)`\n\ntensor with non-zero floats can represent the wrong input.\n\n**Write cross-contamination tests.** If you use async inference queues or multi-threading for vector generation, verify that Vector X actually belongs to Input X under concurrent load.\n\n**Cosine similarity is your friend.** A simple similarity check between concurrent outputs and sequential baselines catches contamination that no other test detects.\n\n```\n# Clone the repo\ngit clone https://github.com/ManSio/mscodebase-intelligence.git\ncd mscodebase-intelligence\n\n# Install dependencies\npip install -e \".[dev]\"\n\n# Run the contamination test\npytest tests/test_ov_concurrent_embed.py -v\n```\n\nHas anyone else bumped into silent cross-talk in ONNX Runtime, OpenVINO, or TensorRT async queues? How do you validate thread isolation in your embedding pipelines?\n\n*Built with MSCodeBase Intelligence — an MCP server for codebase intelligence with incident memory and root cause prediction.*", "url": "https://wpnews.pro/news/the-silent-vector-contamination-bug-why-your-concurrent-embeddings-might-be-to", "canonical_source": "https://dev.to/mansio/the-silent-vector-contamination-bug-why-your-concurrent-embeddings-might-be-lying-to-you-5fg7", "published_at": "2026-07-21 22:00:48+00:00", "updated_at": "2026-07-21 22:29:46.397784+00:00", "lang": "en", "topics": ["machine-learning", "developer-tools", "ai-infrastructure"], "entities": ["OpenVINO", "E5-small", "AsyncInferQueue"], "alternates": {"html": "https://wpnews.pro/news/the-silent-vector-contamination-bug-why-your-concurrent-embeddings-might-be-to", "markdown": "https://wpnews.pro/news/the-silent-vector-contamination-bug-why-your-concurrent-embeddings-might-be-to.md", "text": "https://wpnews.pro/news/the-silent-vector-contamination-bug-why-your-concurrent-embeddings-might-be-to.txt", "jsonld": "https://wpnews.pro/news/the-silent-vector-contamination-bug-why-your-concurrent-embeddings-might-be-to.jsonld"}}