The Silent Vector Contamination Bug: Why Your Concurrent Embeddings Might Be Lying to You

A developer discovered a silent race condition in OpenVINO's AsyncInferQueue that caused concurrent embedding requests to return vectors belonging to other inputs. The bug, which passed standard unit tests, was caught by a cosine-similarity contamination test that verified vector identity under high concurrency.

TL;DR:If you run concurrent inference e.g., via OpenVINO AsyncInferQueue or custom threading for text/code embeddings, your tests might show 0 exceptions and 0 errors , while silently returning embeddings belonging tootherinputs in the batch. Here is how we caught a subtle race condition using a cosine-similarity contamination test. We use OpenVINO with an INT8 quantized E5-small https://huggingface.co/keisuke-miyako/multilingual-e5-small-onnx-int8 model for in-process code embedding. To maximize throughput on multi-core CPUs, we set up an asynchronous inference queue AsyncInferQueue with jobs=4 . In standard unit testing, everything looked pristine: None values or zero-filled tensors were returnedHowever, during end-to-end RAG retrieval tests, we noticed weird semantic anomalies: searching for authentication logic would occasionally return chunks related to database migrations or UI components with unreasonably high confidence. The root cause was a subtle race condition in callback/userdata mapping inside the async wrapper. Because the inputs were processed concurrently across multiple execution streams, a shared user-data context wasn't strictly isolated per inference request. When Request A auth.py and Request B payment.py were scheduled back-to-back: Standard assertion tests like assert output vector is not None or assert output vector.shape == 384, passed 100% of the time. The pipeline was silently corrupting the vector store. Before fix: shared results dict across concurrent calls self. ov results = {} def callback request, userdata : BUG: userdata is a global index 0, 1, 2, ... Two concurrent calls reuse the same indices self. ov results userdata = request.get tensor .data The problem: userdata was a simple integer counter 0, 1, 2, ... that reset between embed batch calls. When two calls overlapped, they wrote to the same dictionary keys. To catch this reliably in CI, we wrote an explicit cross-contamination test designed for concurrent embedding queues. python import asyncio import numpy as np import pytest def cosine similarity a: np.ndarray, b: np.ndarray - float: """Calculate cosine similarity between two vectors.""" return float np.dot a, b / np.linalg.norm a np.linalg.norm b @pytest.mark.asyncio async def test async embedder no cross contamination embedder : """Verify that concurrent embedding doesn't cross-contaminate vectors.""" 1. Semantically distinct inputs samples = { "auth": "def authenticate user username, password hash : return verify jwt token ", "sql": "SELECT u.id, u.email FROM users u JOIN orders o ON u.id = o.user id WHERE o.status = 'active'", "html": "<div class='flex-container' <span id='user-profile' Profile View</span </div ", "rust": "pub fn allocate buffer size: usize - Result<Vec<u8 , MemoryError { Vec::with capacity size }" } 2. Sequential baseline ground truth baseline vectors = {} for key, text in samples.items : baseline vectors key = await embedder.embed single sync text 3. High-concurrency stress test with randomized queue order async tasks = keys order = list samples.keys 10 40 concurrent requests np.random.shuffle keys order for key in keys order: async tasks.append embedder.embed async samples key async results = await asyncio.gather async tasks 4. Verify identity & cross-isolation via Cosine Similarity for expected key, async vec in zip keys order, async results : Self-similarity against ground truth must be ~1.0 self sim = cosine similarity async vec, baseline vectors expected key assert self sim 0.98, f"Contamination detected Vector for '{expected key}' drifted " f" sim={self sim:.2f}, expected 0.98 " Cross-similarity against distinct inputs must remain low for other key, other vec in baseline vectors.items : if other key = expected key: cross sim = cosine similarity async vec, other vec assert cross sim < 0.6, f"Cross-talk detected between '{expected key}' and " f"'{other key}' sim={cross sim:.2f}, expected <0.6 " Running this test on the unpatched queue revealed the contamination: | Metric | Unpatched | Patched | Expected | |---|---|---|---| | Self-similarity auth↔auth | 0.34 ❌ | 0.99 ✅ | 0.98 | | Cross-similarity auth↔sql | 0.98 ❌ | 0.32 ✅ | <0.6 | | Exceptions thrown | 0 | 0 | 0 | | Zero tensors | 0 | 0 | 0 | The unpatched queue showed 0 exceptions while vectors were completely swapped. The fix was simple once we understood the problem: python After fix: isolated results per call self. ov results = {} def callback request, userdata : FIX: userdata is now index, local results dict Each embed batch call creates its own dict index, local dict = userdata local dict index = request.get tensor .data Each embed batch call now creates its own isolated dictionary. The callback writes to the call-specific dict, not a shared global. No locks needed — complete isolation by design. 0 exceptions ≠ Correctness. Silent data corruption doesn't throw errors. Valid shape ≠ Valid embedding. A 384, tensor with non-zero floats can represent the wrong input. Write cross-contamination tests. If you use async inference queues or multi-threading for vector generation, verify that Vector X actually belongs to Input X under concurrent load. Cosine similarity is your friend. A simple similarity check between concurrent outputs and sequential baselines catches contamination that no other test detects. Clone the repo git clone https://github.com/ManSio/mscodebase-intelligence.git cd mscodebase-intelligence Install dependencies pip install -e ". dev " Run the contamination test pytest tests/test ov concurrent embed.py -v Has anyone else bumped into silent cross-talk in ONNX Runtime, OpenVINO, or TensorRT async queues? How do you validate thread isolation in your embedding pipelines? Built with MSCodeBase Intelligence — an MCP server for codebase intelligence with incident memory and root cause prediction.