The Silent Vector Contamination Bug: Why Your Concurrent Embeddings Might Be Lying to You

TL;DR:If you run concurrent inference (e.g., via OpenVINOAsyncInferQueue

or custom threading) for text/code embeddings, your tests might show0 exceptions

and0 errors

, while silently returning embeddings belonging tootherinputs in the batch. Here is how we caught a subtle race condition using a cosine-similarity contamination test.

We use OpenVINO with an INT8 quantized E5-small model for in-process code embedding. To maximize throughput on multi-core CPUs, we set up an asynchronous inference queue (AsyncInferQueue

) with jobs=4

.

In standard unit testing, everything looked pristine:

None

values or zero-filled tensors were returnedHowever, during end-to-end RAG retrieval tests, we noticed weird semantic anomalies: searching for authentication logic would occasionally return chunks related to database migrations or UI components with unreasonably high confidence.

The root cause was a subtle race condition in callback/userdata mapping inside the async wrapper.

Because the inputs were processed concurrently across multiple execution streams, a shared user-data context wasn't strictly isolated per inference request. When Request A (auth.py

) and Request B (payment.py

) were scheduled back-to-back:

Standard assertion tests like assert output_vector is not None

or assert output_vector.shape == (384,)

passed 100% of the time. The pipeline was silently corrupting the vector store.

self._ov_results = {}

def _callback(request, userdata):
    self._ov_results[userdata] = request.get_tensor().data

The problem: userdata

was a simple integer counter (0, 1, 2, ...

) that reset between embed_batch

calls. When two calls overlapped, they wrote to the same dictionary keys.

To catch this reliably in CI, we wrote an explicit cross-contamination test designed for concurrent embedding queues.

import asyncio
import numpy as np
import pytest

def cosine_similarity(a: np.ndarray, b: np.ndarray) -> float:
    """Calculate cosine similarity between two vectors."""
    return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))

@pytest.mark.asyncio
async def test_async_embedder_no_cross_contamination(embedder):
    """Verify that concurrent embedding doesn't cross-contaminate vectors."""

    samples = {
        "auth": "def authenticate_user(username, password_hash): return verify_jwt(token)",
        "sql": "SELECT u.id, u.email FROM users u JOIN orders o ON u.id = o.user_id WHERE o.status = 'active'",
        "html": "<div class='flex-container'><span id='user-profile'>Profile View</span></div>",
        "rust": "pub fn allocate_buffer(size: usize) -> Result<Vec<u8>, MemoryError> { Vec::with_capacity(size) }"
    }

    baseline_vectors = {}
    for key, text in samples.items():
        baseline_vectors[key] = await embedder.embed_single_sync(text)

    async_tasks = []
    keys_order = list(samples.keys()) * 10  # 40 concurrent requests
    np.random.shuffle(keys_order)

    for key in keys_order:
        async_tasks.append(embedder.embed_async(samples[key]))

    async_results = await asyncio.gather(*async_tasks)

    for expected_key, async_vec in zip(keys_order, async_results):
        self_sim = cosine_similarity(async_vec, baseline_vectors[expected_key])
        assert self_sim > 0.98, (
            f"Contamination detected! Vector for '{expected_key}' drifted "
            f"(sim={self_sim:.2f}, expected >0.98)"
        )

        for other_key, other_vec in baseline_vectors.items():
            if other_key != expected_key:
                cross_sim = cosine_similarity(async_vec, other_vec)
                assert cross_sim < 0.6, (
                    f"Cross-talk detected between '{expected_key}' and "
                    f"'{other_key}' (sim={cross_sim:.2f}, expected <0.6)"
                )

Running this test on the unpatched queue revealed the contamination:

The unpatched queue showed 0 exceptions while vectors were completely swapped.

The fix was simple once we understood the problem:

self._ov_results = {}

def _callback(request, userdata):
    index, local_dict = userdata
    local_dict[index] = request.get_tensor().data

Each embed_batch

call now creates its own isolated dictionary. The callback writes to the call-specific dict, not a shared global. No locks needed — complete isolation by design.

** 0 exceptions ≠ Correctness.** Silent data corruption doesn't throw errors.

Valid shape ≠ Valid embedding. A (384,)

tensor with non-zero floats can represent the wrong input.

Write cross-contamination tests. If you use async inference queues or multi-threading for vector generation, verify that Vector X actually belongs to Input X under concurrent load.

Cosine similarity is your friend. A simple similarity check between concurrent outputs and sequential baselines catches contamination that no other test detects.

git clone https://github.com/ManSio/mscodebase-intelligence.git
cd mscodebase-intelligence

pip install -e ".[dev]"

pytest tests/test_ov_concurrent_embed.py -v

Has anyone else bumped into silent cross-talk in ONNX Runtime, OpenVINO, or TensorRT async queues? How do you validate thread isolation in your embedding pipelines?

Built with MSCodeBase Intelligence — an MCP server for codebase intelligence with incident memory and root cause prediction.