Qdrant Recall Inconsistency: It Took 300 Test Runs to Discover the Index Wasn't Refreshed

An engineer discovered that Qdrant's asynchronous index refresh caused recall inconsistency in an AI agent's memory system, with recall rates dropping from 98% to 60%. They automated consistency testing with pytest and Qdrant's in-memory mode, revealing that upserts without wait=True lead to unstable query results. The fix involves using wait=True to force index readiness before querying.

At 2 AM, a user reported that the AI Agent suddenly forgot details of a project we discussed yesterday. I groggily opened Grafana and saw the memory recall rate had dropped from 98% to 60%. My first thought was the embedding model acting up again, but after digging through logs, I found a "time gap" between Qdrant writes and queries—data was upserted, yet queries intermittently returned nothing. This wasn't the first time, so I decided to automate recall consistency testing with pytest + Qdrant, and discovered the pitfalls were deeper than expected. The AI Agent's memory storage uses Qdrant to store vectors and payloads, with the core requirement of "immediately recallable after write." But in real scenarios, recall inconsistency is weird: local tests all pass, but CI occasionally fails; the same query, executed twice in a row, yields different results. The root cause is that Qdrant's write and index building are asynchronous—the upsert method returns without waiting for index refresh by default, so subsequent queries may read empty or partial results. A common workaround is manually adding time.sleep 2 in code, but that's not engineering practice, and the sleep duration is unpredictable—treating the symptom, not the cause. Worse, recall consistency also involves distance metrics and score threshold choices; a careless test assertion can mislead you. I chose pytest as the testing framework because its fixtures and parametrization are great for isolation and covering multiple scenarios. Qdrant's official Python client supports :memory: mode, so each test case gets a clean vector database instance fully isolated—no mocks needed; mocks can't reveal real index behavior. Why not unittest? Because fixture cleanup and parametrization are too verbose. Why not mock? Because we need to verify real Qdrant recall behavior; mocks only hide the async index problem. Architecturally, I define a qdrant client fixture in conftest.py ; each test automatically creates an independent collection and cleans it up after. Core tests cover three scenarios: single-point recall, batch-write recall, and recall with payload filters. This code solves test environment isolation, ensuring each test uses an independent Qdrant instance and collection. python conftest.py import pytest from qdrant client import QdrantClient from qdrant client.models import VectorParams, Distance @pytest.fixture def qdrant client : 内存模式：每个测试完全隔离，不落盘，速度快 client = QdrantClient ":memory:" yield client client.close @pytest.fixture def mem collection qdrant client : collection name = "agent memory test" 向量维度 3，余弦距离 qdrant client.create collection collection name=collection name, vectors config=VectorParams size=3, distance=Distance.COSINE return qdrant client, collection name This code exposes the first pitfall: querying immediately after a direct upsert gives unstable recall results. python test recall failure.py import pytest from qdrant client.models import PointStruct def test immediate recall without wait mem collection : client, collection = mem collection 写入一条向量，注意：没有 wait=True client.upsert collection name=collection, points= PointStruct id=1, vector= 0.1, 0.2, 0.3 , payload={"user": "alice"} 立即查询相同向量，期望能召回 result = client.query points collection name=collection, query= 0.1, 0.2, 0.3 , limit=1 这一行经常失败：result.points 为空 assert len result.points == 1 Running this test, it fails 6 out of 10 times with AssertionError: assert 0 == 1 . That's exactly the cause of online recall rate jitter. This code fixes the index refresh problem: wait=True forces the client to wait for the index to be ready before querying. python test recall fix.py import time from qdrant client.models import PointStruct, VectorParams, Distance def test immediate recall with wait mem collection : client, collection = mem collection 关键参数：wait=True，确保写入后索引刷新完成 client.upsert collection name=collection, points= PointStruct id=1, vector= 0.1, 0.2, 0.3 , payload={"user": "alice"} , wait=True 官方文档没明说，但这是解决异步索引的关键 现在查询稳定返回结果 result = client.query points collection name=collection, query= 0.1, 0.2, 0.3 , limit=1 assert len result.points == 1 assert result.points 0 .id == 1 还可以进一步验证 payload 一致性 assert result.points 0 .payload "user" == "alice" If you don't want to add wait=True to every call, you can...