From 30 Minutes to 3 Seconds: Automated LLM Memory Recall Testing with pytest + FAISS

At 1:30 a.m., the user group blew up: "Why did the AI forget my dietary restriction again?" I dragged myself out of bed and checked. The memory "no cilantro" was still in the memory store, but top-k recall just didn't include it. Manually comparing 20 memories took 30 minutes. As my eyelids grew heavy, it hit me: this kind of regression testing should have been automated long ago.

A typical LLM memory pipeline looks like this: conversation -> extract memory -> vectorize -> write to FAISS/vector store -> embed query -> top-k recall. The root causes of recall inconsistency are often subtle: unnormalized vectors, embedding model version drift, order changes during index rebuild, top-k truncation, or incorrect similarity threshold settings. Occasional production hiccups are only investigated after users complain, and running a few manual queries simply cannot prevent regressions. We need an automated verification suite that runs in seconds locally and on every commit.

I chose pytest as the test framework. Fixtures manage index lifecycle, parametrize covers query cases in bulk, and assertions are clear. The vector index is built in memory with FAISS and paired with deterministic embeddings—simulating real embeddings, replaceable with the OpenAI API—so every run is reproducible. Why not a real vector database? External dependencies are slow and state is hard to clean. Why not LangChain memory tests? Too black-box; you can only see the final answer and cannot locate problems at the vector layer. Plain unittest is not flexible enough. pytest + FAISS is the minimal reproducible unit that can pin recall problems exactly at the vector layer.

The first code block solves deterministic vectorization. Real embedding models are often uncontrollable and slow, so we generate fixed-dimension vectors from a hash function and apply L2 normalization, semantically approximating "same text, same vector". Production can seamlessly replace it with OpenAI embeddings.

import hashlib
import numpy as np
import faiss

DIM = 128

def text2vec(text: str, dim: int = DIM) -> np.ndarray:
    """确定性文本转向量，模拟 embedding。生产可替换为真实模型输出。"""
    digest = hashlib.sha256(text.encode("utf-8")).digest()
    arr = np.frombuffer(digest, dtype=np.uint8).astype(np.float32)
    vec = np.resize(arr, (dim,))
    norm = np.linalg.norm(vec)
    if norm > 0:
        vec = vec / norm
    return vec.astype(np.float32)

The second code block solves test isolation and index construction. Each test gets an independent FAISS index to avoid cross-test contamination. A dataclass manages memory records, and the fixture returns both the index and original data for convenient assertions.

import pytest
from dataclasses import dataclass

@dataclass
class MemoryRecord:
    id: str
    content: str
    metadata: dict = None

MEMORIES = [
    MemoryRecord("m1", "用户不吃香菜", {"user_id": "u1"}),
    MemoryRecord("m2", "用户对花生过敏", {"user_id": "u1"}),
    MemoryRecord("m3", "用户喜欢喝美式咖啡", {"user_id": "u1"}),
    MemoryRecord("m4", "用户是素食主义者", {"user_id": "u2"}),
]

@pytest.fixture(scope="function")
def memory_index():
    """每个测试独立构建索引，避免状态污染"""
    index = faiss.IndexFlatIP(DIM)  # 内积索引，要求向量已归一化
    vectors = np.zeros((len(MEMORIES), DIM), dtype=np.float32)
    for i, mem in enumerate(MEMORIES):
        vectors[i] = text2vec(mem.content)
    index.add(vectors)
    return index, MEMORIES

The third code block solves recall consistency assertions. Multiple queries are parametrized, asserting top-1 must hit the expected memory. It also verifies that recall results remain unchanged after rebuilding the index, covering the "redeploy" scenario.

python
@pytest.mark.parametrize("query,expected_id", [
    ("不吃香菜", "m1"),
    ("花生过敏", "m2"),
    ("美式咖啡", "m3"),
    ("素食", "m4"),
])
def test_recall_top1_consistency(memory_index, query, expected_id):
    index, memories = memory_index
    q_vec = text2vec(query).reshape(1, -1)
    scores, ids = index.search(q_vec, k=3)
    top1_id = memories[ids[0][0]].id
    assert top1_id == expected_id, f"召回不一致: 期望 {expected_id}, 实际 {top1_id}"
    assert -1.0 <= scores[0][0] <= 1.0

def test_rebuild_index_recall_consistent(memory_index):
    """模拟索引重建，验证同一批记忆重建后召回结果不变"""
    index, memories = memory_index
    new_index = faiss.IndexFlatIP(DIM)
    vectors = np.zeros((len(memories), DIM), dtype=np.float32)
    for