From 30 Minutes to 3 Seconds: Automated LLM Memory Recall Testing with pytest + FAISS

A developer built an automated regression testing suite for LLM memory recall using pytest and FAISS, reducing manual testing time from 30 minutes to 3 seconds. The suite uses deterministic embeddings and in-memory FAISS indexes to pin down recall inconsistencies at the vector layer, covering scenarios like top-k truncation and index rebuilds.

At 1:30 a.m., the user group blew up: "Why did the AI forget my dietary restriction again?" I dragged myself out of bed and checked. The memory "no cilantro" was still in the memory store, but top-k recall just didn't include it. Manually comparing 20 memories took 30 minutes. As my eyelids grew heavy, it hit me: this kind of regression testing should have been automated long ago. A typical LLM memory pipeline looks like this: conversation - extract memory - vectorize - write to FAISS/vector store - embed query - top-k recall. The root causes of recall inconsistency are often subtle: unnormalized vectors, embedding model version drift, order changes during index rebuild, top-k truncation, or incorrect similarity threshold settings. Occasional production hiccups are only investigated after users complain, and running a few manual queries simply cannot prevent regressions. We need an automated verification suite that runs in seconds locally and on every commit. I chose pytest as the test framework. Fixtures manage index lifecycle, parametrize covers query cases in bulk, and assertions are clear. The vector index is built in memory with FAISS and paired with deterministic embeddings—simulating real embeddings, replaceable with the OpenAI API—so every run is reproducible. Why not a real vector database? External dependencies are slow and state is hard to clean. Why not LangChain memory tests? Too black-box; you can only see the final answer and cannot locate problems at the vector layer. Plain unittest is not flexible enough. pytest + FAISS is the minimal reproducible unit that can pin recall problems exactly at the vector layer. The first code block solves deterministic vectorization. Real embedding models are often uncontrollable and slow, so we generate fixed-dimension vectors from a hash function and apply L2 normalization, semantically approximating "same text, same vector". Production can seamlessly replace it with OpenAI embeddings. python import hashlib import numpy as np import faiss DIM = 128 def text2vec text: str, dim: int = DIM - np.ndarray: """确定性文本转向量，模拟 embedding。生产可替换为真实模型输出。""" digest = hashlib.sha256 text.encode "utf-8" .digest arr = np.frombuffer digest, dtype=np.uint8 .astype np.float32 vec = np.resize arr, dim, L2 归一化，保证内积等于余弦相似度（IndexFlatIP 依赖这个前提） norm = np.linalg.norm vec if norm 0: vec = vec / norm return vec.astype np.float32 The second code block solves test isolation and index construction. Each test gets an independent FAISS index to avoid cross-test contamination. A dataclass manages memory records, and the fixture returns both the index and original data for convenient assertions. python import pytest from dataclasses import dataclass @dataclass class MemoryRecord: id: str content: str metadata: dict = None 测试用记忆库 MEMORIES = MemoryRecord "m1", "用户不吃香菜", {"user id": "u1"} , MemoryRecord "m2", "用户对花生过敏", {"user id": "u1"} , MemoryRecord "m3", "用户喜欢喝美式咖啡", {"user id": "u1"} , MemoryRecord "m4", "用户是素食主义者", {"user id": "u2"} , @pytest.fixture scope="function" def memory index : """每个测试独立构建索引，避免状态污染""" index = faiss.IndexFlatIP DIM 内积索引，要求向量已归一化 vectors = np.zeros len MEMORIES , DIM , dtype=np.float32 for i, mem in enumerate MEMORIES : vectors i = text2vec mem.content index.add vectors return index, MEMORIES The third code block solves recall consistency assertions. Multiple queries are parametrized, asserting top-1 must hit the expected memory. It also verifies that recall results remain unchanged after rebuilding the index, covering the "redeploy" scenario. python @pytest.mark.parametrize "query,expected id", "不吃香菜", "m1" , "花生过敏", "m2" , "美式咖啡", "m3" , "素食", "m4" , def test recall top1 consistency memory index, query, expected id : index, memories = memory index q vec = text2vec query .reshape 1, -1 scores, ids = index.search q vec, k=3 top1 id = memories ids 0 0 .id assert top1 id == expected id, f"召回不一致: 期望 {expected id}, 实际 {top1 id}" 额外断言得分范围，防止归一化失效 assert -1.0 <= scores 0 0 <= 1.0 def test rebuild index recall consistent memory index : """模拟索引重建，验证同一批记忆重建后召回结果不变""" index, memories = memory index new index = faiss.IndexFlatIP DIM vectors = np.zeros len memories , DIM , dtype=np.float32 for