One Missed Test Case Cost Me 8 Hours — How I Built a Zero-Regression Memory Test Suite with Pytest + Docker

I got woken up by an alert call at 2 a.m. – the production chatbot had suddenly developed “amnesia.” It was completely unable to recall order information we had just discussed that morning. I scrambled to investigate and discovered the root cause: we had migrated the memory storage from Redis to PostgreSQL + pgvector. The migration script looked fine, but one historical memory entry’s vector index hadn’t been properly built, causing the similarity recall to return empty results. Before going live, I had manually spot-checked 20 conversations as a regression check – and you guessed it, the one broken entry was the 21st I missed. From that moment on, I resolved that regression testing for LLM memory storage must be automated, and every change must replay historical memories. This post shares how I built that automated regression suite with Pytest + Docker, along with two near-fatal pitfalls along the way.

“LLM memory storage” is the mechanism that stores chat histories, user preferences, summary snippets, etc., so the bot can retrieve them later via vector similarity or keyword search and inject them into prompts as context. Common implementations include Redis (with RediSearch/Vector modules), Milvus, PostgreSQL + pgvector, or even local FAISS files.

The biggest engineering pain point isn’t “can we store it?” but rather “does the system still remember after we upgrade?” Memory storage has many churn points: switching the underlying database, changing the embedding model, tweaking similarity thresholds, altering expiration policies, or even bumping a minor pgvector version. Any of these can cause missing memories, score deviations, or empty result lists – but traditional manual tests only cover a few happy paths and simply can’t cover weeks of accumulated historical data.

The conventional approaches aren’t great either: full end-to-end load testing with manual comparison is slow and irreproducible; mocking the storage layer to test business logic bypasses the exact vector retrieval logic that’s most fragile. What you really need is an environment that can quickly spin up a real storage instance, replay real historical data, and assert that every memory is correctly recalled. That’s exactly where Pytest + Docker shine.

I considered several options:

The core idea is to use testcontainers-python

to dynamically manage storage containers. During regression tests, you load a snapshot of historical data (sampled from production or generated golden files), then call the memory storage service’s query APIs and assert that the recalled IDs, scores, and ordering match expectations. If someone changes storage logic or upgrades a model, running this regression suite will tell you within 10 minutes whether old memories are broken.

This code solves “each test needs a fresh, disposable database.” I chose PostgreSQL + pgvector because it’s the most common and hides the most surprises.

import pytest
from testcontainers.postgres import PostgresContainer
from testcontainers.redis import RedisContainer
import psycopg2
import redis

class PostgresWithVector(PostgresContainer):
    def __init__(self, image="pgvector/pgvector:pg16"):
        super().__init__(image=image)
        self.with_env("POSTGRES_USER", "test")
        self.with_env("POSTGRES_PASSWORD", "test")
        self.with_env("POSTGRES_DB", "memory_test")

@pytest.fixture(scope="session")
def pg_vector_store():
    """Start a pgvector container and return usable connection params"""
    postgres = PostgresWithVector()
    postgres.start()
    host = postgres.get_container_host_ip()
    port = postgres.get_exposed_port(5432)
    dsn = f"postgresql://test:test@{host}:{port}/memory_test"
    conn = psycopg2.connect(dsn)
    with conn.cursor() as cur:
        cur.execute("CREATE EXTENSION IF NOT EXISTS vector;")
        cur.execute("""
            CREATE TABLE IF NOT EXISTS memory_blocks (
                id TEXT PRIMARY KEY,
                user_id TEXT,
                content TEXT,
                embedding vector(1536),
                created_at TIMESTAMP DEFAULT NOW()
            );
            CREATE INDEX ON memory_blocks USING ivfflat (embedding vector_cosine_ops);
        """)
    conn.close()
    yield {"dsn": dsn, "host": host, "port": port}
    postgres.stop()

@pytest.fixture(scope="session")
def redis_memory_store():
    """Similarly, use Redis as an alternative storage for comparison tests"""
    redis_container = RedisContainer("redis/redis-stack-server:latest")
    redis_container.start()
    host = redis_container.get_container_host_ip