{"slug": "one-missed-test-case-cost-me-8-hours-how-i-built-a-zero-regression-memory-test", "title": "One Missed Test Case Cost Me 8 Hours — How I Built a Zero-Regression Memory Test Suite with Pytest + Docker", "summary": "A developer built an automated regression test suite using Pytest and Docker to prevent LLM memory storage failures after a production incident where a missed test case caused an 8-hour outage. The suite uses testcontainers-python to spin up real database instances like PostgreSQL with pgvector, replay historical data, and assert correct memory recall. The approach aims to catch regressions from storage migrations, model changes, or configuration updates within minutes.", "body_md": "I got woken up by an alert call at 2 a.m. – the production chatbot had suddenly developed “amnesia.” It was completely unable to recall order information we had just discussed that morning. I scrambled to investigate and discovered the root cause: we had migrated the memory storage from Redis to PostgreSQL + pgvector. The migration script looked fine, but one historical memory entry’s vector index hadn’t been properly built, causing the similarity recall to return empty results. Before going live, I had manually spot-checked 20 conversations as a regression check – and you guessed it, the one broken entry was the 21st I missed. From that moment on, I resolved that regression testing for LLM memory storage **must be automated**, and every change must replay historical memories. This post shares how I built that automated regression suite with Pytest + Docker, along with two near-fatal pitfalls along the way.\n\n“LLM memory storage” is the mechanism that stores chat histories, user preferences, summary snippets, etc., so the bot can retrieve them later via vector similarity or keyword search and inject them into prompts as context. Common implementations include Redis (with RediSearch/Vector modules), Milvus, PostgreSQL + pgvector, or even local FAISS files.\n\nThe biggest engineering pain point isn’t “can we store it?” but rather “does the system still remember after we upgrade?” Memory storage has many churn points: switching the underlying database, changing the embedding model, tweaking similarity thresholds, altering expiration policies, or even bumping a minor pgvector version. Any of these can cause missing memories, score deviations, or empty result lists – but traditional manual tests only cover a few happy paths and simply can’t cover weeks of accumulated historical data.\n\nThe conventional approaches aren’t great either: full end-to-end load testing with manual comparison is slow and irreproducible; mocking the storage layer to test business logic bypasses the exact vector retrieval logic that’s most fragile. What you really need is an environment that can quickly spin up a real storage instance, replay real historical data, and assert that every memory is correctly recalled. That’s exactly where Pytest + Docker shine.\n\nI considered several options:\n\nThe core idea is to use `testcontainers-python`\n\nto dynamically manage storage containers. During regression tests, you load a snapshot of historical data (sampled from production or generated golden files), then call the memory storage service’s query APIs and assert that the recalled IDs, scores, and ordering match expectations. If someone changes storage logic or upgrades a model, running this regression suite will tell you within 10 minutes whether old memories are broken.\n\nThis code solves “each test needs a fresh, disposable database.” I chose PostgreSQL + pgvector because it’s the most common and hides the most surprises.\n\n``` python\n# conftest.py\nimport pytest\nfrom testcontainers.postgres import PostgresContainer\nfrom testcontainers.redis import RedisContainer\nimport psycopg2\nimport redis\n\n# Custom pgvector container: the official postgres image doesn't include pgvector\nclass PostgresWithVector(PostgresContainer):\n    def __init__(self, image=\"pgvector/pgvector:pg16\"):\n        super().__init__(image=image)\n        self.with_env(\"POSTGRES_USER\", \"test\")\n        self.with_env(\"POSTGRES_PASSWORD\", \"test\")\n        self.with_env(\"POSTGRES_DB\", \"memory_test\")\n\n@pytest.fixture(scope=\"session\")\ndef pg_vector_store():\n    \"\"\"Start a pgvector container and return usable connection params\"\"\"\n    postgres = PostgresWithVector()\n    postgres.start()\n    # Get the dynamically mapped host port\n    host = postgres.get_container_host_ip()\n    port = postgres.get_exposed_port(5432)\n    dsn = f\"postgresql://test:test@{host}:{port}/memory_test\"\n    # Initialize schema and vector extension\n    conn = psycopg2.connect(dsn)\n    with conn.cursor() as cur:\n        cur.execute(\"CREATE EXTENSION IF NOT EXISTS vector;\")\n        cur.execute(\"\"\"\n            CREATE TABLE IF NOT EXISTS memory_blocks (\n                id TEXT PRIMARY KEY,\n                user_id TEXT,\n                content TEXT,\n                embedding vector(1536),\n                created_at TIMESTAMP DEFAULT NOW()\n            );\n            CREATE INDEX ON memory_blocks USING ivfflat (embedding vector_cosine_ops);\n        \"\"\")\n    conn.close()\n    yield {\"dsn\": dsn, \"host\": host, \"port\": port}\n    postgres.stop()\n\n@pytest.fixture(scope=\"session\")\ndef redis_memory_store():\n    \"\"\"Similarly, use Redis as an alternative storage for comparison tests\"\"\"\n    redis_container = RedisContainer(\"redis/redis-stack-server:latest\")\n    redis_container.start()\n    host = redis_container.get_container_host_ip\n```\n\n", "url": "https://wpnews.pro/news/one-missed-test-case-cost-me-8-hours-how-i-built-a-zero-regression-memory-test", "canonical_source": "https://dev.to/_eb7f2a654e97a60ae9f96e/one-missed-test-case-cost-me-8-hours-how-i-built-a-zero-regression-memory-test-suite-with-pytest-23o8", "published_at": "2026-07-19 01:05:26+00:00", "updated_at": "2026-07-19 01:27:28.749384+00:00", "lang": "en", "topics": ["large-language-models", "developer-tools", "mlops", "ai-infrastructure"], "entities": ["Pytest", "Docker", "PostgreSQL", "pgvector", "Redis", "testcontainers-python"], "alternates": {"html": "https://wpnews.pro/news/one-missed-test-case-cost-me-8-hours-how-i-built-a-zero-regression-memory-test", "markdown": "https://wpnews.pro/news/one-missed-test-case-cost-me-8-hours-how-i-built-a-zero-regression-memory-test.md", "text": "https://wpnews.pro/news/one-missed-test-case-cost-me-8-hours-how-i-built-a-zero-regression-memory-test.txt", "jsonld": "https://wpnews.pro/news/one-missed-test-case-cost-me-8-hours-how-i-built-a-zero-regression-memory-test.jsonld"}}