{"slug": "taking-over-llm-memory-store-testing-with-pytest-90-fewer-state-inconsistencies", "title": "Taking Over LLM Memory Store Testing with Pytest: 90% Fewer State Inconsistencies", "summary": "A developer at an AI startup built a Pytest-based test suite to automatically verify state consistency in LLM memory stores, reducing state inconsistencies by 90%. The suite simulates concurrent writes, compaction, and reads across Redis, vector databases, and PostgreSQL, using fixtures and parametrize to catch race conditions and partial failures that manual testing missed.", "body_md": "At 2:17 a.m., a Slack message exploded with user feedback: “Your AI assistant acts like it has amnesia—it just stored meeting minutes, but when I ask again, it says nothing was saved.” I dug into the logs and found that the Redis write was acknowledged, but the entry in the vector store was empty. This “intermittent amnesia” had already pissed off the user for the third time. Previously, we’d just manually curl a few conversation turns and call that a test pass—clearly that approach was broken. I was forced to set a rule the next morning: **All state‑consistency logic for memory storage must be automatically verified by Pytest, or it’s not allowed to land on main.**\n\nAn LLM’s “memory store” typically isn’t a single backend: short‑term memory lives in a Redis List or an in‑memory ring buffer, long‑term memory gets embedded and pushed to a vector database (Qdrant/Milvus), and there’s often a raw text copy in PostgreSQL. During a single `save_memory`\n\ncall, if you just write to Redis and PG synchronously and then push to the vector store asynchronously, there’s a window where Redis has already returned but the vector store hasn’t persisted—and a read request during that window can get incomplete history.\n\nEven nastier is “memory compaction.” When the conversation length exceeds a threshold, you trigger summary generation while also cleaning up original messages and updating summary vectors. If any step fails, you end up with both “compacted” and leftover original messages coexisting, leading to duplicate or contradictory content.\n\nThe usual remedy is adding a manual `verify`\n\nstep or running reconciliation scripts in production to sweep up dirty data. This has three fatal flaws:\n\nSo I decided to bake state‑consistency verification directly into a Pytest test suite and let CI do the heavy lifting.\n\nIn terms of tooling, the test framework was non‑negotiable: Pytest. The reasons are simple—fixtures elegantly manage the lifecycle of multiple storage backends, and `parametrize`\n\ncan cover dozens of failure combinations in one go.\n\nWhy not the seemingly obvious unit tests?\n\n`pytest-docker`\n\nor simply `testcontainers-python`\n\nso each test case gets isolated, fresh storage instances that are automatically destroyed after the run.The architectural idea is straightforward: each test case follows a pattern of **concurrent writes + multiple reads + eventual assertions**. We simulate two coroutines simultaneously writing messages for the same user, then trigger a compaction signal, and finally poll all backends with an `assert_eventually`\n\nstyle until they are consistent. No guessing with `time.sleep(3)`\n\n; we rely on `tenacity`\n\n’s `retry`\n\nor `asyncio.wait_for`\n\nfor timed conditional waits.\n\nI also explicitly modeled the memory store interface as a `Protocol`\n\n, so the test suite doesn’t just cover the current implementation—if we swap out the vector database or caching layer later, the exact same test cases can run with zero changes.\n\nThis code answers the question “what interface do we actually test?” by abstracting the production‑level memory storage into a `MemoryStoreProtocol`\n\n. All subsequent tests depend only on this protocol.\n\n``` python\nfrom typing import Protocol, List, Dict, Any, runtime_checkable\n\n@runtime_checkable\nclass MemoryStoreProtocol(Protocol):\n    \"\"\"大模型记忆存储必须实现的接口\"\"\"\n    async def append_message(self, user_id: str, role: str, content: str) -> None:\n        \"\"\"追加一条对话消息到短期记忆\"\"\"\n        ...\n\n    async def get_history(self, user_id: str, limit: int = 10) -> List[Dict[str, Any]]:\n        \"\"\"按时间顺序返回最近的消息列表\"\"\"\n        ...\n\n    async def compact(self, user_id: str) -> None:\n        \"\"\"触发记忆压缩：生成摘要并清理原始短期消息\"\"\"\n        ...\n\n    async def consistent_state(self, user_id: str) -> bool:\n        \"\"\"检查所有后端数据是否一致（用于测试断言）\"\"\"\n        ...\n```\n\nTo make the tests meaningful, I deliberately wrote a “buggy” implementation called `FragileMemoryStore`\n\n—it uses a Python `list`\n\nin place of Redis, a `dict`\n\nin place of the vector store, and randomly `await asyncio.sleep(0)`\n\nto simulate async delays. This code is the target we’re shooting at.\n\n``` python\nimport asyncio\nimport random\nfrom dataclasses import dataclass, field\n\n@dataclass\nclass FragileMemoryStore:\n    # 模拟三层存储\n    short_term: Dict[str, List[dict]] = field(default_factory=dict)   # Redis List\n    vector: Dict[str, List[str]] = field(default_factory=dict)       # 向量库\n    raw_pg: Dict[str, List[dict]] = field(default_factory=dict)      # PG 原始表\n\n    async def append_message(self, user_id: str, role: str, content: str) -> None:\n        msg = {\"role\": role, \"content\": content}\n        # 1. 先写短期缓存（同步写）\n        self.short_term.setdefault(user_id, []).append(msg)\n        # 2. 写 PG 原始表（同步写）\n        self.raw_pg.setdefault(user_id, []).append(msg)\n        # 3. 推向量库是异步的，模拟延迟\n        await asyncio.sleep(random.uniform(0, 0.02))  # 随机窗口\n        self.vector.setdefault(user_id,\n```\n\n", "url": "https://wpnews.pro/news/taking-over-llm-memory-store-testing-with-pytest-90-fewer-state-inconsistencies", "canonical_source": "https://dev.to/_eb7f2a654e97a60ae9f96e/taking-over-llm-memory-store-testing-with-pytest-90-fewer-state-inconsistencies-cjp", "published_at": "2026-07-18 12:05:08+00:00", "updated_at": "2026-07-18 12:29:03.656508+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "developer-tools", "ai-agents", "ai-safety"], "entities": ["Redis", "Qdrant", "Milvus", "PostgreSQL", "Pytest", "testcontainers-python", "tenacity", "FragileMemoryStore"], "alternates": {"html": "https://wpnews.pro/news/taking-over-llm-memory-store-testing-with-pytest-90-fewer-state-inconsistencies", "markdown": "https://wpnews.pro/news/taking-over-llm-memory-store-testing-with-pytest-90-fewer-state-inconsistencies.md", "text": "https://wpnews.pro/news/taking-over-llm-memory-store-testing-with-pytest-90-fewer-state-inconsistencies.txt", "jsonld": "https://wpnews.pro/news/taking-over-llm-memory-store-testing-with-pytest-90-fewer-state-inconsistencies.jsonld"}}