I Wrote a Playwright Script to Test LLM Long-Term Memory — and Found 3 Critical Bugs

A developer built a Playwright script to test an LLM-powered customer support assistant's long-term memory and discovered three critical framework-level bugs. The automated test revealed that the memory module, built with LangChain's ConversationSummaryBufferMemory, was unreliable in real-world scenarios, with issues including probabilistic forgetting and memory leaking between concurrent sessions.

It was 1 a.m. when the PM dropped a screenshot into the group chat: “Your AI assistant forgot the customer’s name again. Third time.” I zoomed in and dragged my finger across the words “Dear user, hello ”. My heart sank — the “long-term memory module” we had just shipped was completely unreliable in the real world. I had manually tested dozens of conversations. It always felt fine, but the moment it hit real users, everything fell apart. I closed Slack, opened my IDE, and decided to build an automated test with Playwright that would slap the system in the face, repeatedly. In the end, the test didn’t just pin down the problem — it also unearthed three fatal framework-level bugs that were making our memory go schizophrenic. We were building an LLM-powered customer support assistant, with LangChain’s ConversationSummaryBufferMemory handling long-term storage on the backend. The dream was simple: the user’s name, requests, and preferences should be remembered across sessions. The reality? Memory is probabilistic — the model forgets sometimes, the framework messes up sometimes, and edge cases like concurrent sessions can cause memory to leak from one user to another. Manual testing feels impossible because: Regular unit tests can verify the memory component in isolation, but the true end-to-end memory flow — from user input to memory summarization to cross-session retrieval — only reveals itself through real browser interaction. That’s when I turned to Playwright: it can click pages, log in, operate multiple windows just like a real user, and reliably extract the LLM’s reply text from the DOM for assertions. A lot of people asked me: “Why do you need a browser to verify memory? Can’t you just call the /chat endpoint?” If we were only checking model outputs, sure. But the full long-term memory chain involves frontend session management and browser-side state retention . For example: sessionStorage . The backend only recognizes that ID.So browser testing is unavoidable. Next came tooling: async/await with a synchronous API too , auto-waiting for elements, network interception, and the ability to open multiple browser contexts in a single script to simulate different users. Perfect.For architecture, we used pytest to drive Playwright. Each test case simulates a full memory flow: This approach keeps every test case independent and parallelizable . This code solves the “start/stop browser for every test” problem. We use a session -scoped fixture to launch Chromium only once. python conftest.py import pytest from playwright.sync api import sync playwright @pytest.fixture scope="session" def browser : with sync playwright as p: headless=False 可以在调试时看到浏览器操作，CI 里设 True browser = p.chromium.launch headless=False, slow mo=100 yield browser browser.close @pytest.fixture def page browser : context = browser.new context 隔离的浏览器上下文，相当于一个全新用户 page = context.new page yield page context.close This test validates the most basic long-term memory correctness — that something you put in can be pulled out on a different day. python python test memory.py import re import pytest def test basic long term memory page : page.goto "http://localhost:3000/login" 登录（假设一个简单的表单） page.fill "input name='username' ", "alice" page.fill "input name='password' ", "secret" page.click "button:has-text '登录' " page.wait for selector ".chat-container" 等待主聊天界面出现 --- 会话 A：注入记忆 --- page.fill "textarea.chat-input", "我叫王小明，工号 9812，负责华东区售后。" page.click "button:has-text '发送' " 等待 LLM 回复出现在最后一条消息中（必须确定有文字，不能只是 DOM 挂载） page.wait for selector ".message.assistant:last-child :text-is '好的' ", timeout=10000 新建会话，模拟跨天 page.click "button:has-text '新会话' " page.wait for selector ".chat-container" --- 会话 B：验证记忆 --- page.fill "textarea.chat-input", "请问我的工号是多少？" page.click "button:has-text '发送' " 关键点：不用 sleep，用文本断言等待 page.wait for selector ".message.assistant:last-child", timeout=10000 re