# I Wrote a Playwright Script to Test LLM Long-Term Memory — and Found 3 Critical Bugs

> Source: <https://dev.to/_eb7f2a654e97a60ae9f96e/i-wrote-a-playwright-script-to-test-llm-long-term-memory-and-found-3-critical-bugs-2c71>
> Published: 2026-07-09 01:05:11+00:00

It was 1 a.m. when the PM dropped a screenshot into the group chat: “Your AI assistant forgot the customer’s name again. Third time.” I zoomed in and dragged my finger across the words “Dear user, hello!”. My heart sank — the “long-term memory module” we had just shipped was completely unreliable in the real world. I had manually tested dozens of conversations. It always *felt* fine, but the moment it hit real users, everything fell apart. I closed Slack, opened my IDE, and decided to build an automated test with Playwright that would slap the system in the face, repeatedly. In the end, the test didn’t just pin down the problem — it also unearthed three fatal framework-level bugs that were making our memory go schizophrenic.

We were building an LLM-powered customer support assistant, with LangChain’s `ConversationSummaryBufferMemory`

handling long-term storage on the backend. The dream was simple: the user’s name, requests, and preferences should be remembered across sessions. The reality? **Memory is probabilistic** — the model forgets sometimes, the framework messes up sometimes, and edge cases (like concurrent sessions) can cause memory to leak from one user to another. Manual testing feels impossible because:

Regular unit tests can verify the memory component in isolation, but the true end-to-end memory flow — from user input to memory summarization to cross-session retrieval — only reveals itself through real browser interaction. That’s when I turned to Playwright: it can click pages, log in, operate multiple windows just like a real user, and reliably extract the LLM’s reply text from the DOM for assertions.

A lot of people asked me: “Why do you need a browser to verify memory? Can’t you just call the `/chat`

endpoint?” If we were only checking model outputs, sure. But the full long-term memory chain involves **frontend session management** and **browser-side state retention**. For example:

`sessionStorage`

. The backend only recognizes that ID.So browser testing is unavoidable. Next came tooling:

`async/await`

(with a synchronous API too), auto-waiting for elements, network interception, and the ability to open multiple browser contexts in a single script to simulate different users. Perfect.For architecture, we used `pytest`

to drive Playwright. Each test case simulates a full memory flow:

This approach keeps every test case **independent and parallelizable**.

This code solves the “start/stop browser for every test” problem. We use a `session`

-scoped fixture to launch Chromium only once.

``` python
# conftest.py
import pytest
from playwright.sync_api import sync_playwright

@pytest.fixture(scope="session")
def browser():
    with sync_playwright() as p:
        # headless=False 可以在调试时看到浏览器操作，CI 里设 True
        browser = p.chromium.launch(headless=False, slow_mo=100)
        yield browser
        browser.close()

@pytest.fixture
def page(browser):
    context = browser.new_context()      # 隔离的浏览器上下文，相当于一个全新用户
    page = context.new_page()
    yield page
    context.close()
```

This test validates the most basic long-term memory correctness — that something you put in can be pulled out on a different day.

``` python
python
# test_memory.py
import re
import pytest

def test_basic_long_term_memory(page):
    page.goto("http://localhost:3000/login")

    # 登录（假设一个简单的表单）
    page.fill("input[name='username']", "alice")
    page.fill("input[name='password']", "secret")
    page.click("button:has-text('登录')")
    page.wait_for_selector(".chat-container")   # 等待主聊天界面出现

    # --- 会话 A：注入记忆 ---
    page.fill("textarea.chat-input", "我叫王小明，工号 9812，负责华东区售后。")
    page.click("button:has-text('发送')")
    # 等待 LLM 回复出现在最后一条消息中（必须确定有文字，不能只是 DOM 挂载）
    page.wait_for_selector(".message.assistant:last-child :text-is('好的')", timeout=10000)

    # 新建会话，模拟跨天
    page.click("button:has-text('新会话')")
    page.wait_for_selector(".chat-container")

    # --- 会话 B：验证记忆 ---
    page.fill("textarea.chat-input", "请问我的工号是多少？")
    page.click("button:has-text('发送')")
    # 关键点：不用 sleep，用文本断言等待
    page.wait_for_selector(".message.assistant:last-child", timeout=10000)
    re
```