I Wrote a Playwright Script to Test LLM Long-Term Memory — and Found 3 Critical Bugs

It was 1 a.m. when the PM dropped a screenshot into the group chat: “Your AI assistant forgot the customer’s name again. Third time.” I zoomed in and dragged my finger across the words “Dear user, hello!”. My heart sank — the “long-term memory module” we had just shipped was completely unreliable in the real world. I had manually tested dozens of conversations. It always felt fine, but the moment it hit real users, everything fell apart. I closed Slack, opened my IDE, and decided to build an automated test with Playwright that would slap the system in the face, repeatedly. In the end, the test didn’t just pin down the problem — it also unearthed three fatal framework-level bugs that were making our memory go schizophrenic.

We were building an LLM-powered customer support assistant, with LangChain’s ConversationSummaryBufferMemory

handling long-term storage on the backend. The dream was simple: the user’s name, requests, and preferences should be remembered across sessions. The reality? Memory is probabilistic — the model forgets sometimes, the framework messes up sometimes, and edge cases (like concurrent sessions) can cause memory to leak from one user to another. Manual testing feels impossible because:

Regular unit tests can verify the memory component in isolation, but the true end-to-end memory flow — from user input to memory summarization to cross-session retrieval — only reveals itself through real browser interaction. That’s when I turned to Playwright: it can click pages, log in, operate multiple windows just like a real user, and reliably extract the LLM’s reply text from the DOM for assertions.

A lot of people asked me: “Why do you need a browser to verify memory? Can’t you just call the /chat

endpoint?” If we were only checking model outputs, sure. But the full long-term memory chain involves frontend session management and browser-side state retention. For example:

sessionStorage

. The backend only recognizes that ID.So browser testing is unavoidable. Next came tooling:

async/await

(with a synchronous API too), auto-waiting for elements, network interception, and the ability to open multiple browser contexts in a single script to simulate different users. Perfect.For architecture, we used pytest

to drive Playwright. Each test case simulates a full memory flow:

This approach keeps every test case independent and parallelizable.

This code solves the “start/stop browser for every test” problem. We use a session

-scoped fixture to launch Chromium only once.

import pytest
from playwright.sync_api import sync_playwright

@pytest.fixture(scope="session")
def browser():
    with sync_playwright() as p:
        browser = p.chromium.launch(headless=False, slow_mo=100)
        yield browser
        browser.close()

@pytest.fixture
def page(browser):
    context = browser.new_context()      # 隔离的浏览器上下文，相当于一个全新用户
    page = context.new_page()
    yield page
    context.close()

This test validates the most basic long-term memory correctness — that something you put in can be pulled out on a different day.

python
import re
import pytest

def test_basic_long_term_memory(page):
    page.goto("http://localhost:3000/login")

    page.fill("input[name='username']", "alice")
    page.fill("input[name='password']", "secret")
    page.click("button:has-text('登录')")
    page.wait_for_selector(".chat-container")   # 等待主聊天界面出现

    page.fill("textarea.chat-input", "我叫王小明，工号 9812，负责华东区售后。")
    page.click("button:has-text('发送')")
    page.wait_for_selector(".message.assistant:last-child :text-is('好的')", timeout=10000)

    page.click("button:has-text('新会话')")
    page.wait_for_selector(".chat-container")

    page.fill("textarea.chat-input", "请问我的工号是多少？")
    page.click("button:has-text('发送')")
    page.wait_for_selector(".message.assistant:last-child", timeout=10000)
    re