At 1 AM, a voice message from the product manager popped up: “User ‘Zhang San’ said he just told the assistant about his peanut allergy, but the very next message, the assistant completely forgot.” Rubbing my eyes, I opened Chroma, manually typed 4 queries — each time waiting for the page to refresh, copying a token, then assembling the request in Postman. After debugging, I realized the frontend had made an extra call to a legacy API endpoint, overwriting the freshly written memory with empty text. That path? Our manual tests had never covered it.
That night I had had enough. Testing memory storage in a RAG application manually with Postman is like inspecting the facade of a skyscraper with a flashlight — completely unreliable.
The memory write pipeline of a RAG app is far more complex than a normal API:
The old workflow: send a few fixed conversations via Postman, then manually search in Chroma’s Jupyter. That has three fatal flaws:
We needed an end‑to‑end automated solution that could drive a real browser through a full conversation and then dive directly into the vector store to inspect memory chunks — like performing a gastroscopy on the RAG.
Option A: pure API tests (requests + Chroma client)
This can’t reproduce real frontend behavior. The frontend might do extra processing on the context (e.g., filtering short texts, merging system prompts). Pure API tests would miss those paths.
Option B: Selenium + custom assertions
Selenium requires a lot of boilerplate for async waits and network interception. Playwright, on the other hand, was built for modern SPAs — waitForResponse
hooks directly into the target API, and traces let you replay DOM snapshots when something fails.
Option C: mock the vector store with a lightweight fake
Tests must hit the real Chroma, because different vector stores handle embedding normalization and index refresh strategies differently. Test with mocks, and you’re likely to have a 50% failure rate in production.
So the choice was clear: Playwright as the browser driver + a direct Chroma client for verification. Everything runs locally or in CI, and a single pytest test_memory.py
does the whole job.
This snippet creates the browser driver and initializes an embedded Chroma instance whose collection name matches production.
import chromadb
from chromadb.config import Settings
from playwright.sync_api import sync_playwright
chroma_client = chromadb.Client(
Settings(
chroma_db_impl="duckdb+parquet",
persist_directory="./chroma_data"
)
)
collection = chroma_client.get_or_create_collection("chat_memories")
def get_browser_page():
"""启动无头浏览器并返回 page 对象"""
playwright = sync_playwright().start()
browser = playwright.chromium.launch(headless=True)
page = browser.new_page()
return page, browser, playwright
This test completes a full memory‑storage loop: the user types personal information on the page → the server replies and asynchronously writes to Chroma → the test verifies the correct entry directly in the database.
import time
import pytest
from test_setup import collection, get_browser_page
MEMORY_EXTRACTION_TIMEOUT = 5 # 异步写入最多等 5 秒
def test_user_profile_memory():
page, browser, playwright = get_browser_page()
try:
page.goto("http://localhost:3000/chat")
page.fill("textarea#user-input", "我叫张三，我对花生过敏，记住这个。")
with page.expect_response(lambda resp: "/api/chat" in resp.url and resp.status == 200):
page.click("button#send")
start = time.time()
while time.time() - start < MEMORY_EXTRACTION_TIMEOUT:
fresh_collection = chroma_client.get_collection("chat_memories")
results = fresh_collection.query(
query_texts=["花生过敏"],
n_results=1,
where={"user_id": "张三"} #
Here, the test would first see the result for “peanut allergy” in memory; you can then add assertions to check that the content matches expectations. The entire process is fully automated, and a full regression run takes just a few seconds.