# 20x Faster RAG Memory Testing: Trade Postman for Playwright + Chroma

> Source: <https://dev.to/_eb7f2a654e97a60ae9f96e/20x-faster-rag-memory-testing-trade-postman-for-playwright-chroma-33ii>
> Published: 2026-07-27 01:05:47+00:00

At 1 AM, a voice message from the product manager popped up: “User ‘Zhang San’ said he just told the assistant about his peanut allergy, but the very next message, the assistant completely forgot.” Rubbing my eyes, I opened Chroma, manually typed 4 queries — each time waiting for the page to refresh, copying a token, then assembling the request in Postman. After debugging, I realized the frontend had made an extra call to a legacy API endpoint, overwriting the freshly written memory with empty text. That path? Our manual tests had never covered it.

That night I had had enough. Testing memory storage in a RAG application manually with Postman is like inspecting the facade of a skyscraper with a flashlight — completely unreliable.

The memory write pipeline of a RAG app is far more complex than a normal API:

The old workflow: send a few fixed conversations via Postman, then manually search in Chroma’s Jupyter. That has three fatal flaws:

We needed an end‑to‑end automated solution that could drive a real browser through a full conversation and then dive directly into the vector store to inspect memory chunks — like performing a gastroscopy on the RAG.

**Option A: pure API tests (requests + Chroma client)**

This can’t reproduce real frontend behavior. The frontend might do extra processing on the context (e.g., filtering short texts, merging system prompts). Pure API tests would miss those paths.

**Option B: Selenium + custom assertions**

Selenium requires a lot of boilerplate for async waits and network interception. Playwright, on the other hand, was built for modern SPAs — `waitForResponse`

hooks directly into the target API, and traces let you replay DOM snapshots when something fails.

**Option C: mock the vector store with a lightweight fake**

Tests must hit the real Chroma, because different vector stores handle embedding normalization and index refresh strategies differently. Test with mocks, and you’re likely to have a 50% failure rate in production.

So the choice was clear: **Playwright as the browser driver + a direct Chroma client for verification**. Everything runs locally or in CI, and a single `pytest test_memory.py`

does the whole job.

This snippet creates the browser driver and initializes an embedded Chroma instance whose collection name matches production.

``` python
# test_setup.py
import chromadb
from chromadb.config import Settings
from playwright.sync_api import sync_playwright

# 嵌入式 Chroma，数据持久化到 ./chroma_data，和开发环境一致
chroma_client = chromadb.Client(
    Settings(
        chroma_db_impl="duckdb+parquet",
        persist_directory="./chroma_data"
    )
)
# 确保 collection 存在（线上用的 "chat_memories"）
collection = chroma_client.get_or_create_collection("chat_memories")

def get_browser_page():
    """启动无头浏览器并返回 page 对象"""
    playwright = sync_playwright().start()
    browser = playwright.chromium.launch(headless=True)
    page = browser.new_page()
    return page, browser, playwright
```

This test completes a full memory‑storage loop: the user types personal information on the page → the server replies and asynchronously writes to Chroma → the test verifies the correct entry directly in the database.

``` python
# test_memory.py
import time
import pytest
from test_setup import collection, get_browser_page

MEMORY_EXTRACTION_TIMEOUT = 5  # 异步写入最多等 5 秒

def test_user_profile_memory():
    page, browser, playwright = get_browser_page()
    try:
        # 1. 打开对话页面
        page.goto("http://localhost:3000/chat")
        # 2. 输入记忆信息
        page.fill("textarea#user-input", "我叫张三，我对花生过敏，记住这个。")
        # 3. 拦截 /api/chat 响应，保证 LLM 回复已返回
        with page.expect_response(lambda resp: "/api/chat" in resp.url and resp.status == 200):
            page.click("button#send")
        # 4. 等待异步写入完成（简单轮询）
        start = time.time()
        while time.time() - start < MEMORY_EXTRACTION_TIMEOUT:
            # 重新获取集合（踩坑点：不 refresh 查不到最新数据）
            fresh_collection = chroma_client.get_collection("chat_memories")
            results = fresh_collection.query(
                query_texts=["花生过敏"],
                n_results=1,
                where={"user_id": "张三"}  #
```

Here, the test would first see the result for “peanut allergy” in memory; you can then add assertions to check that the content matches expectations. The entire process is fully automated, and a full regression run takes just a few seconds.