{"slug": "how-i-m-building-reproducible-agentic-tests-for-ai-girlfriend-apps", "title": "How I'm Building Reproducible Agentic Tests for AI Girlfriend Apps", "summary": "A developer building NoFilterReview has launched five paid hands-on tests of AI companion apps and is developing an agentic benchmark to make claims like 'long-term memory' and 'consistent characters' reproducible. The benchmark separates fact introduction from recall checks, tracks character drift, and plans to expose artifacts and rubrics for every published result.", "body_md": "AI companion and AI girlfriend apps are a useful stress test for a problem that affects many stateful AI products: the claims users care about are easy to market and surprisingly hard to verify.\n\n\"Long-term memory\" can mean remembering a name ten messages later, retaining a fact across sessions, or keeping dozens of facts straight after the conversation has accumulated noise. \"Consistent characters\" can mean tone, biography, appearance, or all three. A single successful image says almost nothing about the next ten generations.\n\nYet many review pages turn those claims into precise-looking scores without showing the paid plan, test date, prompt sequence, transcript, retries, or failed generations behind the number.\n\nI wanted a test another person could actually rerun.\n\nI started NoFilterReview by buying the products myself. Five paid hands-on tests are live today.\n\nFor each product I record:\n\nThat already catches a lot of bad review data. Prices change. Models get replaced. A feature shown on a landing page may not exist in the tested plan. A reviewer can easily mistake one lucky generation for normal quality.\n\nBut manual testing alone has a serious weakness: I am part of the instrument. My phrasing, timing, patience, and expectations can change the result.\n\nThe next step is an agentic benchmark that runs the same scenario against every product.\n\nThe benchmark is still in build. I am not presenting the following system as finished.\n\nThe runner needs to control five things.\n\nA memory test should plant facts at known points, then surround them with realistic conversational noise.\n\nA draft scenario might include:\n\n```\n{\n  \"fact_id\": \"F07\",\n  \"fact\": \"My sister's dog is named Pepper\",\n  \"introduced_in_session\": 1,\n  \"recall_check_session\": 4,\n  \"acceptable_answers\": [\"Pepper\"],\n  \"distractors\": [\n    \"My coworker's dog is named Milo\",\n    \"I used to have a cat named Luna\"\n  ]\n}\n```\n\nThe important part is not the JSON. It is the separation between what was introduced, when it was introduced, and what counts as a correct answer.\n\nA ten-message chat is not a long-term memory test.\n\nThe runner needs separate sessions, controlled gaps, topic changes, conflicting details, and delayed recall checks. It also needs to distinguish:\n\nMemory and character consistency are related but not identical.\n\nA character can remember a fact while drifting into a different personality. The evaluator should track changes in:\n\nA useful benchmark should not reward a model for parroting facts while the character itself falls apart.\n\nMedia tests need repeated requests, not one cherry-picked result.\n\nFor every app, the runner should request comparable scenes and record:\n\nThe hard part is scoring identity consistency without hiding the judgment inside another black box.\n\nA vision model can help produce candidate scores, but the benchmark still needs a published rubric, calibration examples, and periodic human review. Otherwise the evaluator simply replaces an opaque product claim with an opaque benchmark claim.\n\nThese products use different currencies, retry rules, queues, and subscription limits. Normalizing the marketing price alone is not enough.\n\nI want to report both the advertised price and the observed cost of completing a fixed test scenario. Failed generations, retries, and timeouts stay in the denominator.\n\nThe testing agent should not grade its own work.\n\nMy current design separates:\n\nThat separation does not eliminate bias, but it makes errors easier to inspect. If the score looks wrong, a reader can trace it back to a particular artifact and rubric rule.\n\nA benchmark is not reproducible if readers only see a leaderboard.\n\nFor each published result, I plan to expose:\n\nThe goal is not to produce a magical objective score. The goal is to make every important judgment inspectable.\n\nFirst, how long should a memory test run before \"long-term\" means anything? Four sessions may be practical, but practical is not the same as valid.\n\nSecond, what is the least misleading way to score identity consistency across images and video? Embedding similarity is tempting, but it can miss the exact visual drift a human notices immediately.\n\nIf you have built evaluation systems for stateful agents, multimodal models, or long-running conversations, I would value specific criticism of the protocol.\n\nThe current manual methodology and paid-test standard are here: [https://nofilterreview.com/how-we-test/](https://nofilterreview.com/how-we-test/)\n\n*Disclosure: I used an AI writing tool to help organize this article. The paid tests, product observations, methodology decisions, and final edits are mine.*", "url": "https://wpnews.pro/news/how-i-m-building-reproducible-agentic-tests-for-ai-girlfriend-apps", "canonical_source": "https://dev.to/samm_nofilterreview/how-im-building-reproducible-agentic-tests-for-ai-girlfriend-apps-4a06", "published_at": "2026-08-15 07:15:23+00:00", "updated_at": "2026-08-15 07:41:04.961237+00:00", "lang": "en", "topics": ["ai-agents", "ai-products", "ai-tools", "ai-ethics"], "entities": ["NoFilterReview"], "alternates": {"html": "https://wpnews.pro/news/how-i-m-building-reproducible-agentic-tests-for-ai-girlfriend-apps", "markdown": "https://wpnews.pro/news/how-i-m-building-reproducible-agentic-tests-for-ai-girlfriend-apps.md", "text": "https://wpnews.pro/news/how-i-m-building-reproducible-agentic-tests-for-ai-girlfriend-apps.txt", "jsonld": "https://wpnews.pro/news/how-i-m-building-reproducible-agentic-tests-for-ai-girlfriend-apps.jsonld"}}