AI companion and AI girlfriend apps are a useful stress test for a problem that affects many stateful AI products: the claims users care about are easy to market and surprisingly hard to verify.
"Long-term memory" can mean remembering a name ten messages later, retaining a fact across sessions, or keeping dozens of facts straight after the conversation has accumulated noise. "Consistent characters" can mean tone, biography, appearance, or all three. A single successful image says almost nothing about the next ten generations.
Yet many review pages turn those claims into precise-looking scores without showing the paid plan, test date, prompt sequence, transcript, retries, or failed generations behind the number.
I wanted a test another person could actually rerun.
I started NoFilterReview by buying the products myself. Five paid hands-on tests are live today.
For each product I record:
That already catches a lot of bad review data. Prices change. Models get replaced. A feature shown on a landing page may not exist in the tested plan. A reviewer can easily mistake one lucky generation for normal quality.
But manual testing alone has a serious weakness: I am part of the instrument. My phrasing, timing, patience, and expectations can change the result.
The next step is an agentic benchmark that runs the same scenario against every product.
The benchmark is still in build. I am not presenting the following system as finished.
The runner needs to control five things.
A memory test should plant facts at known points, then surround them with realistic conversational noise.
A draft scenario might include:
{
"fact_id": "F07",
"fact": "My sister's dog is named Pepper",
"introduced_in_session": 1,
"recall_check_session": 4,
"acceptable_answers": ["Pepper"],
"distractors": [
"My coworker's dog is named Milo",
"I used to have a cat named Luna"
]
}
The important part is not the JSON. It is the separation between what was introduced, when it was introduced, and what counts as a correct answer.
A ten-message chat is not a long-term memory test.
The runner needs separate sessions, controlled gaps, topic changes, conflicting details, and delayed recall checks. It also needs to distinguish:
Memory and character consistency are related but not identical.
A character can remember a fact while drifting into a different personality. The evaluator should track changes in:
A useful benchmark should not reward a model for parroting facts while the character itself falls apart.
Media tests need repeated requests, not one cherry-picked result.
For every app, the runner should request comparable scenes and record:
The hard part is scoring identity consistency without hiding the judgment inside another black box.
A vision model can help produce candidate scores, but the benchmark still needs a published rubric, calibration examples, and periodic human review. Otherwise the evaluator simply replaces an opaque product claim with an opaque benchmark claim.
These products use different currencies, retry rules, queues, and subscription limits. Normalizing the marketing price alone is not enough.
I want to report both the advertised price and the observed cost of completing a fixed test scenario. Failed generations, retries, and timeouts stay in the denominator.
The testing agent should not grade its own work.
My current design separates:
That separation does not eliminate bias, but it makes errors easier to inspect. If the score looks wrong, a reader can trace it back to a particular artifact and rubric rule.
A benchmark is not reproducible if readers only see a leaderboard.
For each published result, I plan to expose:
The goal is not to produce a magical objective score. The goal is to make every important judgment inspectable.
First, how long should a memory test run before "long-term" means anything? Four sessions may be practical, but practical is not the same as valid.
Second, what is the least misleading way to score identity consistency across images and video? Embedding similarity is tempting, but it can miss the exact visual drift a human notices immediately.
If you have built evaluation systems for stateful agents, multimodal models, or long-running conversations, I would value specific criticism of the protocol.
The current manual methodology and paid-test standard are here: https://nofilterreview.com/how-we-test/
Disclosure: I used an AI writing tool to help organize this article. The paid tests, product observations, methodology decisions, and final edits are mine.