{"slug": "one-ai-answer-eight-brands-designing-a-benchmark-without-multiplying-the", "title": "One AI Answer, Eight Brands: Designing a Benchmark Without Multiplying the Evidence", "summary": "A developer ran a China AI benchmark for eight luxury-jewelry brands and found that Piaget appeared in all answers about brands with official China channels but in none of the wedding-jewelry recommendations. The developer designed an answer-once collection method to avoid multiplying evidence, separating raw model answers from brand-level evaluations. The benchmark highlighted the need for multiple metrics and careful accounting of invalid responses.", "body_md": "I recently ran a small China AI benchmark for eight luxury-jewelry brands.\n\nThe most interesting result was not a platform ranking.\n\nIt was a disagreement between two kinds of visibility.\n\nPiaget appeared in all four answers about brands with verifiable official China channels. It appeared in none of the four answers recommending brands for wedding jewelry.\n\nThat is a tiny sample, so it is not evidence that Piaget has a stable visibility problem.\n\nIt is evidence that I needed more than one metric.\n\nIt also forced me to separate two units that are often quietly conflated in AI visibility datasets:\n\nThat distinction changed the collector, the denominator and what I was willing to publish.\n\nThe cohort contained eight brands:\n\nI had three neutral buyer questions, two API surfaces and two replicates.\n\nA naive brand-expanded collector could have produced:\n\n```\n8 brands\n× 3 questions\n× 2 surfaces\n× 2 replicates\n= 96 provider calls\n```\n\nThat would look like a larger dataset.\n\nIt would also change the experiment.\n\nIf I asked the same neutral question eight times—once for every target brand—I would collect eight different generated answers and then compare brands across different evidence.\n\nRandomness, provider conditions and answer variation would be mixed with brand treatment.\n\nThe larger row count would not mean I had more independent evidence about the same answer. It would mean I had asked for more answers.\n\nSo I used an answer-once design.\n\n```\n3 questions\n× 2 surfaces\n× 2 replicates\n= 12 valid raw answers\n```\n\nEach raw answer was then evaluated against the eight brands:\n\n```\n12 answers\n× 8 brands\n= 96 answer-brand cells\n```\n\nThe dataset still contains 96 brand-level observations, but it does not pretend that 96 model responses were collected.\n\nThe collection unit was:\n\n```\nplatform_surface_id\n× question_id\n× replicate_id\n```\n\nThe derived analysis unit was:\n\n```\nanswer_id\n× target_brand_id\n```\n\nA simplified record relationship looks like this:\n\n```\nAnswer\n├── answer_id\n├── platform_surface_id\n├── question_id\n├── replicate_id\n├── raw_answer\n├── validity\n└── AnswerBrandCell[8]\n    ├── target_brand_id\n    ├── mentioned\n    ├── recommended\n    ├── official_channel_asserted\n    └── human_review_state\n```\n\nThis matters because Mention, Recommendation and channel accuracy are not properties of the API request alone.\n\nThey are judgments about a particular brand inside a particular answer.\n\nThe raw answer should remain immutable. Brand-level labels can be reviewed, corrected and replayed without recollecting the model output.\n\nA third API surface was planned for the pre-wave.\n\nAll six requests returned an account-state `403`\n\n.\n\nThose rows belong in the collection record because the attempts happened under a declared instrument. They do not belong in any answer-content denominator.\n\nThe correct accounting was:\n\n```\nplanned answer cells = 18\nvalid answers        = 12\ninvalid by reason    = 6\n```\n\nIt was not:\n\n```\nvalid answers        = 12\nbrand absent         = 6\n```\n\nAn error row proves that a collection attempt occurred. It contains no evidence about whether Cartier, Piaget or any other brand was mentioned or recommended.\n\nI also retained two earlier Doubao timeouts from an instrument-configuration run. They remain in the audit trail but sit outside the final planned set because the collector had not carried forward the reasoning-mode configuration used during access testing.\n\nKeeping failures does not require pretending every failure belongs to the published denominator.\n\nThe three questions represented different buyer decisions:\n\nThe aggregate results were:\n\n| Measurement | Numerator | Denominator |\n|---|---|---|\n| Wedding shortlist recommendation | 21 | 32 |\n| Target-brand mention in the daigou-risk answer | 0 | 32 |\n| Official-channel assertion | 25 | 32 |\n| Exact current China-local route match | 15 | 19 |\n\nThose denominators are not interchangeable.\n\nThe recommendation denominator contains the eight brand cells from the four wedding answers.\n\nThe risk-question denominator contains the eight brand cells from a different four answers.\n\nThe route-match denominator contains only exact domain assertions that were eligible for truth review.\n\nCombining them into one “AI visibility score” would make the output simpler and the diagnosis worse.\n\nThe Piaget row demonstrates why:\n\n| Brand | Wedding recommendation | Official-channel appearance |\n|---|---|---|\n| Piaget | 0/4 | 4/4 |\n\nThe brand was recognized when the task was channel verification. It was absent when the task was open-category recommendation.\n\nA domain or entity project might improve verification while doing nothing to answer why the brand did not enter the shortlist.\n\nThe inverse problem also appeared:\n\n| Brand | Wedding recommendation | Official-channel appearance |\n|---|---|---|\n| Chaumet | 3/4 | 2/4 |\n\nA brand can enter consideration while its verification route is reproduced less consistently.\n\nThese are different business problems, with different owners and different retest questions.\n\nA count can be reproducible while the population is wrong.\n\nTwo filters can both return 32 rows and still return different sets of 32 rows.\n\nFor every published metric, I therefore stored the exact denominator and numerator memberships.\n\nA simplified manifest looks like this:\n\n```\n{\n  \"metric_id\": \"wedding_recommendation\",\n  \"unit\": \"answer_brand_cell\",\n  \"eligible_predicate\": {\n    \"question_id\": \"JW-CORE-01\",\n    \"answer_validity\": \"VALID\"\n  },\n  \"denominator\": {\n    \"member_ids\": [\n      \"answer-001::cartier\",\n      \"answer-001::tiffany\"\n    ],\n    \"sha256\": \"...\"\n  },\n  \"numerator\": {\n    \"predicate\": \"human_recommendation_label == POSITIVE_SHORTLIST\",\n    \"member_ids\": [\n      \"answer-001::cartier\"\n    ],\n    \"sha256\": \"...\"\n  },\n  \"excluded_by_reason\": {\n    \"invalid_answer\": [],\n    \"not_eligible_question\": []\n  }\n}\n```\n\nThe real manifest contains the full member sets rather than the abbreviated example above.\n\nA separate verifier rebuilds those sets from the stored row states and checks invariants such as:\n\n```\nassert(isSubset(numeratorIds, denominatorIds));\nassert(plannedCount === validCount + invalidByReasonCount);\nassert(intersection(errorRowIds, denominatorIds).size === 0);\nassert(hash(rebuiltDenominatorIds) === storedDenominatorHash);\nassert(hash(rebuiltNumeratorIds) === storedNumeratorHash);\n```\n\nThe important guarantee is not merely:\n\n```\n21 / 32 can be recalculated\n```\n\nIt is:\n\n```\nthe same 21 cells and the same 32 cells can be reconstructed\n```\n\nThat catches two implementations producing the same percentage from different observations.\n\nA string match was sufficient for a first-pass entity mention.\n\nIt was not sufficient for Recommendation.\n\nA brand name can appear because the answer:\n\nEvery counted recommendation was therefore reviewed in the full answer and assigned to the target brand cell.\n\nThe rule was deliberately narrower than positive sentiment:\n\n```\nCount the brand only when the answer explicitly recommends it\nor places it on a positive shortlist for the buyer decision.\n```\n\nComparison, exclusion and incidental mention did not qualify.\n\nThis is slower than a regex. For a 96-cell exploratory study, it was still practical and materially safer.\n\nThe two valid surfaces were retrieval-off APIs.\n\nThe answers can be studied as dated model outputs. They cannot establish which sources influenced the result.\n\nEven if an answer prints a domain, that does not prove that the model retrieved or used the page during the request.\n\nSo the study reports:\n\n```\nSupported target Citation Rate: NOT_OBSERVABLE\n```\n\nIt does not convert missing citation evidence into zero.\n\nThat distinction is important for longitudinal work. If a later wave uses a consumer search surface with observable retrieval and attributable citations, it should become a separate series rather than silently extending the retrieval-off API series.\n\nThe implementation lesson is straightforward:\n\n`NOT_OBSERVABLE`\n\n, not zero.The more interesting consequence is that a visibility audit should not end with one score.\n\nFor a luxury brand, I now want at least these separate layers:\n\n```\nEntity recognition\n→ Open-category consideration\n→ Positive recommendation\n→ Official-channel accuracy\n→ Supported source, when observable\n```\n\nA brand can fail at any transition.\n\nKnowing which transition failed is more useful than knowing that an overall score moved from 42 to 47.\n\nThis pre-wave contains only four answer opportunities per brand for each task. It is enough to find a measurement problem, not enough to establish a stable brand position.\n\nA formal next wave would need:\n\nI would deepen the buyer journey before adding dozens of brands.\n\nFor jewelry, that means questions around wedding purchases, anniversary gifts, diamond selection, high-jewelry commissions, boutique-versus-daigou risk and mainland after-sales confidence.\n\nThe goal is not to manufacture a larger dataset.\n\nIt is to find out where a brand moves from being known, to being considered, to being recommended, to being correctly verified.\n\nIn this small pre-wave, Piaget made the final distinction visible: the models could place the brand on the official China channel map, but they did not place it on the wedding shortlist.\n\nThat is not a conclusion about the brand.\n\nIt is a much better next question than “What is its AI visibility score?”", "url": "https://wpnews.pro/news/one-ai-answer-eight-brands-designing-a-benchmark-without-multiplying-the", "canonical_source": "https://dev.to/visibilityatlas/one-ai-answer-eight-brands-designing-a-benchmark-without-multiplying-the-evidence-1o45", "published_at": "2026-08-13 09:30:05+00:00", "updated_at": "2026-08-13 09:46:19.657835+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-research", "ai-products"], "entities": ["Piaget", "Cartier", "Doubao"], "alternates": {"html": "https://wpnews.pro/news/one-ai-answer-eight-brands-designing-a-benchmark-without-multiplying-the", "markdown": "https://wpnews.pro/news/one-ai-answer-eight-brands-designing-a-benchmark-without-multiplying-the.md", "text": "https://wpnews.pro/news/one-ai-answer-eight-brands-designing-a-benchmark-without-multiplying-the.txt", "jsonld": "https://wpnews.pro/news/one-ai-answer-eight-brands-designing-a-benchmark-without-multiplying-the.jsonld"}}