I spend my working life on how Chinese search and AI systems describe international brands, and I got tired of arguing about it from screenshots. So I built a harness and recorded 4,704 answers across six Chinese AI engines.
This post is about one slice of it: the CRM category, where the result was more extreme than I expected.
8 international CRM brands (Salesforce, HubSpot, Zoho CRM, Pipedrive, Microsoft Dynamics 365, Freshsales, SugarCRM, Monday CRM) × 42 Chinese-language buyer questions × 2 runs = 672 recorded answers.
The questions come in five groups: branded checks, open category discovery, comparison, decision intent, and risk. That grouping turned out to matter more than anything else in the study.
Every Chinese engine I tested speaks the OpenAI chat-completions dialect, so the collector is boring on purpose:
async function ask(prompt) {
const body = {
model: cfg.model,
messages: [{ role: "user", content: prompt }],
temperature: 0.3,
max_tokens: 1200,
};
// Ark reasoning models bury the answer in reasoning_content unless you do this
if (process.env.BENCH_THINKING === "disabled") body.thinking = { type: "disabled" };
const res = await fetch(`${cfg.baseUrl}/chat/completions`, {
method: "POST",
headers: {
"content-type": "application/json",
authorization: `Bearer ${cfg.apiKey}`,
},
body: JSON.stringify(body),
signal: AbortSignal.timeout(Number(process.env.BENCH_TIMEOUT_MS || 90000)),
});
if (!res.ok) throw new Error(`${res.status} ${(await res.text()).slice(0, 200)}`);
const j = await res.json();
return j.choices?.[0]?.message?.content || "";
}
One env-var swap per engine — base URL, model ID, platform label — and the same panel runs everywhere. Total API spend for the whole study was under $5.
Mention rate on open category questions ("which CRM should I consider in China?", brand name never mentioned in the prompt):
| Brand | Open-question mention |
|---|---|
| Salesforce | 87.5% |
| Microsoft Dynamics 365 | 50.0% |
| Zoho CRM | 12.5% |
| HubSpot | 0.0% |
| Pipedrive | 0.0% |
| Freshsales | 0.0% |
| SugarCRM | 0.0% |
| Monday CRM | 0.0% |
Five of eight global brands never surfaced. Not once, across every phrasing and both runs.
The engine's own framing, repeated almost verbatim across answers: "若企业有全球化需求，优先考虑 Salesforce 或 Microsoft Dynamics" — if you need global reach, consider Salesforce or Dynamics. The "international option" isn't a shelf with several slots. It's one seat, and Salesforce is in it. Everything else in the list goes to domestic vendors (销售易, 纷享销客, Kingdee).
Ask the same engines about HubSpot by name and you get a competent, fair, well-structured paragraph. Branded-question mention rate was ~100% for every brand, on every engine.
So the check most teams run — "I asked DeepSeek about us and it looked fine" — returns a false positive by construction. The prompt hands the model the answer. The number that predicts whether a buyer ever meets you is the open one, and for this sample it was 23% across all categories.
Reading HubSpot's 84 branded answers, three things stood out:
Things I'd want to know if I were reading someone else's version of this:
content: ""
with 2,000 characters in reasoning_content
until I forced answer-only mode.deepseek-chat
entirely and forced a rename. These products move faster than the analysis does.Aggregate results and the reusable 42-question bilingual panel are CC BY 4.0:
https://github.com/David88666/china-ai-visibility-benchmark
Swap in your own brands and category — the panel has {brand}
/ {category}
/ {competitor}
slots — and you can reproduce the whole thing in an afternoon.
If you work on something with international reach, the cheap version of this test is worth running once: three engines, five open category questions in the local language, twice each. It takes an hour, and it's a very different picture from asking the model about yourself by name.