{"slug": "an-llm-described-a-website-in-detail-the-website-doesn-t-exist", "title": "An LLM described a website in detail. The website doesn't exist.", "summary": "A developer's measurement harness testing six Chinese LLM APIs found that GLM hallucinated a detailed, non-existent Chinese website for Airtable, complete with a .cn domain, pricing in RMB, and localized policies. The developer caught the error by checking domain registration and HTTP status, revealing that the domain was privately owned and returned a 502 error. The incident highlights the danger of LLMs fabricating mundane, checkable details that humans rarely verify.", "body_md": "An LLM described a company's official website to me in detail: complete Simplified Chinese interface, pricing displayed in RMB marked \"tax not included\", China-specific terms of service, a localized privacy policy. It told me this site was \"the most important source\" for verifying the company's credibility.\n\nThe domain has been registered to a private individual since 2016. It returns a 502. None of those pages have ever existed.\n\nThis is the hallucination shape that worries me most — not \"the model made something up,\" which everyone expects, but **the model made up specific, checkable, mundane details that a human would never think to check**. Nobody verifies a privacy policy's existence. You verify the big claim and assume the supporting texture came from somewhere.\n\nHere's how I caught it, why my own pipeline sat on it for a week, and the check that generalizes.\n\nI run a measurement harness against six Chinese LLM APIs — DeepSeek, Doubao, Qwen, Kimi, ERNIE, GLM — asking buyer-style questions about international software brands and logging every answer. 4,023 valid responses, retrieval off, everything stored as JSONL.\n\nOne question type asks, in Chinese and English, some version of *\"what are this brand's official channels, and how would you verify them?\"*\n\nGLM's answer for one brand:\n\nAirtable China Official Website (Airtable中国官网)\n\nURL:`https://www.airtable.cn/`\n\nWhat to look for: This is the most important source. Its existence signals a formal commitment to the Chinese market.\n\nAnd in a separate answer, in Chinese:\n\n域名\n\n`.cn`\n\n是中国的国家顶级域名，由 Airtable 官方运营，这本身就是一种官方身份的声明\n\n(The .cn domain is China's country-code TLD, operated officially by Airtable — this is itself a declaration of official identity.)\n\nConfident, structured, and it reasons about *why* the evidence counts. That last part is what makes it dangerous.\n\nMy extractor pulled URLs with a regex and recorded the domains. `airtable.cn`\n\nwent into the citation column as a cited source, indistinguishable from a real one.\n\nEvery quality check I had was a **rate**: error count, empty-answer rate, answer-length distribution, language distribution. All of them were green, because nothing about this row was anomalous. One URL among 1,416, in a well-formed answer of normal length in the expected language.\n\nRates catch a class of rows that changes size between runs. They cannot catch a class that was wrong from the first run and stayed wrong at a stable size. A reviewer put it better than I did: *rates catch a class that shrinks, asserts catch a class that was never right.*\n\nThree lookups per domain. No tools, no API, about ten minutes for eight brands.\n\n```\n# 1. Does anything answer for it?\ndig +short airtable.cn A\n# → 223.26.56.104   (someone registered it and pointed it somewhere)\n\n# 2. Who holds it?\nwhois airtable.cn | grep -iE \"^(Registrant|Registration Time|Sponsoring)\"\n# → Registrant: (a private individual)\n# → Registrant Contact Email: (a free QQ mail address)\n# → Registration Time: 2016-02-08\n\n# 3. What does it actually serve?\ncurl -s -o /dev/null -w \"%{http_code}\\n\" http://www.airtable.cn/\n# → 502\n```\n\nFor contrast, the one brand in my sample that does own its `.cn`\n\n:\n\n```\nRegistrant: BRIAN TYLER EVANS\nRegistrant Contact Email: help@clickup.com\nSponsoring Registrar: GoDaddy.com, LLC\n```\n\nThat's what ownership looks like in a registration record: a company contact, at the company's own domain. It takes one line to tell the two cases apart, and my pipeline had never looked.\n\nRunning all eight brands from the study:\n\n| Domain | Held by | Serves |\n|---|---|---|\n| clickup.cn | the brand | nothing (parked) |\n| airtable.cn | private individual | 502 |\n| wrike.cn | private individual | 502 |\n| asana.cn | private individual | \"domain for sale\" |\n| smartsheet.cn | private individual, registered 2025\n|\n\"域名转让 — The domain is on sale!\" |\n| notion.cn | a domain-holding company | 403 |\n| monday.cn |\nthe same domain-holding company |\n403 |\n| basecamp.cn | private individual, registered through 2034 | a bicycle apparel manufacturer in Dongguan |\n\nSeven of eight belong to someone other than the brand. I'm not publishing registrant names — registering an available domain is legal and these are private individuals. The interesting part is on the other side.\n\nSame engine, same collection window, a differently-worded question:\n\nYou do not access a separate\n\n`airtable.cn`\n\nwebsite. Instead, your Airtable China account is configured to use the China-hosted infrastructure.\n\nTwo incompatible accounts of the same fact, days apart, neither hedged.\n\nThat's the generalizable detection signal, and it's cheap: **ask the same factual question several ways and diff the answers.** A model that knows something answers consistently. A model that is constructing something plausible constructs differently each time, because there's no underlying fact constraining it.\n\nIn my open-question data, 18.8% of question-pairs changed outcome between two runs on the same day. If you're evaluating an LLM's factual output and you only ask once, you have no way to distinguish knowledge from confabulation.\n\n**A predicate check on extracted URLs.** A URL inside a clause that denies its existence is not a citation. My extractor was matching tokens without reading the sentence around them:\n\n``` js\nconst URL_NEGATION_CUES =\n  /没有|不存在|并无|未(设立|开设|推出|建立)|无(独立|专门|官方)|不提供|尚未|(?:does not|doesn't|no)\\s+(?:have|exist|operate)/i;\n\nexport function urlIsNegated(text, index) {\n  return URL_NEGATION_CUES.test(clauseAround(text, index));\n}\n```\n\nOne implementation note that cost me a wrong result: URL predicates need **tighter clause boundaries than entity mentions do**. Split on sentence punctuation only, and `\"并没有推出中文官网，其主要官网是 https://basecamp.com\"`\n\nflags that URL as negated — but the negation targets the Chinese site and the URL is being *affirmed*, one comma later. Splitting on commas as well fixed it: zero false flags across 1,416 URLs, and 74 genuine anti-citations in the bare-domain form my original regex never captured at all.\n\n**Assertions at the joins.** Anywhere two vocabularies meet, assert a hit that must be there or refuse to run. In my harness there were three such seams and all three were quietly broken:\n\n**Provenance on every row**, so a number can be reconstructed later rather than silently changing when the scorer improves: `scoring_version`\n\n, `finish_reason`\n\n, completion and reasoning token counts, a response hash, and a validity enum decided *before* any content scoring runs.\n\nThe failure wasn't that a model hallucinated. It's that the hallucination was **operationally indistinguishable from a fact** at every layer of my pipeline, and every quality metric I had was green while it sat there.\n\nIf you're building anything that treats LLM output as evidence — extraction, enrichment, research automation, RAG evaluation — the questions worth asking are:\n\nI've published corrections to my own numbers four times in three weeks doing this. Every single one was found either by reading raw output by hand or by a stranger asking a question I couldn't answer. Neither is a metric you can add to a dashboard, which I think is the actual lesson.\n\nHarness, labelled validation samples, and the re-scoring scripts are public under CC BY 4.0: [github.com/David88666/china-ai-visibility-benchmark](https://github.com/David88666/china-ai-visibility-benchmark)", "url": "https://wpnews.pro/news/an-llm-described-a-website-in-detail-the-website-doesn-t-exist", "canonical_source": "https://dev.to/visibilityatlas/an-llm-described-a-website-in-detail-the-website-doesnt-exist-3ldp", "published_at": "2026-08-05 03:13:21+00:00", "updated_at": "2026-08-05 03:42:52.541325+00:00", "lang": "en", "topics": ["large-language-models", "ai-safety", "ai-research"], "entities": ["GLM", "Airtable", "DeepSeek", "Doubao", "Qwen", "Kimi", "ERNIE", "ClickUp"], "alternates": {"html": "https://wpnews.pro/news/an-llm-described-a-website-in-detail-the-website-doesn-t-exist", "markdown": "https://wpnews.pro/news/an-llm-described-a-website-in-detail-the-website-doesn-t-exist.md", "text": "https://wpnews.pro/news/an-llm-described-a-website-in-detail-the-website-doesn-t-exist.txt", "jsonld": "https://wpnews.pro/news/an-llm-described-a-website-in-detail-the-website-doesn-t-exist.jsonld"}}