{"slug": "how-wrong-is-a-model-about-a-company-when-it-has-no-register-to-check", "title": "How wrong is a model about a company when it has no register to check?", "summary": "A developer measured how the claude-sonnet-5 model answers company-register questions with and without access to a new MCP tool called registry-mcp, which unifies Norway's Brønnøysundregistrene, the UK's Companies House, and Sweden's Bolagsverket. Without tools, the model was correct on 8 of 45 audited calls, confidently wrong on 3, and honestly hedged on 34, while with tools attached it passed 18 of 27 eligible cases (67%) under a strict all-three-trials rule. The developer noted that beating a model that already says it cannot check a live register is not a meaningful win.", "body_md": "`registry-mcp` puts three national company registers behind one MCP tool — Brønnøysundregistrene / Enhetsregisteret (**brreg**) by **organisasjonsnummer** (orgnr, org.nr), Companies House by company number, and Bolagsverket by organisationsnummer. The pitch has always rested on a counterfactual: without a register, an agent answers company questions confidently and wrongly. That is an easy thing to assert in a README. On 2026-09-10 we measured it, and the answer is not the one the pitch would have picked.\n\nTwo arms, one model — `claude-sonnet-5` — on the same day, over the same hand-written prompts.\n\n`tools` and `system` omitted from the API call entirely. Three trials each, so 45 calls.\nCall by call, manually audited: **8 correct, 3 wrong (confident, dangerous), 34 honest hedges, 0 unclear.**\n\nThe hedges are the part nobody would have put on a slide. Per case, the model **hedged on every single trial for 9 of the 15 (60%)** — \"I can't check a live register\", and nothing more. It was **confidently and dangerously wrong on at least one of three trials for 2 of 15 (13%)**, and produced a **correct-but-unsourced answer on at least one trial for 4 of 15 (27%)**.\n\nSo the honest version of our own pitch is: a current model mostly refuses. It does not mostly lie. That refusal is good behaviour, and we do not score it as a win for the product — beating a model that already said \"I don't know\" is not beating anything.\n\nTwo caveats belong on those numbers rather than in a footnote. First, that is the **audited** count, not the harness's. The automatic classifier's raw first pass was **6 correct, 7 wrong, 27 hedge, 5 unclear**; the manual read moved **12 of the 45 calls**, every one of them a failure mode this project had already named once — the classifier cannot tell an assertion about *this* company from the same words used generically, as in \"status (active, dissolved, etc.)\". Second, three trials smooths sampling noise. It does not remove it.\n\n\"3 of 45\" is abstract. These are not.\n\nOn one trial, asked what a Norwegian company owed next, the model invented a detailed obligation set with no hedge anywhere in it: an SEC Form 20-F \"due ... by 30 April 2026\", an Oslo Børs Q4 release \"early February 2026\". Neither is the register-derived Norwegian deadline the case is about. The real one is `2026-07-31`.\n\nOn two of three trials, asked for a headcount, the model disclaimed access to the register and then gave \"330,000–360,000 employees\" anyway, as general knowledge. A caveat followed by a specific number is exactly the shape an agent pipeline strips the caveat off.\n\nSame model, same day, tools attached. **18 of the 31 cases passed outright (58%)** under the harness's strict rule, which requires all three trials to agree before a case counts as passed at all. Four of the 31 never get scored — two are live-only smoke cases skipped by design, two hit a gap in the mock coverage that has nothing to do with the model — so against the **27 cases actually eligible, that is 18/27 (67%)**.\n\nBefore the strict rule is applied, the raw per-trial rates were **19/27 (70%), 23/27 (85%) and 20/27 (74%)**, a mean of **62/81 (77%)**. An eighteen-percentage-point spread between the best and the worst trial, on identical cases with nothing else changed, is why the strict rule exists — and why one trial of anything is not evidence.\n\nOne case is kept failing deliberately, and it is the most useful line in the report. Asked whether a UK company is VAT-registered, the model called `search_company` to confirm the company's identity, then answered \"Companies House ... doesn't publish VAT registration data\" — correct, and never checked against that record's actual `vat_registered: null`. The search result carries no VAT field at all, so the only way to answer it *from the tool* is to look the company up and read the null. Pass rate: 1 of 3.\n\nThe no-tools arm found the same behaviour from the other side the same day. On that same case it answered \"Yes, Tesco PLC is VAT-registered ... since its taxable turnover far exceeds the threshold\" — confident, reasoned, unsourced. Same answer shape; one of them merely had a tool available to skip.\n\nWhich is the thing a register actually buys, and it is not accuracy. It is provenance: the difference between an answer that happens to be right and an answer with a record behind it.\n\n`tests/fixtures/*.json`, not the live registers. This measures whether a model uses the tool surface correctly — not brreg's, Companies House's or Bolagsverket's own data quality, coverage or uptime.\nThe caveat sentence to carry away is the report's own: this measures whether one named model, on one day, answering 31 questions someone wrote by hand against mocked data, uses this tool correctly and knows the difference between \"the register says no\" and \"the register doesn't say\". It says nothing about a harder question, a different model, or the real registers' own accuracy.\n\nThe no-tools half cost $0.20 to produce — **45 calls, 1,281 input and 19,520 output tokens** at list pricing. The harness, the cases and both reports are in the repo, so the disagreement can be with the data rather than with us.\n\n```\nclaude mcp add registry-mcp --transport http \"https://api.foretak.dev/mcp?src=devto\"\n# or locally, over stdio: uvx registry-mcp\n```\n\nHarness, cases and reports (`evals/`), MIT: [https://github.com/foretak/registry-mcp](https://github.com/foretak/registry-mcp)", "url": "https://wpnews.pro/news/how-wrong-is-a-model-about-a-company-when-it-has-no-register-to-check", "canonical_source": "https://dev.to/fargeroddotcom/how-wrong-is-a-model-about-a-company-when-it-has-no-register-to-check-27c2", "published_at": "2026-09-10 08:16:03+00:00", "updated_at": "2026-09-10 08:22:15.089791+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "large-language-models", "ai-research", "developer-tools"], "entities": ["registry-mcp", "claude-sonnet-5", "Brønnøysundregistrene", "Companies House", "Bolagsverket", "Tesco PLC", "Oslo Børs"], "alternates": {"html": "https://wpnews.pro/news/how-wrong-is-a-model-about-a-company-when-it-has-no-register-to-check", "markdown": "https://wpnews.pro/news/how-wrong-is-a-model-about-a-company-when-it-has-no-register-to-check.md", "text": "https://wpnews.pro/news/how-wrong-is-a-model-about-a-company-when-it-has-no-register-to-check.txt", "jsonld": "https://wpnews.pro/news/how-wrong-is-a-model-about-a-company-when-it-has-no-register-to-check.jsonld"}}