This week the US military had a close call: it used an AI-generated intelligence report that was hallucinated, and the error nearly drove a real decision. In the same news cycle, a top model solved a century-old cipher — impressive, and beside the point.
Both stories are about the same thing. Models have gotten very good at being right impressively often. They have not gotten better at knowing when they're wrong. And in high-stakes work, the second skill is the one that keeps the lights on.
Every headline you read about AI capability is a benchmark: this model scores X on reasoning, Y on code, Z on math. Nobody benchmarks the thing that actually decides whether you lose money — confidently wrong output that looks exactly like confidently right output.
A model that's right 95% of the time and flags its 5% is safe to deploy. A model that's right 97% of the time and states its 3% with total conviction is a liability. The difference never shows up in a score. It shows up in a spreadsheet three weeks later.
For a cross-border seller, the hallucination isn't an abstraction. It's a product listing with a fabricated spec. A tax code cited from a law that doesn't exist. A customer reply promising a policy you never had. An automation script that "handles refunds" by inventing a refund. Stop asking "can it do this?" Ask "how would I know if it got this wrong?"
If you can't answer the second question cheaply, the tool isn't ready for the stakes. That single filter reorganizes everything: The instinct is to demand a smarter model. The durable move is to design for distrust. Assume every output is wrong until something independent says otherwise. Then spend your trust budget where the blast radius is small.
A model that solved a WWI cipher is a nice demo. A model that knows the limits of its own certainty is a business asset. Only one of those two shows up in the benchmark.
Confidence is not accuracy — it's just accuracy's most convincing forgery. Build the check before you build the trust.