# The confidence number is the product (and here is ours, with the caveat)

> Source: <https://dev.to/turingcorp/the-confidence-number-is-the-product-and-here-is-ours-with-the-caveat-4mmk>
> Published: 2026-09-18 13:35:40+00:00

Most AI tools answer the question. The harder job is judging between two answers you already have: two drafts, two plans, two offers. You usually lean one way already — what you cannot do is price your own uncertainty.

That is the only thing we built: **Decider**, a judge that takes a question plus two candidate answers and returns the stronger one, a calibrated confidence value, and the reasoning for the call.

Here is the part most products would leave out. On raw pick rate, Decider **ties** a plain single-model baseline — roughly 92.5% against 92.7% on JudgeBench. We do not claim to be more accurate.

What a single pass cannot give you is *how close the call was*. Every judgment carries a confidence value calibrated against outcomes:

| Confidence reported | Share of judgments | Observed accuracy | 
|---|---|---|
| ≥ 90% | 45.6% | **99.6%** | 
| 80–90% | 29.7% | 94.0% | 
| 70–80% | 13.2% | 84.1% | 
| < 70% | 10.5% | 67.7% | 

JudgeBench, 614 judgments. These are our own runs on the official protocol; the runs that failed are disclosed rather than dropped, and the raw judgments are published.

A high band is permission to commit and stop re-reading both options. A low band is not a failure — it is the answer you actually needed: the two are genuinely close, so take the one you already preferred and spend your time somewhere else. Terminating the comparison is the value.

We recorded 27 real runs across nine domains — writing, engineering, business, research, career, money, people, travel, everyday — and published each one end to end: the question, both candidate answers, the pick, the confidence and the full reasoning, verbatim. They are browsable as static pages:

[https://api.turingcorp.net/platform/poe/examples](https://api.turingcorp.net/platform/poe/examples)

Nothing is charged for reading them, and the same set ships inside the app as free recorded runs.

Decider runs as an app on Poe: **[https://poe.com/TuringCorp-Decider](https://poe.com/TuringCorp-Decider)** — pay per decision, no subscription (list $0.50, launch offer $0.25). What it is, how it is measured and what it costs: [https://api.turingcorp.net/platform/poe](https://api.turingcorp.net/platform/poe)

If your pipeline needs a quality gate rather than a chat, there is also an OpenAI-compatible endpoint: [https://api.turingcorp.net/](https://api.turingcorp.net/)

*Disclosure: this post is written by the team that builds Decider.*