At a 0.9 Threshold, Three Decision APIs Implied Under 5% Error. Five Annotators Said 17–37%. A pre-registered test on 300 SNLI sentence pairs, each labeled by five annotators, found that Google's Gemini Decisions, OpenAI Decisions and TypeSafe's Jev 1.13.0 implied under 5% error at a 0.9 threshold on the pairs they kept, yet were wrong 17% to 37% of the time when scored by the share of annotators who chose another label. On the 100 pairs where the five annotators split, Gemini and Jev still expected about 5 points more support than the annotators gave and OpenAI about 7, each outside the range a calibrated model would land in, according to the author's pre-registered correction. The author, who reports no relationship with Google, OpenAI, TypeSafe or Anthropic beyond using their products, notes all three vendors advise setting thresholds on labeled data from one's own traffic. Google’s Decisions API documentation lists “Calibrated probabilities you can threshold” as a reason to use it. So I put a 0.9 threshold on the top probability from three decision APIs, Google’s Gemini Decisions, OpenAI Decisions and TypeSafe’s Jev, and scored what got through on 300 SNLI sentence pairs, each labeled by five annotators. Reweighted to SNLI test frequencies, all three APIs’ own numbers implied under 5% error on what they kept. Counted by the share of the five annotators who chose another label, they were wrong 17% to 37% of the time. Those are descriptive numbers. I also pre-registered a narrower test, on the 100 pairs where the five annotators split: did each API expect more agreement than it got? Asked the obvious way, that test convicts even a perfectly calibrated model, because pairs picked for disagreement pull the vote down. The pre-registered version corrects for how the pairs were picked, and with it Gemini and Jev still expected about 5 points more support than the annotators gave, and OpenAI about 7, each outside the range a calibrated model would land in. The correction assumes the five labels on each pair are interchangeable. The data suggest they aren’t quite; the evidence is below. I have no relationship with Google, OpenAI, TypeSafe or Anthropic beyond using their products. I paid for the OpenAI and TypeSafe calls myself; Gemini’s ran on its free tier and the Claude arm on my own subscription. Google’s Decisions API documentation says it “returns calibrated probabilities.” TypeSafe’s docs, on the probability of the option your code will act on: “It’s the probability that the chosen option is correct, so a threshold on it is a statement about risk that anyone can read.” Jev, per TypeSafe’s model page, is “trained with RLCD to return calibrated decisions.” OpenAI’s Decisions guide says a yes/no predicate answer’s probability "is the model's estimate that the condition is true," and that choice answers "return a probability distribution and a separate confidence field." It never uses the word "calibrated." All three vendors tell you to set or test thresholds on your own data. Google: “Set thresholds using labeled examples from your own traffic.” OpenAI: “Use labeled examples from your application to set thresholds for routing, filtering, or review.” TypeSafe: “Start with conservative thresholds, test with your own data, and adjust as you observe results.” This piece tests their out-of-the-box behavior on one public benchmark. Four arms, one question each per sentence pair: I pinned Jev 1.13.0 because JevBench, a leaderboard Benchmark Heaven publishes independently of TypeSafe, scores its calibration slightly above the newer 1.14.0’s 90.6 against 90.1 on the board’s own scale ; I didn’t test 1.14. Every arm got the same input Premise: ... and Hypothesis: ... , the same instruction, "What is the relationship between the premise and the hypothesis?", and the same three options: contradiction: The hypothesis is definitely false given the premise.entailment: The hypothesis is definitely true given the premise.neutral: The hypothesis might be true or false given the premise. Option order needs care. Jev’s docs say that in some cases “the order of a Choice’s options can affect the answer, and Jev leans toward the option that comes first.” Kartik Anand tested all six orders of three choices and found the share of cases that kept the same route across all six was “91.6–91.8%” for Jev and “64.2%” for OpenAI Decisions, which Anand reached through OpenRouter. Gemini’s API sorts option keys alphabetically whatever order you send. For the other three arms I shuffled the order once per pair, with a fixed seed, and kept it the same in every pass. That can show a lean toward the first option on average, though not whether a given answer survives a reorder. If the order made no difference, the top choice would be the first-listed option a third of the time. It was 38.3% for OpenAI, 33.7% for Jev and 33.0% for Claude, and each one’s 95% interval includes a third. The 300 pairs are from the SNLI 1.0 test set: 100 where all five annotators agreed, 100 where four of five did, and 100 where three of five did. Of those last 100, 74 split 3 to 2 and 26 split 3–1–1. Each arm went through all 300 three times on November 16, 2026. That’s 3,600 calls, and every one returned a valid answer on its first try as far as my harness could see; it never retried. Pass 1 is the scored pass. Passes 2 and 3 are there for replication and for stability. The design, the seeds, the thresholds and the analysis code were frozen before the first study call. The frozen manifest records the seeds and thresholds and pins the design, the analysis code, the 300 pairs and the archived vendor pages by SHA-256, and its own SHA-256 is e53d85dc39f42a5a6932877825c002eeb63a421faecd87b47e488eb5ebc4581f. The commit that froze it is timestamped 11:13 UTC on November 16, and the first study call went out seconds later. Pre-registered here means frozen in my own repository before the first call, not filed with an outside registry. Afterward, a second implementation of the analysis, written from the design document without access to my analysis code, rebuilt the results from the raw response bytes. The two agreed on all 65,515 values compared. I’ll send the manifest, the design document, the raw responses and the analysis code to anyone who asks through davidjknudson.com https://www.davidjknudson.com/ . The hash lets you confirm that the manifest you get is the one this article describes. It can’t show when the manifest was written; the commit time is my own repository’s record. For each threshold, keep the pairs whose top probability is at that threshold or higher, then compare three error rates on what you kept: Everything in it is reweighted to SNLI test frequencies. The test set has far more unanimous pairs than split ones, so the 5 of 5 level carries 54.7% of the weight, 4 of 5 carries 29.2% and 3 of 5 carries 16.1%. On average, that undoes my equal-thirds sampling. Here is the 0.9 row in the sentence form I fixed before the study, where “above 0.9” means a top probability of 0.9 or more and “promised at most” is the promised error defined above: I picked 0.9 because it’s a round number people actually use; Figure 2 has all six thresholds. For each of the three decision APIs, the promised error sat below the error against these five annotators at every threshold where it kept anything. Against the SNLI majority label, the errors at 0.9 were smaller but still above the promised figure: 8.7% for Gemini, 30.1% for OpenAI and 8.5% for Jev. Claude, the reference arm, on the same basis: above 0.9, Claude promised at most 7.2% wrong and was wrong 8.6% of the time against these five annotators. Take Gemini on the 100 split pairs. Its mean top probability was 0.817, and these five annotators gave its choice 0.518 of their votes. Read naively, that’s 30 points of overconfidence. But Gemini’s own probabilities, run through the way I picked those pairs, expected only 0.568. So about 25 of the 30 points come from how the pairs were picked, and about 5 remain: Gemini’s own gap. OpenAI’s raw gap of about 39 points and Jev’s of 34 shrink the same way, to about 7 and 5. Here’s why the picking matters. The split pairs were chosen because their five labels split, and that drags the annotators’ share down, even for an arm that knows the true odds exactly. An item whose wider-population split is 0.8, 0.1, 0.1 lands in 5/5 with probability 0.328, in 4/5 with 0.411 and in 3/5 with 0.221, and has no majority with 0.041. Its expected share for the 0.8 label is 1.000, 0.799 and 0.582 in those strata. So an arm that says 0.8, and is exactly right, should expect only about 0.58 of the vote on a pair that landed in the 3 of 5 group. Compare its 0.8 with the annotators’ 0.6 and you’d call it overconfident when it isn’t. Before the freeze I ran the naive version of the test on 2,000 simulated studies built like this one, with an arm whose probabilities were exactly right. It rejected that arm at the 0.05 level in all 2,000 and in 1,868 of 2,000 with a flatter mix of items . The fix is to ask what each arm’s own probabilities predict for a pair that landed in its group. For every pair I enumerated all 243 ways five labels could fall, kept the ones that produce that pair’s agreement level, and weighted them by the arm’s own probabilities, with a small floor, ε, under each one. That gives e: the share of the five the arm should expect for its choice, given how the pair was chosen. Call the share the annotators actually gave y. The headline statistic, θ, is the mean of e minus y: the average confidence gap on split items, after allowing for how those items were chosen. Positive θ means the arm expected more support from these five annotators than they gave. On the same 2,000 simulated studies, this version rejected the exactly-right arm 78 times at the 0.05 level, about 4%, inside the 3.5% to 6.5% window I set in advance. It caught an arm built to be overconfident 1,988 times. On the 100 split pairs, θ was positive for all three: A gap of 0.05 is five points of annotator support per pair, or one annotator’s vote in every four pairs. This result depends on an assumption: the test treats the five annotators’ labels on each pair as exchangeable. In a pre-freeze simulation with one version of a first-label effect, the test rejected an exactly calibrated arm at the 0.05 level in 453 of 2,000 studies with an SNLI-like mix of pairs, and in 511 of 2,000 with a flat mix. These pairs show such an effect: the first of the five labels agrees with the majority on 92 of the 100 pairs at 4 of 5, where 80 would be expected if the five were interchangeable, and on 72 of the 100 at 3 of 5, where 60 would be. The simulation can’t say how often the effect alone would push p as low as 0.000020, and I don’t know how strong the real effect is. The test drew 100,000 sets of labels from each arm’s own probabilities, conditioned on each pair’s agreement level. None of the draws reached the observed gap for any of the three, which gives a two-sided p = 0.000020, the smallest value 100,000 draws can return. Holm’s correction across the three left all three standing. I also wrote down four conditions that could only take a finding away, and all three passed each one: A note on d : Gemini and OpenAI returned identical probabilities in all three passes more on that below , so for them replication could only pass. Claude’s row is there for reference: 0.031, where a calibrated arm would land between −0.015 and 0.019. It sat outside the test family before any data existed, so I make no claim about it either way. Two limits on what θ means. It’s an average, so it is not a full calibration score: an arm that is overconfident on some pairs and underconfident on others can average out near zero. And the three gaps aren’t ranked against each other; the 4 of 5 and 5 of 5 rows are descriptive, shown with their bands and no test. Figure 4’s 5 of 5 rows, which I didn’t test, show bigger gaps than the tested ones: 0.148 for Gemini and Jev, and 0.402 for OpenAI. That’s the selection correction working the other way. On the pairs where all five agreed, each arm’s own probabilities predicted, on average, 0.97 to 0.99 of the vote for its choice, so a miss counts almost in full. As with Gemini’s and Jev’s smaller gaps, OpenAI’s 0.402 is mostly the unanimous contradictions it called neutral, which I come to below. The secondary measure is plain SNLI majority-label correctness: did the top choice match the label most annotators gave? Reweighted to SNLI test frequencies, it was 81.6% for Gemini, 63.7% for OpenAI, 82.6% for Jev and 88.8% for Claude. It’s descriptive, with no test attached. This tally wasn’t part of the pre-registered analysis; I counted it from the per-item results afterward. On the 100 pairs where all five annotators agreed, OpenAI’s top choice matched them 59% of the time. Of the 44 unanimous contradictions, OpenAI called 39 “neutral,” which accounts for 39 of its 41 misses on those pairs. The other arms called some of the same pairs neutral: Gemini 18 of the 44, Jev 14 and Claude 5. Here’s one of the 39: Premise: Cheerleaders standing on a football field. Hypothesis: Cheerleaders are standing on a bridge. All five annotators said contradiction. OpenAI put 0.96 on neutral, 0.04 on contradiction and 0.00 on entailment, the same numbers on all three passes. Gemini, Jev and Claude each put 0.9 or more on contradiction. You can send the pair yourself: it’s SNLI test pair 2872819024.jpg 2r1c, with the input, instruction and options printed above, sent as a choice question named nli, and OpenAI saw the options in the order contradiction, neutral, entailment. My parsing didn’t cause this. The harness reads OpenAI’s probabilities by label name, never by position, and the independent reimplementation, reading the raw responses on its own, got the same top choice on all 3,600 answers. I don’t know why OpenAI’s model does this. This is stability observed over three calls in the window, and the mechanisms differ, so the rows don’t sit on one scale. Gemini and OpenAI returned exactly the same probabilities, all three times, on all 300 pairs. With no flips in 300 pairs, the one-sided 95% upper bound on the flip rate is 1.0%. Gemini’s API “generates zero text tokens,” in Google’s words. OpenAI’s API has a reasoning-token count in its usage. On all 900 of my calls it read 0, as did its output tokens. Gemini doesn’t report which model version answered, so an update inside the window would be invisible to me. Jev’s probabilities moved on 55.3% of the pairs, and its top choice flipped on 1.3% of them. On the median pair, the largest change between two calls was 0.01. TypeSafe reports run-to-run changes too: on its own benchmark of borderline moderation posts, its cookbook says “a post’s least stable decision changes 4.0% of the time on average” for Jev 1.14. That’s a different version, task and measure, so I don’t set the two numbers side by side. Claude is sampled text. Its numbers moved on 80.3% of pairs, and its top choice flipped on 4.3%. Across the three passes it used 85 distinct whole-number percentages. On Jev, two audits scored it against other targets. TypedBench https://www.alphaxiv.org/abs/2610.11392 , posted in October by Rahul Sharma and colleagues, found the hosted Jev 1.13.0, the version I pinned, “systematically underconfident” on items generated from stated facts and policies, whose labels are exact and whose wording isn’t natural. Lingsen Meng’s 2026 audit https://escholarship.org/uc/item/4t4449jv scored Jev 1.13.0 against the gold labels of two public classification tasks and found that its calibration shifted with how a question was posed, and that on a four-class news task its answers below a top probability of 0.93 were overconfident as a group. In this study, I scored Jev against five annotators’ votes on natural sentences. Most of the other tests I read scored these APIs against a known answer or an outcome. Divye Singh https://pub.towardsai.net/can-jev-replace-the-llm-judge-measuring-correctness-before-speed-7118e9008389 found Jev well calibrated on judging items with a known correct answer, and noted that the HelpSteer2 ratings “come from human annotators who do not always agree,” the case this piece measures. Ankit Agrawal https://pub.towardsai.net/jev-vs-clef-vs-openai-decisions-api-i-tested-six-decision-apis-on-700-items-05404e886126 ran 700 items with ground-truth labels and found OpenAI Decisions overconfident on both tasks checked for calibration: on tool gating, “it averaged 0.96 confidence while being right 82% of the time.” Others used trading outcomes Abozar Alizadeh https://pub.towardsai.net/17-days-with-jev-i-put-a-sub-second-system-one-ai-on-a-real-trading-account-c851c8eba1a1 , synthetic cases Kartik Anand https://medium.com/@kartikanand 11915/jev-vs-openai-decisions-vs-microsoft-decision-1-same-cases-same-harness-no-universal-winner-217eb0cae2de , situations with a known answer Synthpop.AI https://www.synthpop.ai/resources/does-a-decision-model-know-when-it-is-guessing-post-1 and BANKING77’s gold labels Ankur Tomar https://towardsdatascience.com/scikit-learn-matched-or-beat-jevs-calibration-on-banking77/ , and Ali Süleyman Topuz https://pub.towardsai.net/a-decision-model-as-a-gate-fast-well-calibrated-and-still-wrong-16-of-the-time-2ddb226b1ed5 set out a pre-registered gate test of Jev without running it yet. System1-Arena https://arxiv.org/abs/2610.21244 , posted in October by Tianyu Chen and colleagues, comes closest. It ran Jev 1.13, OpenAI Decisions and other models across 105 tasks, and its Brier score targets “an annotator distribution” where a source has one, ChaosNLI’s SNLI and MNLI pairs with 100 annotations each among them. It reports the Brier score as one average over its 87 non-agentic tasks, and I found no result for those pairs on their own. Of the tests I found, it is the only one that scored a decision API against a split among several annotators, and only inside that average. None of these tested Gemini Decisions. On November 18, Yaroslav’s comparison on Towards AI https://pub.towardsai.net/googles-gemini-decisions-api-compared-with-microsoft-decision-1-jev-and-openai-85bcebb8125d reported Gemini Decision 1’s score on Artificial Analysis’s Decision Index https://artificialanalysis.ai/methodology/decision-index , which scores a model’s probabilities against a correct answer on one of its six evaluations, a reference answer built from two language models’ probabilities and by whether they hold when a question is reworded, not against how annotators split. Maja Pavlovic’s soft-label MNIST testbed https://towardsdatascience.com/a-soft-label-mnist-testbed-for-human-vs-model-confidence/ offers a related target for handwritten digits, averaged over three to ten annotators per image who could each mark more than one digit, but tests no decision API. ChaosNLI’s 100 labels per pair make a finer target than SNLI’s five. I stayed with the 300 SNLI pairs from my October study, which used SNLI because ChaosNLI is licensed for non-commercial use only. On these 300 pairs, the ordering held: for each of the three, every higher threshold that kept anything meant fewer errors against these five annotators. Use the number to sort your cases. Take the error rate from your own hard ones. I run my own AI tooling on a Mac mini and write up what measuring it turns up, from cross-vendor model panels to decision APIs. More at davidjknudson.com , where you can also ask for this study’s materials. The Google, OpenAI and TypeSafe doc pages the study relied on were archived when the design was frozen on November 16, 2026, except TypeSafe’s consistency cookbook. All of them, and JevBench, were re-checked on November 17, 2026. TypedBench, System1-Arena and the Singh, Agrawal, Alizadeh, Anand, Synthpop, Topuz, Tomar and Pavlovic pieces were read on November 17, 2026; Yaroslav’s comparison, the Artificial Analysis methodology and Meng’s preprint on November 18, 2026. At a 0.9 Threshold, Three Decision APIs Implied Under 5% Error. Five Annotators Said 17–37%. https://pub.towardsai.net/at-a-0-9-threshold-three-decision-apis-implied-under-5-error-five-annotators-said-17-37-70a55bf1abf0 was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.