AI JEV | Calibration Is the Feature: What "90% Confidence" Actually Has to Mean

A theoretical/practical deep dive into the math behind Jev's confidence scores — and how to check whether your own integration is honest about what it doesn't know.

The thing nobody tells you about confidence scores

Every Jev call comes back with a confidence field. Most integrations treat it the same way they'd treat a gut feeling from a colleague: high number, trust it; low number, escalate. That's a reasonable first instinct, and it's also not remotely enough to build a production system on — because a confidence score is a claim, and claims can be wrong in a specific, measurable, fixable way that has nothing to do with whether the underlying decision was right or wrong.

This is the part that doesn't show up in a quickstart guide: a confidence score is only useful if it's calibrated, and calibration is a property you have to go check for yourself, on your own data, because it's specific to your criteria, your domain, and how your state object is built. Nobody — not TypeSafe AI, not this article — can hand you a guarantee that your Jev integration's 90% is an honest 90%. You have to measure it.

This post is about what "calibrated" actually means mathematically, why an accurate model can still be badly calibrated, and how to build the diagnostic tooling yourself in about 40 lines of Python.

Accuracy and calibration are different axes

Here's the distinction that trips people up first: a decision system can be highly accurate and badly calibrated at the same time. These are not the same measurement, and optimizing one doesn't automatically fix the other.

Accuracy asks: of all the decisions made, how many were correct? Calibration asks a subtler question: of all the decisions made at confidence level X, how many were correct? A system can get 95% of decisions right overall while being wildly overconfident on the 5% it gets wrong — reporting 92% confidence on calls it's actually right about only 60% of the time. That gap is invisible if you only track overall accuracy. It becomes very visible the first time someone downstream trusts a 92%-confidence decision that was actually a coin flip.

This matters specifically for the confidence-gated routing pattern — the one where you send high-confidence decisions straight through and route low-confidence ones to a human or a more expensive fallback. The entire pattern's safety property depends on the confidence number meaning what it says. If your threshold is 0.85 and your system's actual accuracy at reported-confidence-0.85 is 70%, you are silently auto-approving a much riskier slice of traffic than your threshold implies.

Defining calibration precisely

A decision system is perfectly calibrated if, for every confidence level p it reports, the decisions it makes at that confidence level are correct exactly p fraction of the time. Formally, for a binary correctness outcome:

P(correct | confidence = p) = p for every p

Nothing is ever perfectly calibrated in practice — this is a target you converge toward, not a bar you clear once. What you actually compute is a deviation from this ideal, and there are two standard tools for it: the reliability diagram and the Expected Calibration Error (ECE).

The reliability diagram

You bucket every decision by its reported confidence (typically into 10 bins: 0.0–0.1, 0.1–0.2, … 0.9–1.0), then for each bucket you plot two numbers: the average reported confidence in that bucket, and the actual accuracy of decisions in that bucket. Perfect calibration is the diagonal line — reported confidence equals actual accuracy, bucket by bucket. Any bucket sitting below the diagonal means the system is overconfident in that range (it says 80%, reality says 65%). Any bucket above the diagonal means it's underconfident (it says 60%, reality says 78% — less dangerous, but it means you're routing decisions to expensive fallback paths that didn't need it).

Expected Calibration Error (ECE)

The reliability diagram is qualitative; ECE turns it into a single number you can track over time and alert on. It's the weighted average gap between confidence and accuracy across all bins:

ECE = Σ (n_b / N) * |accuracy(b) - confidence(b)|

where n_b is the number of decisions in bin b, N is the total number of decisions, accuracy(b) is the actual correctness rate in that bin, and confidence(b) is the average reported confidence in that bin. Lower is better; 0 is perfect. As a rough field guide, an ECE under 0.03–0.05 is generally considered well-calibrated for a production decision system; above 0.1 means the confidence field is actively misleading whatever's consuming it.

Building the calibration checker

Here's the whole thing, assuming you're logging every Jev decision with its reported confidence and — eventually, once ground truth becomes available (a human review, a downstream outcome, a dispute resolution) — whether it was actually correct.

python

import numpy as np

def compute_calibration(confidences, correctness, n_bins=10):
    """
    confidences: array of reported confidence values, one per decision, in [0, 1]
    correctness: array of 1/0, whether each decision was actually correct
    Returns per-bin stats and the overall Expected Calibration Error.
    """
    confidences = np.array(confidences)
    correctness = np.array(correctness)
    bin_edges = np.linspace(0.0, 1.0, n_bins + 1)
    bin_stats = []
    ece = 0.0
    n_total = len(confidences)

    for i in range(n_bins):
        lo, hi = bin_edges[i], bin_edges[i + 1]
        in_bin = (confidences >= lo) & (confidences < hi if i < n_bins - 1 else confidences <= hi)
        n_bin = in_bin.sum()
        if n_bin == 0:
            bin_stats.append({"range": (lo, hi), "n": 0, "avg_confidence": None, "accuracy": None})
            continue

        avg_confidence = confidences[in_bin].mean()
        accuracy = correctness[in_bin].mean()
        gap = abs(accuracy - avg_confidence)
        ece += (n_bin / n_total) * gap

        bin_stats.append({
            "range": (round(lo, 2), round(hi, 2)),
            "n": int(n_bin),
            "avg_confidence": round(float(avg_confidence), 4),
            "accuracy": round(float(accuracy), 4),
            "gap": round(float(gap), 4),
        })

    return bin_stats, round(float(ece), 4)

def print_reliability_report(bin_stats, ece):
    print(f"{'Range':<12} {'N':>6} {'Avg Conf':>10} {'Accuracy':>10} {'Gap':>8}")
    for b in bin_stats:
        if b["n"] == 0:
            continue
        lo, hi = b["range"]
        print(f"{lo:.1f}-{hi:.1f}   {b['n']:>6} {b['avg_confidence']:>10.3f} "
              f"{b['accuracy']:>10.3f} {b['gap']:>8.3f}")
    print(f"\nExpected Calibration Error (ECE): {ece}")
    if ece < 0.03:
        print("→ Well calibrated.")
    elif ece < 0.10:
        print("→ Mild miscalibration — worth investigating which bins are worst.")
    else:
        print("→ Significant miscalibration — do not trust confidence-gated routing until this is fixed.")

Feed it a batch of logged decisions once you have ground truth for them:

python
confidences = [0.95, 0.91, 0.60, 0.88, 0.55, 0.72, 0.97, 0.63, 0.81, 0.90, ...]
correctness  = [1,    1,    0,    1,    1,    0,    1,    0,    1,    0,   ...]

bin_stats, ece = compute_calibration(confidences, correctness)
print_reliability_report(bin_stats, ece)

The output tells you exactly where the problem lives — not just "our confidence is bad" but "our confidence is bad specifically in the 0.8–0.9 range," which is actionable in a way a single accuracy number never is.

Where miscalibration actually comes from

Once you've measured a gap, the useful next question is why. Three sources show up repeatedly in practice:

Criteria drift. The written criteria a decision was calibrated against no longer match the traffic hitting it. A support-triage category written six months ago for one product line doesn't cleanly cover the edge cases a new product line introduces — the model is confidently applying old boundaries to new territory.

Bin sparsity. If only 40 decisions ever land in the 0.95–1.0 bucket, your accuracy estimate for that bucket has a wide confidence interval of its own — don't over-react to a single bad-looking bin without checking n first. This is the most common false alarm in a first calibration report.

Distribution shift between your golden set and live traffic. If the ground-truth examples you calibrated against skew toward easy, unambiguous cases — which hand-labeled sets often do, because ambiguous cases are exactly the ones humans disagree on and therefore tend to exclude — your live-traffic calibration will look worse than your offline evaluation did, for the boring reason that live traffic is harder than your test set.

A practical calibration workflow

Put this together into something you actually run on a schedule, not just once at launch:

Log every decision with its confidence and enough context to determine ground truth later (a human review outcome, a downstream signal, a dispute).

Batch ground-truth collection weekly, not per-decision — sampling a representative slice is enough; you don't need ground truth for every single decision, you need enough per bin to make the accuracy estimate meaningful (aim for at least 30-50 per bin before trusting it).

Compute ECE and the full reliability table every week, and alert if ECE crosses your threshold or if any high-traffic bin's gap exceeds roughly 0.15.

When a bin is miscalibrated, don't just lower the routing threshold — that treats the symptom. Go look at why that bin is wrong: pull ten decisions from it and read them. Criteria drift and bin sparsity look different once you actually read the cases.

Re-baseline after every criteria change or model version pin change — a calibration report from before a criteria update tells you nothing about the system running today.

The takeaway

A confidence field is not a fact about the world — it's a claim the model is making about itself, and like any claim, it can be checked. Confidence-gated routing, cascades, and every other pattern that reads a confidence number and makes a decision based on it are only as trustworthy as that number is calibrated. Computing ECE and a reliability table isn't optional infrastructure for a Jev integration running anything that matters — it's the actual test of whether the "confidence" in "confidence-gated routing" means anything at all.