Beyond Accuracy · Model Evaluation (#4)
This is what building an honest evaluation layer taught me. Right now it runs on a synthetic staging model — I generate the patients, so I know the ground truth exactly, which is the cleanest way to study a measure like calibration before trusting it on anything real. The vision is to grow it toward the fuller, multimodal picture — imaging and clinical together — and onto real data like the Parkinson’s set. Everything below came out of that work.
A model I was evaluating scored 88% accuracy on a four-stage clinical staging task. Respectable. Then I looked at the mistakes it did make — and 44% of them were made while the model was more than 80% sure of itself. It wasn’t just wrong sometimes. It was confidently wrong, and nothing in the accuracy number had warned me.
That gap — between how sure a model sounds and how often it’s actually right — has a name. It may be the most important thing accuracy can’t measure, and once you’ve seen it, you can’t unsee it. It’s called calibration.
This piece is the long version: why we calibrate, what calibration is (and the things people wrongly think it is), what exactly it measures, how you compute it, what it quietly hides, and how — carefully — you fix it.
In the real world, you rarely act on the label alone. You act on the confidence attached to it.
A radiology model that flags a scan at 51% sends the patient for another test. The same model at 99% might start treatment. A fraud score of 51% earns a second look; at 99% it freezes the account. A chatbot that hedges makes you check its work; one that answers in a confident, fluent voice makes you trust it. In every case, the predicted class can be identical — the confidence decides the next move.
So if that confidence is dishonest, every decision built on top of it is dishonest too. Calibration is the property that makes a confidence number safe to act on.
Accuracy asks did the model get it right? Calibration asks was it honest about how sure it was? Those are two different questions — and most evaluations only answer the first.
Here’s the clean definition, borrowed from the most calibrated people on earth: weather forecasters. When a forecaster says “70% chance of rain,” it should actually rain on about 70% of the days they say that. Their confidence is earned. A model is well-calibrated when the same holds: of all the times it says “90% sure,” it’s right about 90% of the time — not 100, not 60.
Now the misconceptions, because each one is a trap I’ve watched smart people fall into:
It is not accuracy. A model can be accurate and badly miscalibrated (right often, but wildly over- or under-sure), or perfectly calibrated and useless (predict the base rate for everyone — always “honest,” never helpful). Accuracy and calibration are two different axes. You want both.
It is not a measure of whether the model is right. It’s a measure of whether the model’s sureness is trustworthy. A calibrated model that says “60%” and is wrong hasn’t failed at calibration — it told you not to lean on it.
A low error score does not mean “safe.” As you’ll see below, the standard one-number calibration score can look excellent while a pocket of confident, wrong, high-stakes predictions hides inside it.
Recalibrating does not always help. Post-hoc “fixes” can make calibration worse. I have the receipts.
Before we can ask whether a model’s confidence is honest, we should be clear on what “confidence” even is — because it isn’t a feeling the model has. It’s a number it computes, and it’s worth seeing how.
A classifier doesn’t really output a single answer. Under the hood, for every possible answer it first produces a raw score — machine-learning people call these the logits. A logit is just an un-normalized score: a bigger one means “more likely,” but the numbers don’t mean anything on their own yet. They can be negative, and they don’t add up to anything tidy.
To turn that pile of raw scores into numbers we can actually read as probabilities, the model runs them through a step called the softmax. You can picture softmax as a fair way of converting scores into percentages: it squashes them so that every option lands between 0% and 100%, and — this is the important part — so that they all add up to exactly 100% across the options. The largest raw score becomes the largest probability; a runaway-large score becomes a near-certainty.
Whichever option ends up on top is the model’s prediction, and the probability sitting on that top option is its confidence — its stated certainty in the answer it just gave. If the model outputs {healthy: 10%, mild: 5%, moderate: 85%}, it predicts “moderate” with 85% confidence.
Calibration is about that number — the stated confidence — not about which option won. So the honest answer to the question everyone asks — is calibration measuring accuracy, or the confidence in the accuracy? — is firmly the second. Calibration measures the confidence in the accuracy. It’s a second axis, sitting on top of correctness, that audits the number the model attaches to its own certainty.
You can’t judge a single prediction’s calibration — one guess is just right or wrong, 0 or 1. So you group predictions by their stated confidence into bins (say, five buckets across the range), and in each bin you ask: what was the average confidence here, and what was the actual accuracy here?
Plot those bin-by-bin — mean confidence on the x-axis, actual accuracy on the y-axis — and you get a reliability diagram. Perfect calibration is the 45° diagonal: confidence equals accuracy everywhere. Points below the line mean overconfidence (it claims more certainty than it earns); points above mean underconfidence (it’s more right than it admits).
Squeeze that whole diagram into one number, and you get Expected Calibration Error (ECE): the average gap between confidence and accuracy across the bins, weighted by how many predictions each bin holds.
ECE = Σ_bins (n_bin / N) · | accuracy_bin − confidence_bin |
Lower is better; zero is perfect. The weighting matters — and it’s exactly where the trouble hides.
This is the heart of the “beyond accuracy” idea, so let me make it concrete with the real bins from that staging model — 75 validation predictions, grouped by how confident the model was (most confident at the top):
First, let’s actually compute the ECE, so the 0.053 isn’t magic. ECE is each bin’s honesty gap (|accuracy − confidence|), weighted by the share of predictions that bin holds:
ECE = (55/75)·|0.93-0.93| + (16/75)·|0.88-0.71| + (4/75)·|0.25-0.56| = 0.73 · 0.00 + 0.21 · 0.17 + 0.05 · 0.31 = 0.000 + 0.036 + 0.017 = 0.053
Now look at what that average buried. The least-confident bin (0.4–0.6) is a genuine disaster — 56% confident, 25% accurate, a 31-point gap. But it holds only 4 of 75 predictions, so its weight (4/75) is tiny, and it adds just 0.017 to the score. Meanwhile, the most-confident bin (0.8–1.0) — 55 of the 75 predictions — is perfectly calibrated (0.93 vs 0.93) and contributes nothing. A huge, well-behaved bin drowns out a small, badly-behaved one, and 0.053 comes out looking healthy. That’s the first way one number lies.
The second way is subtler, and it hides inside the good-looking bin. “93% accurate” still means 7% wrong (100% − 93%). So of the 55 predictions in that most-confident bin, about 4 are flat wrong while the model was ~93% sure. Being calibrated on average doesn’t stop a bin from being full of confident individual mistakes.
To see how much that matters, count the errors the same way — each bin’s misses are its predictions times its miss-rate (1 − accuracy):
The model made about 9 errors — and 4 of them (44%) landed in the most-confident bin, at ≥0.8 confidence. That’s the number ECE never showed you. So the rule I now live by: report ECE and the share of errors made at high confidence. One number describes the crowd; the other finds the dangerous individuals. On a single patient, “the model was 93% sure” is not the reassurance it sounds like.
There’s a third hiding place, specific to multi-class problems. A model outputs a probability for every stage, not just the winning one — and top-label calibration throws the losing probabilities away. So I checked each stage on its own: how much probability the model spends on that stage (its mean predicted probability) versus how often the stage actually occurs (its true base rate). If the model were unbiased, those two would match.
The aggregate looked fine, but the table exposes a directional bias the headline number was blind to: the model under-assigns “mild” and over-assigns “healthy” — systematically shifting probability away from the treatable early stage toward “nothing’s wrong.” That’s the most dangerous error a screening model can make.
And the reason was the most useful lesson of the whole exercise: calibration failed where the classes overlap, not where a class is rare. The rarest stage, severe, was actually the best-calibrated — it’s cleanly separable, so its probabilities are trustworthy. “Mild,” which blurs into both of its neighbours, was the worst. A stage’s calibration tracks how distinguishable it is, not how common it is.
Directly? No — and it’s worth being precise about this. Recalibrating confidence doesn’t change which class the model picks, so it doesn’t move accuracy. What it improves is the trustworthiness of the confidence, which improves every decision built on that confidence: where to set a threshold, when to defer to a human, when to ask for another test.
But indirectly, calibration is one of the best diagnostics you have for where accuracy is weak. The places calibration breaks — the overlapping “mild” stage above — are precisely where the model’s signal is muddy. Calibration points a finger at the regions that need better features or more data. It won’t fix accuracy for you, but it tells you where to aim.
If a model’s confidence is dishonest, you don’t have to retrain it from scratch. You can leave the model exactly as it is and just re-map its confidence numbers afterward — a step called post-hoc recalibration. The idea is to learn a small correction on a held-out slice of data — a translation from the model’s raw certainty into honest certainty — and then apply that same correction to every future prediction. Three tools are common; here they are from most flexible to gentlest.
Platt scaling learns a single smooth, S-shaped curve that maps the model’s scores onto better-behaved probabilities. It has only a couple of knobs, so it can straighten out a consistent kind of overconfidence — but because it’s so simple, it can’t fix anything more complicated than a gentle, uniform bend.
Isotonic regression is the flexible cousin. Instead of one smooth curve, it fits a free-form “staircase” that can bend the mapping almost any way the data suggests — with the one rule that it can only ever go up (more confidence should never mean less accuracy). That flexibility cuts both ways: with plenty of data it can repair messy miscalibration, but on small data it cheerfully memorizes noise and overfits, leaving things worse than it found them.
Temperature scaling is the gentlest, and it’s worth understanding because it’s usually the safest. Recall the two ideas from earlier — the logits (the model’s raw scores) and the softmax that turns them into probabilities. Temperature scaling simply divides all of the logits by one positive number, T, before the softmax runs. Think of T as a certainty volume knob: turn it up (T above 1) and the probabilities flatten toward “not so sure”; turn it down (T below 1) and they sharpen toward “very sure.” The crucial trick is that you divide every score by the same number, so their order never changes — which means the model’s top pick never changes and its accuracy stays exactly the same. Temperature scaling only adjusts how loud the confidence is, never the answer itself.
Now the honest part, because this is where I learned the most. On my staging model, the two flexible methods actually made calibration worse. Platt scaling pushed the calibration error up from 0.05 to 0.27, and isotonic to 0.11 — both worse than doing nothing at all, and adding more data didn’t save them.
Here’s why, in plain terms. When I trained the model, I used a setting called class-weighting — I told it to pay extra attention to the rare “severe” stage so those cases wouldn’t get overlooked. That’s good for catching severe patients, but it has a side effect: to protect the rare class, the model deliberately shifts its raw scores away from the true frequencies. So the scores were already bent, on purpose, before any recalibration touched them.
Platt scaling and isotonic regression are both “learn a correction from the data” methods. When they looked at those already-bent scores, they didn’t see a mistake to fix — they saw a pattern to match, and dutifully learned to reproduce the bend, locking the distortion in even harder. The more flexible the method (isotonic especially), the more eagerly it fit the wrong thing.
Temperature scaling couldn’t fall into that trap. It has just one knob — a single number applied to every score at once — so it physically can’t twist itself to match a lopsided distortion. That bluntness is exactly why it was the only method that didn’t hurt here (0.024 → 0.022), and it left accuracy untouched.
The takeaway: recalibration is a hypothesis to test, not a guarantee. Always measure the calibration error before and after, prefer the gentle, accuracy-preserving option first, and be willing to report the unglamorous result: “we tried to recalibrate, and it made things worse.”
Everything above was written about a classical classifier, but it matters more than ever now. When a chatbot states a wrong fact in a fluent, authoritative voice, that is a calibration failure: high confidence, low correctness.
And it’s documented. In its GPT-4 technical report, OpenAI showed that the pretrained model — before the human-feedback fine-tuning — was remarkably well-calibrated: when it said “80%,” it was right about 80% of the time. But after RLHF (reinforcement learning from human feedback — the step that makes a model helpful, polite, and agreeable), its calibration got worse: its confidence numbers no longer tracked reality as tightly.
It’s worth being precise about what actually changed, because your question is a fair one — did we train the humility out, or teach a different humility in? Both, in a sense. RLHF doesn’t necessarily make the model sound less humble; it can still say “I’m not certain” in words. What degrades is its numeric confidence — the probabilities it attaches to its answers drift toward overconfident. So the humility it loses is in its numbers, not its manners — and the numbers are exactly what an automated system downstream will act on. (This overconfidence-from-RLHF effect has since been studied directly — see Taming Overconfidence in LLMs: Reward Calibration in RLHF.)
Whether you’re shipping a boosted tree, an image classifier, or an LLM, the question is the same: can you trust the number it puts on its own certainty? And it’s about to get recursive — when we reach AI grading AI (LLM-as-a-judge), we’ll have to ask it of the judge too: not just what it decides, but whether we can trust how sure it is.
Accuracy measures whether a model knows the answer. Calibration measures whether it knows when it might be wrong. Humans call that humility. In machine learning we call it calibration — and in medicine, in fraud, or in a chatbot you’re about to trust, it can matter as much as accuracy itself.
This is the fourth stop on the map — can you trust the confidence number the model reports, the probability it attaches to its own prediction? Next I’ll put the metrics side by side and revisit the whole journey. If this was useful, tell me where a confident-but-wrong model has bitten you — I read every reply.
A note on the data: the numbers here come from a synthetic four-stage staging model I built for an honest-evaluation project — I generate the patients, so the ground truth is known exactly, which is the cleanest way to study a measure like calibration. The next step is to carry the same evaluation onto real, multimodal data. If you want to see it run on a public dataset you can rerun yourself, that companion piece is coming.
Does Your AI Know When It Might Be Wrong? was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.