Three Metrics, Three Questions: F1, AUROC, and Calibration

A Medium article on clinical AI model evaluation argues that F1, AUROC, and calibration answer distinct questions—predict, rank, trust—and that excelling in one does not guarantee the others, warning that a model with accuracy 91%, AUROC 0.96, and F1 0.89 could still have poor calibration. The author, building an evaluation layer for clinical AI, notes F1 is suited for rare diseases with fixed thresholds but is blind to probability confidence and can be inflated by data leakage, citing a 2025 audit of fraud-detection methods.

Beyond Accuracy · Model Evaluation 5 I’m building an honest evaluation layer for clinical AI models. Today it runs on a synthetic staging model, where I generate the patients and know the ground truth exactly. That lets me study something I’ve become increasingly convinced of: no single metric is enough. A model lands on your desk with a row of numbers that look like a green light: accuracy 91%, AUROC 0.96, F1 0.89. Read those three and you’d deploy it tomorrow. But suppose its calibration is poor — the failure the last piece https://medium.com/towards-artificial-intelligence/does-your-ai-know-when-it-might-be-wrong-1c32f1e1fffc was all about. That model separates sick from healthy beautifully, makes the right call at its chosen threshold, and still, when it looks the neurologist in the eye and says “99%,” that number cannot be believed. That’s the trap this piece is about. F1, AUROC, and calibration each answer a different question, and a model can score brilliantly on one while failing dangerously on another. The goal here is to make those differences concrete: what each metric is really for, the clinical situation where it’s the right tool, and — just as important — the question it cannot answer. The three map cleanly onto the three questions every medical AI paper is ultimately asking: These are three independent dimensions of model quality. Excelling in one does not guarantee excellence in the others — that single fact is the lens for the whole article. Keep the three verbs — predict, rank, trust — in mind: they are not interchangeable, and confusing them is how a model that looks good on paper hurts a patient at the bedside. What it answers. F1 judges a model at the moment of decision: once it has committed to calling a patient “disease” or “healthy,” does it both catch real cases recall and avoid false alarms precision ? Its value is that it survives class imbalance — the situation where plain accuracy quietly lies. A model that calls all 1,000 patients healthy when only 10 are sick scores 99% accuracy and an F1 near zero. F1 refuses to reward that. Where it’s the right tool. Use F1 when the model drives a concrete, fixed action and the disease is rare. Picture an Alzheimer’s screening flag in a busy memory clinic: every “positive” pulls a patient into a costly work-up, and every missed case is a patient who walks out undiagnosed. You’ve chosen an operating threshold, positives are uncommon, and you need one honest number that punishes both failure modes at once. That’s exactly F1’s job — it stays low unless the model is both accurate when it warns and thorough in what it catches. What it cannot tell you. F1 works on hard labels at one chosen threshold, so it’s blind to everything the probabilities carry. It can’t tell you whether a different cutoff would serve you better, and it can’t tell you whether the model’s confidence is honest — a model can have a fine F1 and still hand you a wildly overconfident 99%. It also shifts with prevalence: the same model can post a different F1 in a screening clinic than in a specialist referral population, so an F1 from one setting doesn’t transfer to another. And because it collapses precision and recall into one number, it hides which of the two is weak — you should almost always report precision and recall alongside it. One more caution worth carrying over from the F1 piece: a strong F1 is only as trustworthy as the split that produced it — a 2025 audit of fraud-detection methods https://arxiv.org/html/2506.02703v1 found many reported F1 scores were inflated by data leakage, real numbers from a broken setup. What it answers. AUROC steps back from any single threshold and asks whether the model can order patients correctly: does a randomly chosen sick patient tend to score higher than a randomly chosen healthy one? That plain-language reading is also its exact mathematical definition — AUROC equals the probability that a random positive outranks a random negative, a result that goes back to Hanley & McNeil 1982 https://pubs.rsna.org/doi/abs/10.1148/radiology.143.1.7063747 ; Fawcett 2006 https://www.sciencedirect.com/science/article/abs/pii/S016786550500303X is the standard primer. It summarizes the model’s separating power across every possible cutoff into one number, which makes it the natural metric for comparing models and for any task where ranking — not a fixed yes/no — is the point. Where it’s the right tool. Use AUROC when the output is a priority order rather than a decision. Imagine a radiology worklist that reorders overnight scans so the most suspicious land at the top of the morning queue. No one has fixed a threshold; the value is entirely in putting high-risk patients ahead of low-risk ones. AUROC measures precisely that. It’s also the right lens when you’re choosing between two candidate models before you’ve committed to any operating point — the higher-AUROC model has the better raw ability to tell the classes apart. What it cannot tell you. AUROC won’t choose your threshold — every point on the ROC curve is a threshold, and the summary number averages over all of them: it says the road is good, not where to stop. It’s also quietly misleading under heavy imbalance — ignoring prevalence, a model can post a 0.95 AUROC on a rare disease yet still produce mostly false positives at any clinically useful sensitivity, where a precision–recall view is more honest Saito & Rehmsmeier, 2015 https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0118432 . And it says nothing about whether the probabilities are trustworthy: a model can rank every patient perfectly AUROC = 1.0 while every probability it emits is miscalibrated garbage — ranking cares about order , not value . What it answers. Calibration asks whether the probability means what it says. When a well-calibrated model says 90%, then across all the patients it labels 90%, about 90 in 100 truly have the disease. Expected Calibration Error ECE quantifies the gap between stated confidence and observed reality. This is the metric that matters when the probability itself — not just the class — drives what happens next. It’s the whole subject of the previous piece https://medium.com/towards-artificial-intelligence/does-your-ai-know-when-it-might-be-wrong-1c32f1e1fffc , if you want the deep version. Where it’s the right tool. Use calibration whenever a graded decision hangs on the number. Return to the neurologist reading a risk score: a 51% sends the patient toward more imaging and a six-month follow-up, a 74% prompts a memory-clinic referral, and a 99% opens a conversation about starting treatment. The predicted class is “Alzheimer’s” in all three cases; only the confidence separates the actions. If that 99% is really an overconfident 90%, the model has pushed a patient toward treatment they may not need. Any time the score is shown to a patient, combined with other risk factors, or fed into a cost–benefit threshold, calibration is the metric standing between a probability and a decision. What it cannot tell you. Calibration is silent about whether the model can tell sick from healthy at all. A model that ignores every input and just outputs the population prevalence — 8% for everyone — can be almost perfectly calibrated and completely useless, with an AUROC of 0.5. Calibration is a property of honesty , not skill ; always read it next to a discrimination metric. And a low overall ECE can still hide clusters of confident mistakes — well-calibrated in aggregate, badly overconfident on one subgroup — so one ECE number is necessary but not sufficient. The reason no single metric suffices is that these three qualities are independent. Go back to that green-light row — accuracy 91%, AUROC 0.96, F1 0.89 — and add the number nobody printed: a poor ECE. That model separates sick from healthy beautifully AUROC 0.96 and makes the right call at its chosen threshold F1 0.89 , and still, when it tells the neurologist “99%,” that number can’t be trusted. Ranking and thresholded accuracy were never going to catch that; only calibration does. The failure runs in every direction: Because the three can dissociate, a trustworthy evaluation reports one from each family and refuses to let a strong number in one column excuse a weak one in another. This isn’t a new worry, and it isn’t only a deep-learning one: the standard framework for evaluating clinical prediction models has long insisted on reporting discrimination and calibration together, alongside a measure of clinical usefulness Steyerberg et al., 2010 https://pubmed.ncbi.nlm.nih.gov/20010215/ — and calibration in particular has been called “the Achilles heel of predictive analytics” Van Calster et al., 2019 https://link.springer.com/article/10.1186/s12916-019-1466-7 . F1 asks whether the model predicted right. AUROC asks whether it can rank . Calibration asks whether it can be trusted. A trustworthy clinician doesn’t only make good decisions. They also prioritize correctly, and they know when they might be wrong. A trustworthy model should be held to the same standard — and that standard has three parts, not one. A model strong in only one or two of them isn’t a good model with a minor flaw; it’s a different, and often more dangerous, thing wearing a good model’s numbers. This is the fifth stop on the map, and the one that ties the others together. Next I’ll follow the probabilities one step further — into thresholds and decision utility: once you trust the number, what cutoff do you actually act on? If this was useful, tell me which of the three metrics has burned you — I read every reply. A note on the data: the examples here are drawn from an honest-evaluation project built on a synthetic four-stage staging model — I generate the patients, so the ground truth is known exactly, which is the cleanest way to study how these metrics agree and disagree. The next step is to carry the same evaluation onto real, multimodal data; a companion piece you can rerun yourself is coming. Three Metrics, Three Questions: F1, AUROC, and Calibration https://pub.towardsai.net/three-metrics-three-questions-f1-auroc-and-calibration-7068759a5e8f was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.