Your AI Model Predicts Risk. But Who Decides When to Act?

A synthetic screening model trained on 4,000 generated patients — 448 of them (11%) truly sick — achieved an AUROC of 0.84 and was well calibrated, yet at the default 0.5 probability cutoff it caught only 25% of disease, flagging 110 of the 448 sick patients and missing 338 while raising just 56 false alarms. The author, writing in a personal model-evaluation series, argues the failure lay in the decision rule rather than the model: a threshold chosen without weighing the relative cost of a missed case versus a false alarm converts trustworthy probabilities into poor decisions. The piece frames threshold selection as a value judgment the model cannot supply, the fourth step after F1, AUROC and calibration.

Beyond Accuracy · Model Evaluation 6 Honestly evaluating AI models is a personal project of mine, and I’ve been exploring it on clinical problems — by choice. Health is simply the domain I find most revealing, and the one most people feel in their gut: the cost of a missed cancer, or of an unnecessary biopsy, needs no explanation. But nothing in this piece is really about medicine. Any system that turns a probability into an action faces the same step — fraud scoring, demand forecasting, anomaly detection — those are just harder to feel in the same immediate way. Today the model is a synthetic screening model, where I generate the patients and know the ground truth exactly. That lets me study the step everyone rushes past: a trustworthy probability still isn’t an action — and the number you choose to act on decides everything. Start with a model that does its job well. On 4,000 synthetic patients — about 11% of them truly sick 448 cases — it ranks sick patients ahead of healthy ones reliably AUROC 0.84 — AUROC https://medium.com/towards-artificial-intelligence/a-model-with-0-96-auroc-can-still-bury-you-in-false-alarms-028d05517553 measures how well a model sorts the sick above the healthy . Its probabilities are also honest: the 11% is the overall rate across everyone, but the model doesn’t hand every patient 11% — it gives each one their own risk, and when it says 30% for a group of patients, close to 30% of those patients really are sick. That last property is called calibration https://medium.com/towards-artificial-intelligence/does-your-ai-know-when-it-might-be-wrong-1c32f1e1fffc . Those are the first three questions this series asks of a model — can it make the call F1 , can it order cases by risk AUROC , can you believe its confidence calibration — pulled together most recently in the previous piece https://medium.com/towards-artificial-intelligence/three-metrics-three-questions-f1-auroc-and-calibration-7068759a5e8f . This model clears all three. The model gives each patient a probability. Turning those probabilities into yes/no calls is a separate step, and it’s mine to make, not the model’s. So I did what most pipelines do by default: I treated every patient with a predicted risk above 0.5 — that is, above 50% — as a “positive” worth acting on. At that cutoff, it caught just a quarter of the disease — 25%. Of the 448 truly sick patients, it flagged 110 and missed 338 , while raising only 56 false alarms. The model itself didn’t fail — its ranking is intact, and its probabilities are honest. What failed was the decision rule : a cutoff chosen without weighing the consequences turned good risk estimates into a poor decision. It missed those patients because 0.5 was the wrong place to act, and nobody had chosen it on purpose. That’s the whole subject of this piece: once you trust the number, what do you do with it? This series has been building one question at a time. F1 → can it predict? AUROC → can it rank? Calibration → can it be trusted? Those three are about the model. This one is about you : you now hold a trustworthy probability — what threshold turns it into an action? Call it the fourth verb: Decide. A probability is a statement about the world. A threshold is a statement about what you value. The model can hand you the first; only you can supply the second. Deciding is setting that threshold — the cutoff on the probability above which you act and below which you don’t — and the moment you pick it, you’ve made a value judgment about which mistake you’d rather make. The default cutoff of 0.5 feels neutral. It isn’t. A screening model can make two kinds of mistake: a missed case — a sick patient it waves through a false negative — and a false alarm — a healthy patient it flags for nothing a false positive . Acting only when risk exceeds 50% quietly declares that a missed case and a false alarm cost you exactly the same. Cutting at 50% is the right call only if you’d trade one missed patient for one false alarm and feel you came out even — no worse off either way. That holds only in a simplified picture where those two mistakes are the whole story. Real decisions can carry more — the cost of running the test, the harm of the treatment itself, a patient’s own preferences — and a fuller analysis folds those in too. But the balance between the two errors is the right place to begin, so that’s where we’ll start. In almost no clinical setting are the two errors equal. Missing an early cancer is not the same as calling one more patient back for a scan that turns out clear. And when a disease is uncommon, a 0.5 cutoff is especially punishing. Low prevalence doesn’t guarantee this, but when the features give only moderate discrimination, even a well-calibrated model lands many genuinely sick patients below 50% — it’s seldom more than half-sure about anyone. A rule that acts only above 50% then flags almost none of them, and most of the disease slips through. That’s exactly what happened above — three-quarters of the sick patients, missed, at a threshold no one deliberately chose. This doesn’t contradict the model’s ranking. AUROC is about order — sick scoring higher than healthy — not about where those scores land on the 0-to-1 scale. A sharper model could push many sick patients above 50%; this one, with only moderate discrimination, leaves 338 of the 448 below it — ranked above the healthy, yet short of the halfway mark. The first move, then, is simply to make the choice on purpose. The classic way to pick a cutoff from the data is Youden’s index . It leans on two numbers: sensitivity the share of truly sick patients the model catches and specificity the share of truly healthy patients it correctly leaves alone . Youden’s index picks the threshold that maximizes J = sensitivity + specificity − 1 — the same as maximizing their sum, since the −1 is just a constant. It gives the two rates equal weight, with no regard for how common the disease is or what either error actually costs Youden, 1950 <https://acsjournals.onlinelibrary.wiley.com/doi/10.1002/1097-0142 1950 3:1%3C32::AID-CNCR2820030106%3E3.0.CO;2-3 . On this model it lands at 0.12 , and the picture changes completely: sensitivity jumps from 25% to 77% , specificity is 76% , and the model now catches 344 of the 448 cases instead of 110 — but the false alarms climb with it, from 56 to 848. But look at what Youden’s index quietly assumes: that a point of sensitivity and a point of specificity are worth the same. It’s a statistical balance, blind to what the errors actually cost and to how common the disease is. It’s a reasonable reference point when you genuinely have no cost information — and the wrong anchor the moment you do. Which is almost always. Here’s the idea that reorganizes everything. The ROC curve shows what different thresholds can achieve, but it can’t, by itself, tell you which trade-off is clinically acceptable. That is set by a single question: how many times worse is a missed case than a false alarm? Call that number — how many times worse a missed case is than a false alarm — the cost ratio , and write it R . “Cost” here means real-world harm: a missed case can mean a disease caught too late; a false alarm means an unnecessary, sometimes invasive, follow-up. For a calibrated risk model, when the only things on the scale are the cost of a false positive and the cost of a false negative, the threshold that minimizes expected cost is act if predicted risk ≥ p t, where p t = 1 / 1 + R Here p t is the probability threshold — p for probability, the small t for “threshold.” Read it slowly, because it’s the whole game. The arithmetic is just that one formula: One thing R is not : a promise about headcount. It sets the relative cost of the two errors, not the number of false alarms you’ll actually trade for each extra case caught — that depends on the model and how common the disease is. The nine is a statement of values, not a body count. And the formula assumes the plain yes/no decision we’re in; capacity limits, costs that differ patient to patient, or actions beyond act / don’t-act would each need more than this one line. On this model, that one number moves everything: Same model, same probabilities, same patients. The only thing that changed was an honest statement of what the mistakes cost — and that statement, not the algorithm, decided who got caught. The threshold is where medicine, not machine learning, has to speak. Suppose you’ve chosen a threshold. One question remains, and it’s the one that should be asked first and almost never is: does using this model actually beat the obvious alternatives — referring everyone, or referring no one? This is what Decision Curve Analysis DCA answers — a method from Vickers & Elkin 2006 https://journals.sagepub.com/doi/10.1177/0272989X06295361 , not something I’ve invented, and the first time this series meets it. It rests on a quietly brilliant reframing: the threshold you’d act at is itself a statement of how you weigh the two errors against each other. A clinician who’d act at 10% is telling you a missed case is worth about nine false alarms — so weigh their harms in exactly that proportion. That gives a single number of their devising, net benefit , at any threshold: net benefit = true positives / N − false positives / N × p t/ 1 − p t Here N is the total number of patients evaluated. The true positives are the good the model does; the false positives are penalized in exact proportion to that same trade-off. Compute it across every threshold a reasonable clinician might hold, and plot three lines: the model, refer everyone , and refer no one . In a screening setting those defaults mean testing or referring every patient versus none — not literally treating anyone. Across the range of thresholds a clinician might reasonably consider for a screening referral — roughly 5% to 30% — the model delivers at least as much estimated net benefit as either alternative: referring everyone, or referring no one. The trade-off is revealing. At very low thresholds, referring everyone can make sense, because the priority is to miss as few cases as possible. But as the threshold rises, each unnecessary referral carries more weight, and eventually referring everyone produces less net benefit than referring no one at all. Referring no one, meanwhile, always scores exactly zero in Decision Curve Analysis — a mathematical reference point, not a claim that missing every case does no harm. The decision curve makes this trade-off visible. Instead of asking only how accurately the model predicts, it asks whether acting on those predictions offers a better balance of benefit and harm than the alternatives, at thresholds that reflect real clinical priorities. In this synthetic experiment, the model performs at least as well as both alternatives across that clinically relevant range; only at the very lowest thresholds — below about 2%, where acting on almost any suspicion and referring everyone amount to the same thing — do the two draw level. But estimated is the important word: these results describe the model’s potential value under the experiment’s assumptions; they don’t establish that patients would actually benefit in practice. Showing that takes further validation and, ultimately, a prospective clinical study. Every threshold in this piece rests on one quiet assumption: that the probabilities are calibrated in the population where the decision actually gets made. This model is well calibrated at 11% prevalence — when it says 30%, about 30% of those patients really are sick. Move it to a hospital with a different case mix, or a different disease rate, and that may no longer hold — and a threshold set on the old probabilities would now be aimed at the wrong place. A decision rule is only as trustworthy as the probabilities underneath it. In practice the first move isn’t to nudge the cutoff — it’s to check whether the model is still calibrated in the new population at all. Depending on the kind of shift, a recalibration or a prevalence correction may be enough; but if the relationship between the features and the outcome has changed, simple recalibration may not be enough — the model may need more extensive validation, updating, or redevelopment. Only once the probabilities are trustworthy there does the threshold mean what you think it means — and that is the next stop on the map. So the real next question is whether those probabilities survive the trip: will it work elsewhere? F1 asks whether the model predicted right. AUROC asks whether it can rank. Calibration asks whether you can trust it. Decide asks the question none of them can: at what number do you act, and at what cost? A trustworthy clinician doesn’t just estimate risk well. They know what to do with the estimate — and they know that the right action depends on what’s at stake, not on a number that happens to sit at one-half. A trustworthy AI-assisted decision is held to the same standard: it doesn’t come from the model alone, but from pairing an honest probability with a decision policy that reflects what’s at stake. The model estimates the risk. The policy decides what to do with it. And whether that policy makes sense stays our responsibility — not the machine’s. One caution worth keeping: clearing these four verbs is not a certificate of trustworthiness. A model can pass all four and still fail — under a shift in population, a gap between subgroups, or clumsy, unsafe deployment. That’s why the map keeps going. This is the fourth question in the evaluation journey — the sixth piece in the Beyond Accuracy series. The model has been asked to predict, to rank, to be trusted, and now to inform a decision. Next the journey leaves the model itself and turns to the humans around it — validation, generalization, and whether people should rely on it at all. If this was useful, tell me the worst default-threshold story you’ve seen — I read every reply. Question assumptions. Build intuition. A note on the data: the numbers here come from a synthetic screening model 4,000 patients, ~11% prevalence / 448 cases, AUROC 0.84, decile calibration error ≈ 1% built for an honest-evaluation project — I generate the patients, so every sensitivity, threshold, and net-benefit figure above is computed, not asserted, and reproducible. The generating script lives in the Model Evaluation Handbook repo code/build decide experiment.py ; it uses a fixed random seed, so one run rebuilds the dataset, and these exact numbers fall out. The next step is to carry the same decision-curve analysis onto real, multimodal data; that companion is coming. A note on scope: the worked example is clinical, but the machinery — a probability, a cost ratio, a threshold, a net-benefit check — is domain-neutral. Relabel “sick” as “fraudulent,” “defective,” or “about to churn,” and “refer” as “investigate,” “pull from the line,” or “step in,” and every move in this piece still holds. Rare-event decisions rhyme across domains. Start with the map: The Journey Toward Trustworthy AI https://medium.com/ai-in-plain-english/the-journey-toward-trustworthy-ai-b873c1e0e228 — the overview the whole series follows. Your AI Model Predicts Risk. But Who Decides When to Act? https://pub.towardsai.net/your-ai-model-predicts-risk-but-who-decides-when-to-act-6b07eafab83f was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.