Are your model's probabilities calibrated? Expected calibration error in NumPy

A developer published a NumPy implementation of expected calibration error (ECE) for measuring whether model probabilities match observed outcomes, flagging two common bugs: an unclamped bin index that drops predictions of exactly 1.0, and averaging per-bin gaps without weighting by bin population. In a simulation of 5,000 events drawn from a Beta(2,2) distribution, true probabilities scored an ECE of 0.013 versus 0.090 for an overconfident model, while a model that always predicted the 0.507 base rate scored 0.000 ECE but a worse Brier score of 0.250 — illustrating that calibration alone says nothing about sharpness.

There was a comment on Hacker News this week saying LLM token probabilities are badly calibrated for making decisions. I think that's right, and it's also something you can measure yourself in about fifteen lines, so here's how I'd do it. The usual number is expected calibration error. You take a set of predictions where you already know the outcome and put the stated probabilities into m equal-width bins. In each bin you compare the average stated probability with the fraction of events that actually happened, weight that gap by the share of all predictions that fell in the bin, and add it up. python import numpy as np def ece p, y, m=10 : p, y = np.asarray p, float , np.asarray y, float b = np.minimum p m .astype int , m - 1 p = 1.0 goes in the last bin total = 0.0 for k in range m : s = b == k if s.any : empty bins carry no weight total += s.mean abs y s .mean - p s .mean return total The np.minimum clamp is there because of the bug I see most often. If you write int p m without the clamp, a prediction of exactly 1.0 gets bin number m, one past the end, and you'll either get an index error or worse lose those predictions without a word. The other one I'd look for is a plain average of the per-bin gaps, which means you've let a bin holding three predictions count as much as one holding three thousand. I tried it on 5,000 simulated events whose true chances came from a Beta 2, 2 , with ten bins: | Predictions | ECE | Brier score | |---|---|---| | The true chances | 0.013 | 0.199 | | Overconfident every chance pushed 60% further from a half | 0.090 | 0.210 | | Always the base rate, 0.507 | 0.000 | 0.250 | The true chances don't score zero, because with 5,000 outcomes there's sampling noise in each bin, so I'd treat small ECE differences with some suspicion. The overconfident model put 460 predictions at exactly 1.0, which is the clamp bug waiting to happen. The last row is the one I'd want anyone using probabilities for decisions to see. A model that always says the base rate is perfectly calibrated and tells you nothing at all. That's why I'd always report a Brier score or log loss on the same set as well, since they reward being calibrated and sharp at once, and look at the reliability diagram, because ECE moves around with the number of bins you choose. If you want to write it yourself and test it, the exercise is here, with hints and a worked solution: https://academicquant.co.uk/ml-expected-calibration-error?utm source=devto&utm medium=post&utm campaign=devto-calibration-ece https://academicquant.co.uk/ml-expected-calibration-error?utm source=devto&utm medium=post&utm campaign=devto-calibration-ece