There was a comment on Hacker News this week saying LLM token probabilities are badly calibrated for making decisions. I think that's right, and it's also something you can measure yourself in about fifteen lines, so here's how I'd do it.
The usual number is expected calibration error. You take a set of predictions where you already know the outcome and put the stated probabilities into m equal-width bins. In each bin you compare the average stated probability with the fraction of events that actually happened, weight that gap by the share of all predictions that fell in the bin, and add it up.
import numpy as np
def ece(p, y, m=10):
p, y = np.asarray(p, float), np.asarray(y, float)
b = np.minimum((p * m).astype(int), m - 1) # p = 1.0 goes in the last bin
total = 0.0
for k in range(m):
s = b == k
if s.any(): # empty bins carry no weight
total += s.mean() * abs(y[s].mean() - p[s].mean())
return total
The np.minimum clamp is there because of the bug I see most often. If you write int(p * m) without the clamp, a prediction of exactly 1.0 gets bin number m, one past the end, and you'll either get an index error or (worse) lose those predictions without a word. The other one I'd look for is a plain average of the per-bin gaps, which means you've let a bin holding three predictions count as much as one holding three thousand.
I tried it on 5,000 simulated events whose true chances came from a Beta(2, 2), with ten bins:
| Predictions | ECE | Brier score |
|---|---|---|
| The true chances | 0.013 | 0.199 |
| Overconfident (every chance pushed 60% further from a half) | 0.090 | 0.210 |
| Always the base rate, 0.507 | 0.000 | 0.250 |
The true chances don't score zero, because with 5,000 outcomes there's sampling noise in each bin, so I'd treat small ECE differences with some suspicion. The overconfident model put 460 predictions at exactly 1.0, which is the clamp bug waiting to happen.
The last row is the one I'd want anyone using probabilities for decisions to see. A model that always says the base rate is perfectly calibrated and tells you nothing at all. That's why I'd always report a Brier score or log loss on the same set as well, since they reward being calibrated and sharp at once, and look at the reliability diagram, because ECE moves around with the number of bins you choose.
If you want to write it yourself and test it, the exercise is here, with hints and a worked solution: https://academicquant.co.uk/ml-expected-calibration-error?utm_source=devto&utm_medium=post&utm_campaign=devto-calibration-ece