# I graded my own ML option forecasts. Here's the Brier score.

> Source: <https://dev.to/connerlambden/i-graded-my-own-ml-option-forecasts-heres-the-brier-score-20fp>
> Published: 2026-07-14 22:24:55+00:00

In May I promised — out loud, on the internet, where people can screenshot you — that I'd grade Helium's published `prob_itm`

forecasts when the June 2026 AAPL contracts expired.

June 26 came. The contracts died. So I graded them.

This is not a victory lap. Mean Brier loss on **n=2** is **0.3846**. A coin flip that always says 0.5 scores 0.25 when the world resolves to 0 or 1. We did worse than a coin flip. That is information.

| Contract | Helium `prob_itm`
|
Market-ish implied |
|---|---|---|
| AAPL $310C 2026-06-26 | 0.42 | ~0.50 |
| AAPL $295P 2026-06-26 | 0.23 | ~0.24 |

**AAPL NASDAQ close on 2026-06-26: $283.78**

```
git clone https://github.com/connerlambden/helium-mcp-cookbook
cd helium-mcp-cookbook
python calibration/grade_june_expiry.py
```

Frozen inputs live in [ calibration/june_2026_forecasts.json](https://github.com/connerlambden/helium-mcp-cookbook/blob/main/calibration/june_2026_forecasts.json).

Because the alternative is the usual ML options blog post: a backtest, a vibe, a screenshot of a green P&L. Publishing `prob_itm`

with a fixed expiry is a tiny discipline. Grading it after the fact is the second half of that discipline.

If you log your own forecasts with [recipe 02](https://github.com/connerlambden/helium-mcp-cookbook/blob/main/recipes/02_options_calibration_tracker.py), you can run the same scorer when your contracts expire. I'll keep adding rows.

—

Also adjacent, if you're here for news rather than options: we published a free [212×37 news-outlet framing corpus](https://huggingface.co/datasets/HeliumTrades/helium-news-bias-corpus) with an [explorer](https://connerlambden.github.io/helium-news-explorer/). Different animal. Same "put numbers where people can grade them" instinct.