*A follow-up to my DEV × Kaggle Benchmarking Challenge series.
For the challenge I built a benchmark that asked one thing: does your model know when it doesn't know? It had 200 invented items in four shapes. One in five had the answer taken out, and on those the only right reply was ESCALATE. Each model got two numbers, never merged: how often it got the answerable ones right, and how often it answered anyway when it should have stopped.
The finding wasn't that models can't stop. Five of the twelve hosted models I ran had zero false confidence on every shape. The finding was that the average hides the floor. Claude Haiku 4.5 answered 9 out of 10 judge items it should have escalated, a number you'd never see in its overall score.
When the challenge closed I kept turning the question over, and it stopped being about models. Turn it on a person. Turn it on a stock tip, a betting app, a newsletter that "called" the last crash. Does the forecaster know when it doesn't know? And can you even check?
Since the challenge, the stack I run (local-first, everything on my own box) got a rebuild I've been calling the flatten. It had three copies of its random stream, two record chains and three ways to say "a human approved this", and it ended up with one of each:
ESCALATE, built into the plumbing.
It turns out those four things are exactly what a forecast record needs.
A market call is a wager like any other on the record:
{"symbol": "SPY", "kind": "up", "close_date": "2026-10-23"}
with a probability: "yes, 55%". Then four steps.
The interesting part is step 2. "Before the open" can't rest on my computer's clock, because I could set it back. So it rests on drand. When I make a call, the record writes down drand's newest round. When I seal, the record keeps drand's newest round outside the box. A call counts only if the round after each of those fired before the deadline:
def in_time(latest: object, close_date: str) -> bool:
"""True when ``latest`` (a reader's newest drand round) shows the moment
it was read was before the deadline: its next round fires before it."""
if isinstance(latest, bool) or not isinstance(latest, int):
return False
return beacon.round_time(latest + 1) <= deadline(close_date)
If I sealed late, the proposal escalates with the reason. If the price source is down, the outcome is unreachable, not "no". If the price the row says differs from what the reader says now, it escalates. If a close for that day is already on the record before the call, it escalates. Each of thoseF has a test that tries it and checks for the refusal: 42 tests. I also broke each guard on purpose to make sure a test failed, and 14 out of 14 did.
Here's the part that surprised me. The benchmark rewarded a model for saying "I can't answer this". The ledger rewards a forecaster for the same thing, through the grading.
Every call is scored with a Brier score: the square of the gap between the probability you gave and what happened. A call of 50% scores 0.25 whatever happens. That's the coin flip, and it's the forecaster's ESCALATE: "I don't know which way this goes", said honestly.
So the question the ledger answers isn't "did you make money". It's do you beat the coin, over many calls, with the misses counted? That's the floor-not-average idea from the benchmark, aimed at whoever's making the calls. A forecaster who's confident on everything and right on the easy ones will look great in a highlight reel and lose to the coin on the ledger.
The benchmark asked whether a model knows when it doesn't know. The ledger asks the same of whoever's making the call, with a record they can't edit and a clock they don't control. It's the same idea with a human in the seat, and that was always the part I cared about.
Always open for discussion.
Previously, in the challenge: Day 0 · Day 1 · Day 2 · Day 3 · Day 4
The code: willow-bot, one-script/onescript/market.py.