{"slug": "my-robot-keeps-its-receipts", "title": "My Robot Keeps Its Receipts", "summary": "A developer built a tamper-evident forecasting ledger that scores market calls with Brier scores and uses drand's public randomness beacon as an external clock, so a forecaster cannot backdate a prediction. The system escalates calls that fail its guards — late sealing, a down price source, a mismatched price, or a pre-existing close — and is covered by 42 tests, with all 14 deliberately broken guards caught by a failing test. The work follows the developer's earlier benchmark on whether models know when they don't know, which found that averages hide per-item failures such as Claude Haiku 4.5 answering 9 of 10 judge items it should have escalated.", "body_md": "*A follow-up to my [DEV × Kaggle Benchmarking Challenge](https://dev.to/sean_campbell_840bd62bf7e/does-your-model-know-when-it-doesnt-know-a-benchmark-for-the-escalate-answer-268o) series.\n\nFor the challenge I built a benchmark that asked one thing: **does your model know when it doesn't know?** It had 200 invented items in four shapes. One in five had the answer taken out, and on those the only right reply was `ESCALATE`. Each model got two numbers, never merged: how often it got the answerable ones right, and how often it answered anyway when it should have stopped.\n\nThe finding wasn't that models can't stop. Five of the twelve hosted models I ran had zero false confidence on every shape. The finding was that **the average hides the floor**. Claude Haiku 4.5 answered 9 out of 10 judge items it should have escalated, a number you'd never see in its overall score.\n\nWhen the challenge closed I kept turning the question over, and it stopped being about models. Turn it on a person. Turn it on a stock tip, a betting app, a newsletter that \"called\" the last crash. Does *the forecaster* know when it doesn't know? And can you even check?\n\nSince the challenge, the stack I run (local-first, everything on my own box) got a rebuild I've been calling the flatten. It had three copies of its random stream, two record chains and three ways to say \"a human approved this\", and it ended up with one of each:\n\n`ESCALATE`, built into the plumbing.\nIt turns out those four things are exactly what a forecast record needs.\n\nA market call is a wager like any other on the record:\n\n```\n{\"symbol\": \"SPY\", \"kind\": \"up\", \"close_date\": \"2026-10-23\"}\n```\n\nwith a probability: \"yes, 55%\". Then four steps.\n\nThe interesting part is step 2. \"Before the open\" can't rest on my computer's clock, because I could set it back. So it rests on drand. When I make a call, the record writes down drand's newest round. When I seal, the record keeps drand's newest round outside the box. A call counts only if the round *after* each of those fired before the deadline:\n\n``` php\ndef in_time(latest: object, close_date: str) -> bool:\n    \"\"\"True when ``latest`` (a reader's newest drand round) shows the moment\n    it was read was before the deadline: its next round fires before it.\"\"\"\n    if isinstance(latest, bool) or not isinstance(latest, int):\n        return False\n    return beacon.round_time(latest + 1) <= deadline(close_date)\n```\n\nIf I sealed late, the proposal escalates with the reason. If the price source is down, the outcome is *unreachable*, not \"no\". If the price the row says differs from what the reader says now, it escalates. If a close for that day is already on the record before the call, it escalates. Each of thoseF has a test that tries it and checks for the refusal: 42 tests. I also broke each guard on purpose to make sure a test failed, and 14 out of 14 did.\n\nHere's the part that surprised me. The benchmark rewarded a model for saying \"I can't answer this\". The ledger rewards a forecaster for the same thing, through the grading.\n\nEvery call is scored with a Brier score: the square of the gap between the probability you gave and what happened. A call of 50% scores 0.25 whatever happens. That's the coin flip, and it's the forecaster's `ESCALATE`: \"I don't know which way this goes\", said honestly.\n\nSo the question the ledger answers isn't \"did you make money\". It's **do you beat the coin, over many calls, with the misses counted?** That's the floor-not-average idea from the benchmark, aimed at whoever's making the calls. A forecaster who's confident on everything and right on the easy ones will look great in a highlight reel and lose to the coin on the ledger.\n\nThe benchmark asked whether a model knows when it doesn't know. The ledger asks the same of whoever's making the call, with a record they can't edit and a clock they don't control. It's the same idea with a human in the seat, and that was always the part I cared about.\n\nAlways open for discussion.\n\n*Previously, in the challenge: [Day 0](https://dev.to/sean_campbell_840bd62bf7e/does-your-model-know-when-it-doesnt-know-a-benchmark-for-the-escalate-answer-268o) · [Day 1](https://dev.to/sean_campbell_840bd62bf7e/day-1-most-of-my-bugs-looked-like-model-behaviour-388h) · [Day 2](https://dev.to/sean_campbell_840bd62bf7e/day-2-the-model-i-want-is-the-one-thats-boring-everywhere-610) · [Day 3](https://dev.to/sean_campbell_840bd62bf7e/day-3-the-benchmark-caught-me-too-3hdl) · [Day 4](https://dev.to/sean_campbell_840bd62bf7e/day-4-there-never-needed-to-be-a-check-just-a-gated-pr-a-story-about-finding-the-middle-3791)*\n\n*The code: [willow-bot](https://github.com/willow-memory/willow-bot), `one-script/onescript/market.py`.*", "url": "https://wpnews.pro/news/my-robot-keeps-its-receipts", "canonical_source": "https://dev.to/sean_campbell_840bd62bf7e/my-robot-keeps-its-receipts-3h9g", "published_at": "2026-10-20 21:35:30+00:00", "updated_at": "2026-10-20 21:48:54.622032+00:00", "lang": "en", "topics": ["ai-research", "ai-safety"], "entities": ["drand", "Claude Haiku 4.5", "DEV", "Kaggle"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/my-robot-keeps-its-receipts", "markdown": "https://wpnews.pro/news/my-robot-keeps-its-receipts.md", "text": "https://wpnews.pro/news/my-robot-keeps-its-receipts.txt", "jsonld": "https://wpnews.pro/news/my-robot-keeps-its-receipts.jsonld"}}