{"slug": "how-we-score-an-election-forecast-and-what-the-score-does-not-measure", "title": "How we score an election forecast (and what the score does not measure)", "summary": "A project at elections.lforla.org published a public methodology for scoring AI election forecasts on reasoning quality rather than outcomes, rating each pre-vote answer 0 to 100 on specificity, grounding and calibration using a fixed judge (DeepSeek V4 Pro at temperature 0) that never sees the result. Forecasts were collected between 26 August and 31 October 2026 and sealed with a SHA-256 hash and git tag, with winner accuracy, calibration curves and the Brier score added only after certification. The project discloses that several models under test share a family with the judge and says a future version should use a panel of judges and publish their disagreement.", "body_md": "**TL;DR:** We score the reasoning before the outcome. Each answer is rated 0 to 100 on specificity, grounding and calibration by a fixed judge. After certification we add winner accuracy, calibration curves and the Brier score. A model can be wrong and score well; a vague answer always scores badly.\n\nThe protocol is public at **[elections.lforla.org/methodology.html](https://elections.lforla.org/methodology.html)**.\n\nEvery answer was collected before the vote, between 26 August and 31 October 2026, and is published verbatim. The date attached to each model is the date its forecast entered the benchmark.\n\nThis matters more than any scoring detail. If a forecast can be edited after the first results, the score measures hindsight, not forecasting.\n\nThe snapshot is sealed with SHA-256:\n\n```\n26c1563ea12200b9436e9ab7fd03b1af7f59fd17aea5697a2bd96a25bd38b2d9\n```\n\nGit tag: `elections-forecasts-2026-10-31`.\n\nEach answer is rated by a fixed judge: **DeepSeek V4 Pro at temperature 0**, with a fixed prompt. The judge does not see the election result. It sees the question, the answer, and a rubric.\n\nUsing one judge for every answer keeps the scale comparable. It also creates a conflict: several models under test are from the same family as the judge. We disclose it rather than hide it. A future version should use a panel of judges and publish the disagreement.\n\nEach axis is scored 0 to 100.\n\nDoes the answer name a candidate or party and state a probability?\n\nDoes the answer use evidence that is relevant to the race?\n\nPolls, incumbency, district structure, the electoral calendar, the rules of the second round. A confident answer with no grounding is a guess with better typography.\n\nDoes the stated probability match the real uncertainty?\n\nA 95% call on a coin-flip race is worse than a 60% call on the same race, even if both name the same winner. Calibration rewards honesty about what is unknown.\n\nThe category score is the mean of its questions. The overall score is the mean of the two categories: US 2026 and France 2027. A refusal scores 0.\n\nThe current board is dominated by models that give a named call, a probability and a reason. That is the intended behaviour. The benchmark is not neutral about vagueness.\n\nIt does not measure whether the forecast is right. That is the next layer, and it can only be computed after certification.\n\nIt also does not measure:\n\nA model can be wrong and score well. A vague answer always scores badly. Those are different failure modes, and conflating them is how forecast benchmarks become entertainment.\n\nOnce results are certified, every probabilistic call becomes a number.\n\nThe share of races where the model named the winning party or candidate. Simple, intuitive, and easy to game with a safe consensus pick.\n\nGroup calls by stated probability and compare with outcomes. Did 70% calls happen about 70% of the time? With 18 models and two races, the sample is too small for a real calibration curve. We will publish the curve with that caveat, not instead of it.\n\nThe Brier score is the mean squared error of the probability assigned to the outcome.\n\nFor a binary race, convert every call to the implied probability of Democratic control:\n\n`P(dem) = 0.70`;` P(dem) = 0.30`.\nThen:\n\n```\nBrier = (P(dem) - y)^2\n```\n\nwhere `y = 1` if Democrats control the chamber and `y = 0` otherwise.\n\nExamples:\n\n| Call | Outcome | Brier | \n|---|---|---|\n| 70% Democratic | Democrats win | 0.09 | \n| 70% Democratic | Republicans win | 0.49 | \n| 55% Democratic | Republicans win | 0.3025 | \n| 55% Democratic | Democrats win | 0.2025 | \n\nLower is better. The important property is that a confident wrong call costs more than a hedged one. That is what makes the score useful: it prices false confidence.\n\nThe forecast file and its hash are published in the repository. If you find a scoring bug, the answer is not to adjust the result; it is to publish the correction and re-run the score on the same sealed inputs.\n\n*Originally published at [elections.lforla.org](https://elections.lforla.org/blog/how-we-score-forecasts.html).*", "url": "https://wpnews.pro/news/how-we-score-an-election-forecast-and-what-the-score-does-not-measure", "canonical_source": "https://dev.to/resk/how-we-score-an-election-forecast-and-what-the-score-does-not-measure-2lcl", "published_at": "2026-11-02 17:10:02+00:00", "updated_at": "2026-11-02 17:23:58.451297+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-research"], "entities": ["DeepSeek", "DeepSeek V4 Pro", "elections.lforla.org"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/how-we-score-an-election-forecast-and-what-the-score-does-not-measure", "markdown": "https://wpnews.pro/news/how-we-score-an-election-forecast-and-what-the-score-does-not-measure.md", "text": "https://wpnews.pro/news/how-we-score-an-election-forecast-and-what-the-score-does-not-measure.txt", "jsonld": "https://wpnews.pro/news/how-we-score-an-election-forecast-and-what-the-score-does-not-measure.jsonld"}}