TL;DR: We score the reasoning before the outcome. Each answer is rated 0 to 100 on specificity, grounding and calibration by a fixed judge. After certification we add winner accuracy, calibration curves and the Brier score. A model can be wrong and score well; a vague answer always scores badly.
The protocol is public at elections.lforla.org/methodology.html.
Every answer was collected before the vote, between 26 August and 31 October 2026, and is published verbatim. The date attached to each model is the date its forecast entered the benchmark.
This matters more than any scoring detail. If a forecast can be edited after the first results, the score measures hindsight, not forecasting.
The snapshot is sealed with SHA-256:
26c1563ea12200b9436e9ab7fd03b1af7f59fd17aea5697a2bd96a25bd38b2d9
Git tag: elections-forecasts-2026-10-31.
Each answer is rated by a fixed judge: DeepSeek V4 Pro at temperature 0, with a fixed prompt. The judge does not see the election result. It sees the question, the answer, and a rubric.
Using one judge for every answer keeps the scale comparable. It also creates a conflict: several models under test are from the same family as the judge. We disclose it rather than hide it. A future version should use a panel of judges and publish the disagreement.
Each axis is scored 0 to 100.
Does the answer name a candidate or party and state a probability?
Does the answer use evidence that is relevant to the race?
Polls, incumbency, district structure, the electoral calendar, the rules of the second round. A confident answer with no grounding is a guess with better typography.
Does the stated probability match the real uncertainty?
A 95% call on a coin-flip race is worse than a 60% call on the same race, even if both name the same winner. Calibration rewards honesty about what is unknown.
The category score is the mean of its questions. The overall score is the mean of the two categories: US 2026 and France 2027. A refusal scores 0.
The current board is dominated by models that give a named call, a probability and a reason. That is the intended behaviour. The benchmark is not neutral about vagueness.
It does not measure whether the forecast is right. That is the next layer, and it can only be computed after certification.
It also does not measure:
A model can be wrong and score well. A vague answer always scores badly. Those are different failure modes, and conflating them is how forecast benchmarks become entertainment.
Once results are certified, every probabilistic call becomes a number.
The share of races where the model named the winning party or candidate. Simple, intuitive, and easy to game with a safe consensus pick.
Group calls by stated probability and compare with outcomes. Did 70% calls happen about 70% of the time? With 18 models and two races, the sample is too small for a real calibration curve. We will publish the curve with that caveat, not instead of it.
The Brier score is the mean squared error of the probability assigned to the outcome.
For a binary race, convert every call to the implied probability of Democratic control:
P(dem) = 0.70; P(dem) = 0.30.
Then:
Brier = (P(dem) - y)^2
where y = 1 if Democrats control the chamber and y = 0 otherwise.
Examples:
| Call | Outcome | Brier |
|---|---|---|
| 70% Democratic | Democrats win | 0.09 |
| 70% Democratic | Republicans win | 0.49 |
| 55% Democratic | Republicans win | 0.3025 |
| 55% Democratic | Democrats win | 0.2025 |
Lower is better. The important property is that a confident wrong call costs more than a hedged one. That is what makes the score useful: it prices false confidence.
The forecast file and its hash are published in the repository. If you find a scoring bug, the answer is not to adjust the result; it is to publish the correction and re-run the score on the same sealed inputs.
Originally published at elections.lforla.org.