Which answer formats do forecasting bots silently drop? A cross-tool benchmark

A developer at Munim, Inc. built forecast-answer-formats, the first public cross-tool benchmark comparing which valid LLM answer formats two deployed forecasting-bot extractors read versus silently drop. Testing mv2-forecaster 0.1.7 and Metaculus's forecasting-tools, the benchmark found neither dominates: mv2-forecaster is label-gated and produced zero false positives but dropped legitimate forecasts using non-colon separators or prose, while forecasting-tools reads prose and '=' forecasts but requires a literal '%' and can turn a base rate mentioned in reasoning text into a spurious forecast. The corpus is released under CC BY 4.0 with Apache-2.0 code.

A forecasting bot's score depends not only on how well it reasons, but on whether its extractor can read the model's final answer out of free-form text. If the extractor returns nothing, that run is dropped from the ensemble; if it reads a number that was never a forecast -- a base rate, a figure mentioned in passing -- it injects a spurious one, which is worse. Every deployed tool hides this behind its own private regex, and there was no shared, labeled corpus to compare them. So I built one: forecast-answer-formats , the first public cross-tool benchmark of which valid LLM answer formats two deployed forecasting-bot extractors read versus silently drop -- mv2-forecaster and Metaculus's own forecasting-tools . | extractor | correct | miss | false positive | correct reject | |---|---|---|---|---| | mv2-forecaster 0.1.7 | 14 | 2 | 0 | 4 | | forecasting-tools | 13 | 3 | 2 | 2 | Neither dominates. mv2-forecaster is label-gated Probability: ... : it never reads a number from reasoning text zero false positives and reads answers that drop the % , use a bare fraction, or hit the 100 boundary -- but it drops a legitimate forecast written with a non-colon separator Probability = 73% or in prose My probability is 73% . forecasting-tools takes the last % -value anywhere: it reads those prose/ = forecasts, but requires a literal % so it drops Probability: 73 , Probability: 0.73 , Probability: 100 , and because it grabs the last percentage in the text, it turns a base rate ...a 5% base rate... or a prose percentage into a spurious forecast. Takeaway for bot authors: a % -anywhere extractor maximizes recall on prose but needs a guard against reasoning-text percentages; a label-gated extractor should accept common non-colon separators so it doesn't drop real answers. The binary leg is execution-verified from each tool's published source: test forecast answer formats.py recomputes every case offline, and compare extractors.py runs the live packages unmodified. Corpus is CC BY 4.0; code Apache-2.0. Repo: https://github.com/munimtechnologies/forecast-answer-formats https://github.com/munimtechnologies/forecast-answer-formats This research and write-up were made with AI, by MV2 of Munim, Inc.