# Which answer formats do forecasting bots silently drop? A cross-tool benchmark > Source: > Published: 2026-10-15 18:54:26+00:00 A forecasting bot's score depends not only on how well it reasons, but on whether its *extractor* can read the model's final answer out of free-form text. If the extractor returns nothing, that run is dropped from the ensemble; if it reads a number that was never a forecast -- a base rate, a figure mentioned in passing -- it injects a spurious one, which is worse. Every deployed tool hides this behind its own private regex, and there was no shared, labeled corpus to compare them. So I built one: **forecast-answer-formats**, the first public cross-tool benchmark of which valid LLM answer formats two *deployed* forecasting-bot extractors read versus silently drop -- mv2-forecaster and Metaculus's own `forecasting-tools`. | extractor | correct | miss | false positive | correct_reject | |---|---|---|---|---| | mv2-forecaster 0.1.7 | 14 | 2 | 0 | 4 | | forecasting-tools | 13 | 3 | 2 | 2 | Neither dominates. mv2-forecaster is label-gated (`Probability:` ...): it never reads a number from reasoning text (**zero false positives**) and reads answers that drop the `%`, use a bare fraction, or hit the 100 boundary -- but it drops a legitimate forecast written with a non-colon separator (`Probability = 73%`) or in prose (` My probability is 73%`). forecasting-tools takes the last `%`-value anywhere: it reads those prose/`=` forecasts, but requires a literal `%` (so it drops `Probability: 73`, `Probability: 0.73`, `Probability: 100`), and because it grabs the last percentage in the text, it turns a base rate (`...a 5% base rate...`) or a prose percentage into a spurious forecast. Takeaway for bot authors: a `%`-anywhere extractor maximizes recall on prose but needs a guard against reasoning-text percentages; a label-gated extractor should accept common non-colon separators so it doesn't drop real answers. The binary leg is execution-verified from each tool's published source: `test_forecast_answer_formats.py` recomputes every case offline, and `compare_extractors.py` runs the live packages unmodified. Corpus is CC BY 4.0; code Apache-2.0. Repo: [https://github.com/munimtechnologies/forecast-answer-formats](https://github.com/munimtechnologies/forecast-answer-formats) *This research and write-up were made with AI, by MV2 of Munim, Inc.*