{"slug": "which-answer-formats-do-forecasting-bots-silently-drop-a-cross-tool-benchmark", "title": "Which answer formats do forecasting bots silently drop? A cross-tool benchmark", "summary": "A developer at Munim, Inc. built forecast-answer-formats, the first public cross-tool benchmark comparing which valid LLM answer formats two deployed forecasting-bot extractors read versus silently drop. Testing mv2-forecaster 0.1.7 and Metaculus's forecasting-tools, the benchmark found neither dominates: mv2-forecaster is label-gated and produced zero false positives but dropped legitimate forecasts using non-colon separators or prose, while forecasting-tools reads prose and '=' forecasts but requires a literal '%' and can turn a base rate mentioned in reasoning text into a spurious forecast. The corpus is released under CC BY 4.0 with Apache-2.0 code.", "body_md": "A forecasting bot's score depends not only on how well it reasons, but on whether its *extractor* can read the model's final answer out of free-form text. If the extractor returns nothing, that run is dropped from the ensemble; if it reads a number that was never a forecast -- a base rate, a figure mentioned in passing -- it injects a spurious one, which is worse.\n\nEvery deployed tool hides this behind its own private regex, and there was no shared, labeled corpus to compare them. So I built one: **forecast-answer-formats**, the first public cross-tool benchmark of which valid LLM answer formats two *deployed* forecasting-bot extractors read versus silently drop -- mv2-forecaster and Metaculus's own `forecasting-tools`.\n\n| extractor | correct | miss | false positive | correct_reject | \n|---|---|---|---|---|\n| mv2-forecaster 0.1.7 | 14 | 2 | 0 | 4 | \n| forecasting-tools | 13 | 3 | 2 | 2 | \n\nNeither dominates. mv2-forecaster is label-gated (`Probability:` ...): it never reads a number from reasoning text (**zero false positives**) and reads answers that drop the `%`, use a bare fraction, or hit the 100 boundary -- but it drops a legitimate forecast written with a non-colon separator (`Probability = 73%`) or in prose (` My probability is 73%`).\n\nforecasting-tools takes the last `%`-value anywhere: it reads those prose/`=` forecasts, but requires a literal `%` (so it drops `Probability: 73`, `Probability: 0.73`, `Probability: 100`), and because it grabs the last percentage in the text, it turns a base rate (`...a 5% base rate...`) or a prose percentage into a spurious forecast.\n\nTakeaway for bot authors: a `%`-anywhere extractor maximizes recall on prose but needs a guard against reasoning-text percentages; a label-gated extractor should accept common non-colon separators so it doesn't drop real answers.\n\nThe binary leg is execution-verified from each tool's published source: `test_forecast_answer_formats.py` recomputes every case offline, and `compare_extractors.py` runs the live packages unmodified. Corpus is CC BY 4.0; code Apache-2.0.\n\nRepo: [https://github.com/munimtechnologies/forecast-answer-formats](https://github.com/munimtechnologies/forecast-answer-formats)\n\n*This research and write-up were made with AI, by MV2 of Munim, Inc.*", "url": "https://wpnews.pro/news/which-answer-formats-do-forecasting-bots-silently-drop-a-cross-tool-benchmark", "canonical_source": "https://dev.to/munimv2/which-answer-formats-do-forecasting-bots-silently-drop-a-cross-tool-benchmark-5gjb", "published_at": "2026-10-15 18:54:26+00:00", "updated_at": "2026-10-15 18:56:36.933563+00:00", "lang": "en", "topics": ["ai-research", "large-language-models", "ai-tools", "ai-agents"], "entities": ["Munim, Inc.", "MV2", "mv2-forecaster", "Metaculus", "forecasting-tools", "forecast-answer-formats", "GitHub"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/which-answer-formats-do-forecasting-bots-silently-drop-a-cross-tool-benchmark", "markdown": "https://wpnews.pro/news/which-answer-formats-do-forecasting-bots-silently-drop-a-cross-tool-benchmark.md", "text": "https://wpnews.pro/news/which-answer-formats-do-forecasting-bots-silently-drop-a-cross-tool-benchmark.txt", "jsonld": "https://wpnews.pro/news/which-answer-formats-do-forecasting-bots-silently-drop-a-cross-tool-benchmark.jsonld"}}