Can a Lightweight Multimodal Model Estimate LLM Reasoning Performance? A Study for Compute-Optimal Document Inference

Researchers introduced BudgetDoc, the first multimodal benchmark for model-budget-performance trade-offs in document tasks, and trained DRB (Document-Reasoning Balancer), a 1B-parameter pre-flight estimator using SigLIP-2 and Qwen3-0.6B, achieving a 0.753 weighted F1. In dynamic budget allocation across five frontier models and three datasets, DRB matched or improved F1 scores in 9 of 15 configurations compared to always-maximum-budget baselines while reducing cost.

arXiv:2608.18591v1 Announce Type: cross Abstract: Uniformly allocating inference reasoning budgets to LLMs is expensive and prone to over-thinking penalties; especially in document tasks where visual layouts drive complexity. To address this, we introduce BudgetDoc, the first multimodal benchmark providing explicit supervision for model-budget-performance trade-offs across three document tasks. Using BudgetDoc, we train DRB Document-Reasoning Balancer , an approx. 1B-parameter pre-flight estimator SigLIP-2 + Qwen3-0.6B that predicts ordinal model performance across budget levels, achieving a 0.753 weighted F1. When dynamically allocating reasoning budgets across five frontier models and three datasets, DRB matches or improves F1 scores compared to always-maximum-budget baselines in 9 of 15 configurations while drastically reducing cost. Finally, preliminary evaluations demonstrate DRB's potential to generalize to cross-model selection.