MHE-Former: Multi-Hypothesis Transformers via Entropy Maximization for 3D Mesh Recovery

Researchers introduced MHE-Former, a Transformer-based multi-hypothesis framework that uses entropy maximization to generate diverse 3D hand and body mesh recovery predictions from monocular input, according to a paper posted to arXiv as 2609.10743v1. The framework pairs an exploration phase, which produces plausible and diverse hypotheses, with a context-aware Hypothesis Selection process that leverages a vision-language model's visual understanding and reasoning so users can pick the most plausible estimate using additional evidence and natural language intent. Experiments across multiple datasets show state-of-the-art accuracy and diversity, and a user preference study supports the practicality of the selection process.

arXiv:2609.10743v1 Announce Type: new Abstract: Monocular 3D hand and body mesh recovery often suffers from severe occlusion and ambiguity. Traditional deterministic methods typically regress a single optimal solution, leading to overconfident predictions. In this paper, we introduce an exploration--exploitation paradigm for ambiguous mesh recovery with multi-hypothesis learning and selection. Specifically, during exploration, based on our probabilistic formulation and entropy maximization, we propose a novel multi-hypothesis method referred to as MHE-Former. It is a Transformer-based multi-hypothesis framework, ensuring high training efficiency and label friendliness while generating plausible and diverse hypotheses. During exploitation, we propose Hypothesis Selection, a context-aware process for multiple predictions. Especially leveraging VLM's powerful visual understanding and reasoning capabilities, it allows users to choose the most plausible and desired estimate with additional evidence and natural language intent. Extensive experiments demonstrate that our framework achieves state-of-the-art performance in accuracy and diversity across multiple datasets. The user preference study further shows the practicality of our hypothesis selection process.