Zero-Shot Mission-Level Evaluation for Aerial MLLM Agents

A new benchmark called MissionBench reveals that even the strongest multimodal large language models (MLLMs) succeed on fewer than 35% of aerial missions, compared to 84.4% human performance, according to a preprint on arXiv. The benchmark comprises 120 missions across five simulated 3D environments and four task families, testing agents on long-horizon embodied tasks without aerial-specific fine-tuning. The study, which evaluated 22 open- and closed-source MLLMs, found that scaling improves zero-shot capabilities but that mission-level competence requires coordinating multiple skills beyond spatial perception.

arXiv:2607.22014v1 Announce Type: cross Abstract: Multimodal Large Language Models MLLMs are emerging as core reasoning modules for embodied agents, yet it remains unclear how well general-purpose models can solve long-horizon embodied tasks from a single high-level instruction. We introduce MissionBench, a benchmark for mission-level evaluation of MLLMs in aerial 3D environments. It comprises 120 missions across five simulated 3D environments and four task families. Agents must autonomously plan, navigate, and report outcomes using only egocentric observations and its action history, without aerial-specific fine-tuning. Across 22 open- and closed-source MLLMs, the strongest model succeeds on fewer than 35% of missions compared to 84.4% human performance, highlighting the difficulty of multi-step embodied tasks. Despite large variations between model families, we observe gains from scaling, indicating that larger general-purpose models possess stronger zero-shot embodied capabilities. Our analysis shows that mission-level competence requires coordinating multiple capabilities beyond spatial perception, including multi-step planning and adaptive reasoning. This motivates closed-loop evaluation and highlights both the promise and risk of scaling-driven improvements for embodied AI.