NavBench: Probing Multimodal Large Language Models for Embodied Navigation

Multimodal Large Language Models (MLLMs) have demonstrated strong gener- alization in vision-language tasks, yet their ability to understand and act within embodied environments remains underexplored. We present NavBench, a bench- mark to evaluate the embodied navigation capabilities of MLLMs under zero-shot settings. NavBench consists of two components: (1) navigation comprehension, assessed through three cognitively grounded tasks including global instruction alignment, temporal progress estimation, and local observation-action reason- ing, covering 3,200 question-answer pairs; and (2) step-by-step execution in 432 episodes across 72 indoor scenes, stratified by spatial, cognitive, and execution complexity. To support real-world deployment, we introduce a pipeline that con- verts MLLMs’ outputs into robotic actions. We evaluate both proprietary and open-source models, finding that GPT-4o performs well across tasks, while lighter open-source models succeed in simpler cases. Results also show that models with higher comprehension scores tend to achieve better execution performance. Provid- ing map-based context improves decision accuracy, especially in medium-difficulty scenarios. However, most models struggle with temporal understanding, particu- larly in estimating progress during navigation, which may pose a key challenge.