Planes, Not A380: How Prompting, Context, and Granularity Shape VLM Performance in Aerial Imagery

Vision-Language Models (VLMs) have demonstrated impressive generalization abilities across various vision-and-language tasks, yet their effectiveness for object classification in remote sensing imagery remains uncertain. Accurate overhead object classification, combined with class-agnostic segmenters or region proposal networks (e.g., the Segment Anything Model), can power large-scale synthetic dataset pipelines (like LocateAnythingOnEarth), broadening category sets and improving automated object recognition. In this work, we benchmark several state-of-the-art VLMs, including GPT-4o and Gemini Flash 2.0, across multiple aerial imagery datasets (iSAID, SIOR, FAST). Our experiments reveal that while current VLMs excel in coarse-level object classification (e.g., distinguishing vehicles from buildings), their performance significantly deteriorates when tasked with fine-grained distinctions (e.g., differentiating specific aircraft models). We further show that appropriate visual prompting techniques and contextual information substantially influence model performance: coarse-grained prompts that preserve broader scene context generally outperform precise, fine-grained prompts. Moreover, we find that context can both enhance and hinder model accuracy, depending on its relevance and granularity. These findings highlight critical considerations for leveraging VLMs in remote sensing and provide actionable insights toward developing robust open-vocabulary classification frameworks suitable for aerial imagery.