How Well Does GPT-4o Understand Vision? Evaluating Multimodal Foundation Models on Standard Computer Vision Tasks

Multimodal foundation models (MFMs), such as GPT-4o, have recently made remarkable progress. However, it is not clear how well these models understand vision in detail, es-pecially when it comes to tasks beyond question answering. In this paper, we benchmark the popular MFMs (GPT-4o, o4-mini, Gemini 1.5 Pro and Gemini 2.0 Flash, Claude 3.5 Sonnet, Qwen2-VL, Llama 3.2) on standard computer vi-sion tasks (semantic segmentation, object detection, image classification, depth and surface normal prediction) using es-tablished datasets (e.g., COCO, ImageNet and its variants). The main challenges in performing this analysis are: 1) most models are trained to output text and cannot natively express diverse domains, such as segments or 3D geometry, and 2) many leading models are proprietary and accessi-ble only at an API level, i.e., there is no weight access to adapt them. We address these challenges by translating standard vision tasks into equivalent text-promptable and API-compatible tasks via prompt chaining to create a stan-dardized benchmarking framework. We observe that:

1. The models are not close to the state-of-the-art specialist models at any task.
2. They are respectable generalists; this is remarkable as they are presumably trained on primarily image-text-based tasks.
3. They perform semantic tasks notably better than geomet-ric ones.
4. GPT-4o performs the best among non-reasoning models, securing the top position in 4 out of 6 tasks.
5. Reasoning models, e.g. o3, show improvements in geo-metric tasks.
6. While prompt chaining techniques affect performance, better models are less sensitive to prompt variations.
7. An analysis of models with native image generation, such as the latest GPT-4o, shows they exhibit failure modes, such as hallucinated.