Gaze is one of the most revealing and least understood windows into human cognition. Where people look - toward an object or another person - encapsulates attention, coordination, and interaction. In computer vision, however, gaze has long been reduced to a geometric regression problem: predicting a point on an image plane. This formulation captures spatial direction but overlooks the broader utility of gaze as a cue for understanding behavior. This work aims to make gaze understanding more informative and actionable by improving accuracy, scalability, and interpretability through richer geometric, semantic, and uncertainty modeling.
This thesis redefines gaze following as a bridge between vision and social understanding. It develops computational models that reason not only about gaze target locations, but about the attentional relationships linking people, objects, and their environment. Across data, geometry, semantics, and uncertainty, it presents a unified view of gaze as a structured reasoning problem situated in complex, natural scenes.
At the data level, we introduce new benchmarks that capture multiple layers of visual and social context in natural settings. At the modeling level, we propose several architectural advances that (i) couple spatial geometry with semantic understanding to jointly infer gaze direction and target semantics, (ii) extend gaze following to multi-person scenes without compromising efficiency, allowing real-time deployment, and (iii) model uncertainty explicitly by separating inherent ambiguity in gaze from epistemic uncertainty in model reasoning, resulting in more reliable and interpretable predictions.
The intellectual trajectory of this work follows a natural progression - from pixels to attention. It begins with geometric grounding, then moves toward semantic and social reasoning. Along the way, the thesis reconsiders both the practical utility and ethical framing of gaze prediction, emphasizing accuracy, robustness, and responsible use in socially sensitive domains.
Beyond technical advances, this research contributes a conceptual shift: gaze following is not merely a visual task but a step toward socially intelligent perception. By modeling attention as a relational construct linking vision, semantics, and social inference, it moves gaze understanding closer to the human cognitive process it seeks to emulate. From pixels to attention, it lays the foundations for vision systems capable of perceiving, reasoning, and ultimately engaging with humans in the shared spaces of the real world.
