Simultaneous Segmentation and Object Behavior Classification of Image Sequences

In this paper, we advance the state of the art in variational image segmentation through the fusion of bottom-up segmentation and top-down classification of object behavior over an image sequence. Such an approach is beneficial for both tasks and is carried out through a joint optimization, which enables the two tasks to cooperate, such that knowledge relevant to each can aid in the resolution of the other, thus enhancing the final result. In particular, classification offers dynamic probabilistic priors to guide segmentation, while segmentation supplies its results to classification, ensuring that they are consistent with prior knowledge. The prior models are learned from training data and they adapt dynamically, based on segmentations of earlier images in the sequence. We demonstrate the power of our approach in a hand gesture recognition application, where the combined use of segmentation and classification dramatically improves robustness in the presence of occlusion and background complexity.