Recurrent Convolutional Neural Networks for Scene Labeling

The goal of the scene labeling task is to assign a class label to each pixel in an image. To ensure a good visual coherence and a high class accu- racy, it is essential for a model to capture long range (pixel) label dependencies in images. In a feed-forward architecture, this can be achieved simply by considering a sufficiently large input context patch, around each pixel to be labeled. We propose an approach that consists of a re- current convolutional neural network which al- lows us to consider a large input context while limiting the capacity of the model. Contrary to most standard approaches, our method does not rely on any segmentation technique nor any task- specific features. The system is trained in an end-to-end manner over raw pixels, and mod- els complex spatial dependencies with low infer- ence cost. As the context size increases with the built-in recurrence, the system identifies and cor- rects its own errors. Our approach yields state-of- the-art performance on both the Stanford Back- ground Dataset and the SIFT Flow Dataset, while remaining very fast at test time.