Lecture

Transformers in Vision

Graph Chat

Chat with Graph Search

Ask any question about EPFL courses, lectures, exercises, research, news, etc. or try the example questions below.

DISCLAIMER: The Graph Chatbot is not programmed to provide explicit or categorical answers to your questions. Rather, it transforms your questions into API requests that are distributed across the various IT services officially administered by EPFL. Its purpose is solely to collect and recommend relevant references to content that you can explore to help you answer your questions.

Description

This lecture covers the application of transformers in vision tasks, focusing on ViTs and their adaptation to dense prediction tasks. It explores the challenges faced by transformers in vision, such as quadratic complexity and the need for global receptive fields. The instructor discusses the performance of ViTs in comparison to traditional ConvNets, emphasizing the importance of massive data for training. Additionally, the lecture introduces innovative transformer architectures like Swin Transformer, Dense Prediction Transformers, and Perceiver, highlighting their advantages in handling structured inputs and outputs. Various transformer models, including MLP-Mixer and ConvMixer, are presented, showcasing the evolution of transformer-based architectures in computer vision.

This video is available exclusively on Mediaspace for a restricted audience. Please log in to MediaSpace to access it if you have the necessary permissions.

Watch on Mediaspace

Official source

Related concepts (30)

Computer vision

Computer vision tasks include methods for , , and understanding digital images, and extraction of high-dimensional data from the real world in order to produce numerical or symbolic information, e.g. in the forms of decisions. Understanding in this context means the transformation of visual images (the input to the retina in the human analog) into descriptions of the world that make sense to thought processes and can elicit appropriate action.

Vision science

Vision science is the scientific study of visual perception. Researchers in vision science can be called vision scientists, especially if their research spans some of the science's many disciplines. Vision science encompasses all studies of vision, such as how human and non-human organisms process visual information, how conscious visual perception works in humans, how to exploit visual perception for effective communication, and how artificial systems can do the same tasks.

Convolutional neural network

Convolutional neural network (CNN) is a regularized type of feed-forward neural network that learns feature engineering by itself via filters (or kernel) optimization. Vanishing gradients and exploding gradients, seen during backpropagation in earlier neural networks, are prevented by using regularized weights over fewer connections. For example, for each neuron in the fully-connected layer 10,000 weights would be required for processing an image sized 100 × 100 pixels.

Computer stereo vision

Computer stereo vision is the extraction of 3D information from digital images, such as those obtained by a CCD camera. By comparing information about a scene from two vantage points, 3D information can be extracted by examining the relative positions of objects in the two panels. This is similar to the biological process of stereopsis. In traditional stereo vision, two cameras, displaced horizontally from one another, are used to obtain two differing views on a scene, in a manner similar to human binocular vision.

Visual cortex

The visual cortex of the brain is the area of the cerebral cortex that processes visual information. It is located in the occipital lobe. Sensory input originating from the eyes travels through the lateral geniculate nucleus in the thalamus and then reaches the visual cortex. The area of the visual cortex that receives the sensory input from the lateral geniculate nucleus is the primary visual cortex, also known as visual area 1 (V1), Brodmann area 17, or the striate cortex.

Related lectures (63)

Machine Learning for Feature Extraction

Explores machine learning for feature extraction, 3D vision, and neural networks in mobile robotics.

Deep Learning: Edge Detection and Neural Networks

Discusses edge detection techniques and the evolution of deep learning in neural networks.

Edge Detection: Deep Learning Insights

Explores the evolution of edge detection techniques, from Canny to deep learning insights.

Vision Research to Clinical Therapy

Explores vision research, optic nerve regeneration, and bio-realistic models in the mouse visual cortex, focusing on optic nerve stimulation and visual learning.

Visual Prostheses: History and Intracortical Stimulation