Lecture

Transformers: Unifying Machine Learning Communities

Graph Chat

Chat with Graph Search

Ask any question about EPFL courses, lectures, exercises, research, news, etc. or try the example questions below.

DISCLAIMER: The Graph Chatbot is not programmed to provide explicit or categorical answers to your questions. Rather, it transforms your questions into API requests that are distributed across the various IT services officially administered by EPFL. Its purpose is solely to collect and recommend relevant references to content that you can explore to help you answer your questions.

Description

This lecture discusses the concept of Transformers in machine learning, highlighting their role in unifying various fields such as computer vision, natural language processing, and reinforcement learning. The instructor explains how Transformers emerged in 2017, revolutionizing the way different communities approached machine learning tasks. Initially designed for translation tasks, Transformers have since been adapted for image classification and semantic segmentation, demonstrating their versatility and effectiveness. The lecture covers the architecture of Transformers, including the encoder-decoder structure, and the importance of tokenization and positional encoding. The instructor elaborates on the self-attention mechanism, which allows the model to weigh the relevance of different tokens, and the multi-head attention that enhances the model's ability to capture diverse information. The discussion also touches on the significance of attention mechanisms in processing sequences and the implications for future developments in machine learning. Overall, the lecture provides a comprehensive overview of how Transformers have transformed the landscape of machine learning.

This video is available exclusively on Mediaspace for a restricted audience. Please log in to MediaSpace to access it if you have the necessary permissions.

Watch on Mediaspace

Instructor

Amir Roshan Zamir

Amir Zamir is a Tenure-Track Assistant Professor of Computer Science at the Swiss Federal Institute of Technology in Lausanne (EPFL). Prior to EPFL, he spent time at Stanford, UC Berkeley, and UCF. His research interests are broadly in computer vision, machine learning, perception-for-robotics, and AI. He has been recognized with CVPR (2018) Best Paper Award, CVPR (2016) Best Student Paper Award, CVPR (2020) Best Paper Award Nomination, and NVIDIA Pioneering Research Award (2018), among others. His research has been covered by various press outlets, such as The New York Times or Forbes.

Official source

Related concepts (32)

Attention (machine learning)

Machine learning-based attention is a mechanism mimicking cognitive attention. It calculates "soft" weights for each word, more precisely for its embedding, in the context window. It can do it either in parallel (such as in transformers) or sequentially (such as recursive neural networks). "Soft" weights can change during each runtime, in contrast to "hard" weights, which are (pre-)trained and fine-tuned and remain frozen afterwards. Multiple attention heads are used in transformer-based large language models.

Computer vision

Computer vision tasks include methods for , , and understanding digital images, and extraction of high-dimensional data from the real world in order to produce numerical or symbolic information, e.g. in the forms of decisions. Understanding in this context means the transformation of visual images (the input to the retina in the human analog) into descriptions of the world that make sense to thought processes and can elicit appropriate action.

Transformer (machine learning model)

A transformer is a deep learning architecture that relies on the parallel multi-head attention mechanism. The modern transformer was proposed in the 2017 paper titled 'Attention Is All You Need' by Ashish Vaswani et al., Google Brain team. It is notable for requiring less training time than previous recurrent neural architectures, such as long short-term memory (LSTM), and its later variation has been prevalently adopted for training large language models on large (language) datasets, such as the Wikipedia corpus and Common Crawl, by virtue of the parallelized processing of input sequence.

Natural language processing

Natural language processing (NLP) is an interdisciplinary subfield of linguistics and computer science. It is primarily concerned with processing natural language datasets, such as text corpora or speech corpora, using either rule-based or probabilistic (i.e. statistical and, most recently, neural network-based) machine learning approaches. The goal is a computer capable of "understanding" the contents of documents, including the contextual nuances of the language within them.

Deep reinforcement learning

Deep reinforcement learning (deep RL) is a subfield of machine learning that combines reinforcement learning (RL) and deep learning. RL considers the problem of a computational agent learning to make decisions by trial and error. Deep RL incorporates deep learning into the solution, allowing agents to make decisions from unstructured input data without manual engineering of the state space. Deep RL algorithms are able to take in very large inputs (e.g.