Lecture

From Attention to Transformers

Graph Chat

Chat with Graph Search

Ask any question about EPFL courses, lectures, exercises, research, news, etc. or try the example questions below.

DISCLAIMER: The Graph Chatbot is not programmed to provide explicit or categorical answers to your questions. Rather, it transforms your questions into API requests that are distributed across the various IT services officially administered by EPFL. Its purpose is solely to collect and recommend relevant references to content that you can explore to help you answer your questions.

Description

This lecture covers the transition from attention mechanisms to transformers in modern NLP. Starting with encoder-decoder models, the instructor explains the concept of attention and its enhancement through transformers. The lecture delves into the generation of sequences using decoding from sequence to sequence models. The attentive encoder-decoder model is introduced, highlighting the reduction of temporal bottlenecks and the use of attention to focus on different parts of the input. The attention function, including multiplicative, linear, and scaled dot product attention, is discussed. The lecture concludes with a detailed explanation of self-attention, masked multi-headed attention, and cross-attention in transformers.

Official source

Related concepts (32)

Attention (machine learning)

Machine learning-based attention is a mechanism mimicking cognitive attention. It calculates "soft" weights for each word, more precisely for its embedding, in the context window. It can do it either in parallel (such as in transformers) or sequentially (such as recursive neural networks). "Soft" weights can change during each runtime, in contrast to "hard" weights, which are (pre-)trained and fine-tuned and remain frozen afterwards. Multiple attention heads are used in transformer-based large language models.

Attention

Attention is the concentration of awareness on some phenomenon to the exclusion of other stimuli. It is a process of selectively concentrating on a discrete aspect of information, whether considered subjective or objective. William James (1890) wrote that "Attention is the taking possession by the mind, in clear and vivid form, of one out of what seem several simultaneously possible objects or trains of thought. Focalization, concentration, of consciousness are of its essence.

Transformer (machine learning model)

A transformer is a deep learning architecture that relies on the parallel multi-head attention mechanism. The modern transformer was proposed in the 2017 paper titled 'Attention Is All You Need' by Ashish Vaswani et al., Google Brain team. It is notable for requiring less training time than previous recurrent neural architectures, such as long short-term memory (LSTM), and its later variation has been prevalently adopted for training large language models on large (language) datasets, such as the Wikipedia corpus and Common Crawl, by virtue of the parallelized processing of input sequence.

Natural language processing

Natural language processing (NLP) is an interdisciplinary subfield of linguistics and computer science. It is primarily concerned with processing natural language datasets, such as text corpora or speech corpora, using either rule-based or probabilistic (i.e. statistical and, most recently, neural network-based) machine learning approaches. The goal is a computer capable of "understanding" the contents of documents, including the contextual nuances of the language within them.

Self-harm

Self-harm is intentional behavior that is considered harmful to oneself. This is most commonly regarded as direct injury of one's own skin tissues usually without a suicidal intention. Other terms such as cutting, self-injury, and self-mutilation have been used for any self-harming behavior regardless of suicidal intent. The most common form of self-harm is using a sharp object to cut the skin. Other forms include scratching, hitting, or burning body parts.

Related lectures (64)

Sequence to Sequence Models: Overview and Applications

Covers sequence to sequence models, their architecture, applications, and the role of attention mechanisms in improving performance.

Transformers: Pretraining and Decoding Techniques

Covers advanced transformer concepts, focusing on pretraining and decoding techniques in NLP.

Sequence to Sequence Models: Overview and Attention Mechanisms

Explores sequence to sequence models, attention mechanisms, and their role in addressing model limitations and improving interpretability.

Transformers: Revolutionizing Attention Mechanisms in NLP

Covers the development of transformers and their impact on attention mechanisms in NLP.

Pretraining: Transformers & Models