Lecture

Seq2Seq Models: Attention vs. No Attention

Graph Chat

Chat with Graph Search

Ask any question about EPFL courses, lectures, exercises, research, news, etc. or try the example questions below.

DISCLAIMER: The Graph Chatbot is not programmed to provide explicit or categorical answers to your questions. Rather, it transforms your questions into API requests that are distributed across the various IT services officially administered by EPFL. Its purpose is solely to collect and recommend relevant references to content that you can explore to help you answer your questions.

In course

EE-608: Deep Learning For Natural Language Processing

This course covers advanced topics in deep learning architectures for natural language processing. The focus is on attention-based architectures, structure processing and variational-Bayesian approaches, and why these models are particularly suited to the ...

Description

This lecture covers the implementation of Sequence-to-Sequence (Seq2Seq) models with and without attention mechanisms. It explains the architecture of encoder-decoder models, the role of context vectors, and the process of decoding source sentences to target sentences. The lecture also delves into the use of bidirectional RNNs, teacher forcing, and different types of attention mechanisms like additive and self-attention. Additionally, it discusses the importance of weight initialization, training loops, and the applications of attention mechanisms in various ML and NLP tasks.

Instructor

James Henderson

Teaching & PhD PhD Students Haruki Shirakami, Dina El Zein, Jose Rafael Espinosa Mena, Kimia Alavi, Mojtaba Nafez Past EPFL PhD Students as codirector Lesly Sadiht Miculicich Werlen, Rabeeh Karimi Mahabadi, Alireza Mohammadshahi, Andreas Thomas Marfurt, Jan Frederik Jonas Florian Mai, Melika Behjati, Fabio James Fehr, Andrei Catalin Coman Courses Deep Learning For Natural Language Processing EE-608 This course covers advanced topics in deep learning architectures for natural language proce

Official source

Ontological neighbourhood

Information engineering

Machine learning: Artificial neural networks

Related concepts (29)

Deep learning

Deep learning is part of a broader family of machine learning methods, which is based on artificial neural networks with representation learning. The adjective "deep" in deep learning refers to the use of multiple layers in the network. Methods used can be either supervised, semi-supervised or unsupervised.

Types of artificial neural networks

There are many types of artificial neural networks (ANN). Artificial neural networks are computational models inspired by biological neural networks, and are used to approximate functions that are generally unknown. Particularly, they are inspired by the behaviour of neurons and the electrical signals they convey between input (such as from the eyes or nerve endings in the hand), processing, and output from the brain (such as reacting to light, touch, or heat). The way neurons semantically communicate is an area of ongoing research.

Generative adversarial network

A generative adversarial network (GAN) is a class of machine learning framework and a prominent framework for approaching generative AI. The concept was initially developed by Ian Goodfellow and his colleagues in June 2014. In a GAN, two neural networks contest with each other in the form of a zero-sum game, where one agent's gain is another agent's loss. Given a training set, this technique learns to generate new data with the same statistics as the training set.

Natural language processing

Natural language processing (NLP) is an interdisciplinary subfield of linguistics and computer science. It is primarily concerned with processing natural language datasets, such as text corpora or speech corpora, using either rule-based or probabilistic (i.e. statistical and, most recently, neural network-based) machine learning approaches. The goal is a computer capable of "understanding" the contents of documents, including the contextual nuances of the language within them.

Transformer (machine learning model)

A transformer is a deep learning architecture that relies on the parallel multi-head attention mechanism. The modern transformer was proposed in the 2017 paper titled 'Attention Is All You Need' by Ashish Vaswani et al., Google Brain team. It is notable for requiring less training time than previous recurrent neural architectures, such as long short-term memory (LSTM), and its later variation has been prevalently adopted for training large language models on large (language) datasets, such as the Wikipedia corpus and Common Crawl, by virtue of the parallelized processing of input sequence.

Related lectures (64)

Foundations of Deep Learning: Transformer Architecture Overview

Covers the foundational concepts of deep learning and the Transformer architecture, focusing on neural networks, attention mechanisms, and their applications in sequence modeling tasks.

Binary Sentiment Classifier Training

Covers the training of a binary sentiment classifier using an RNN.

Sequence to Sequence Models: Overview and Applications

Covers sequence to sequence models, their architecture, applications, and the role of attention mechanisms in improving performance.

Language Modelling and Recurrent Neural Networks

Explores language modelling, RNNs, n-gram models, LSTMs, and bidirectional RNNs.

Recurrent Neural Networks: Training and Challenges