Lecture

Graph-to-Graph Transformers: Syntax-aware Graph Encoding

Graph Chat

Chat with Graph Search

Ask any question about EPFL courses, lectures, exercises, research, news, etc. or try the example questions below.

DISCLAIMER: The Graph Chatbot is not programmed to provide explicit or categorical answers to your questions. Rather, it transforms your questions into API requests that are distributed across the various IT services officially administered by EPFL. Its purpose is solely to collect and recommend relevant references to content that you can explore to help you answer your questions.

In course

EE-608: Deep Learning For Natural Language Processing

This course covers advanced topics in deep learning architectures for natural language processing. The focus is on attention-based architectures, structure processing and variational-Bayesian approaches, and why these models are particularly suited to the ...

Description

This lecture covers the Syntax-aware Graph-to-Graph Transformer architecture, which improves the input of graph relations into the self-attention mechanism of the Transformer model. It enables effective conditioning on syntactic dependency graphs for predicting both dependency-based and span-based semantic role labelling graphs.

Instructor

James Henderson

Teaching & PhD PhD Students Haruki Shirakami, Dina El Zein, Jose Rafael Espinosa Mena, Kimia Alavi, Mojtaba Nafez Past EPFL PhD Students as codirector Lesly Sadiht Miculicich Werlen, Rabeeh Karimi Mahabadi, Alireza Mohammadshahi, Andreas Thomas Marfurt, Jan Frederik Jonas Florian Mai, Melika Behjati, Fabio James Fehr, Andrei Catalin Coman Courses Deep Learning For Natural Language Processing EE-608 This course covers advanced topics in deep learning architectures for natural language proce

Official source

Ontological neighbourhood

Mathematics

Discrete mathematics: Graph theory

Related concepts (32)

Transformer (machine learning model)

A transformer is a deep learning architecture that relies on the parallel multi-head attention mechanism. The modern transformer was proposed in the 2017 paper titled 'Attention Is All You Need' by Ashish Vaswani et al., Google Brain team. It is notable for requiring less training time than previous recurrent neural architectures, such as long short-term memory (LSTM), and its later variation has been prevalently adopted for training large language models on large (language) datasets, such as the Wikipedia corpus and Common Crawl, by virtue of the parallelized processing of input sequence.

Graph embedding

In topological graph theory, an embedding (also spelled imbedding) of a graph on a surface is a representation of on in which points of are associated with vertices and simple arcs (homeomorphic images of ) are associated with edges in such a way that: the endpoints of the arc associated with an edge are the points associated with the end vertices of no arcs include points associated with other vertices, two arcs never intersect at a point which is interior to either of the arcs. Here a surface is a compact, connected -manifold.

Semantic similarity

Semantic similarity is a metric defined over a set of documents or terms, where the idea of distance between items is based on the likeness of their meaning or semantic content as opposed to lexicographical similarity. These are mathematical tools used to estimate the strength of the semantic relationship between units of language, concepts or instances, through a numerical description obtained according to the comparison of information supporting their meaning or describing their nature.

Dependency graph

In mathematics, computer science and digital electronics, a dependency graph is a directed graph representing dependencies of several objects towards each other. It is possible to derive an evaluation order or the absence of an evaluation order that respects the given dependencies from the dependency graph. Given a set of objects and a transitive relation with modeling a dependency "a depends on b" ("a needs b evaluated first"), the dependency graph is a graph with the transitive reduction of R.

Planar graph

In graph theory, a planar graph is a graph that can be embedded in the plane, i.e., it can be drawn on the plane in such a way that its edges intersect only at their endpoints. In other words, it can be drawn in such a way that no edges cross each other. Such a drawing is called a plane graph or planar embedding of the graph. A plane graph can be defined as a planar graph with a mapping from every node to a point on a plane, and from every edge to a plane curve on that plane, such that the extreme points of each curve are the points mapped from its end nodes, and all curves are disjoint except on their extreme points.

Related lectures (38)

Language Models: From Theory to Computation

Explores the mathematics of language models, covering architecture design, pre-training, and fine-tuning, emphasizing the importance of pre-training and fine-tuning for various tasks.

Binary Sentiment Classifier Training

Covers the training of a binary sentiment classifier using an RNN.

Non conceptual knowledge systems

Explores the impact of Deep learning on Digital Humanities, focusing on non conceptual knowledge systems and recent advancements in AI.

Generative Models: Self-Attention and Transformers

Covers generative models with a focus on self-attention and transformers, discussing sampling methods and empirical means.

Linear Transformations: Matrices and Kernels