Lecture

Variational Auto-Encoders and NVIB

Graph Chat

Chat with Graph Search

Ask any question about EPFL courses, lectures, exercises, research, news, etc. or try the example questions below.

DISCLAIMER: The Graph Chatbot is not programmed to provide explicit or categorical answers to your questions. Rather, it transforms your questions into API requests that are distributed across the various IT services officially administered by EPFL. Its purpose is solely to collect and recommend relevant references to content that you can explore to help you answer your questions.

Description

This lecture covers Variational Auto-Encoders (VAE) and Nonparametric Variational Information Bottleneck (NVIB). It explains the Bayesian approach to auto-encoders, the reparameterization trick, and the taxonomy of generative models. The lecture delves into the intractability of data likelihood in VAEs, the concept of variational inference, and the use of attention-based latent spaces in Transformers. It also discusses the importance of inducing compressed representations in text data and the benefits of NVIB in generating smooth representation spaces. The presentation concludes with insights on why Transformers are effective models for language processing.

Official source

Related concepts (32)

Bayesian inference

Bayesian inference (ˈbeɪziən or ˈbeɪʒən ) is a method of statistical inference in which Bayes' theorem is used to update the probability for a hypothesis as more evidence or information becomes available. Bayesian inference is an important technique in statistics, and especially in mathematical statistics. Bayesian updating is particularly important in the dynamic analysis of a sequence of data. Bayesian inference has found application in a wide range of activities, including science, engineering, philosophy, medicine, sport, and law.

Statistical inference

Statistical inference is the process of using data analysis to infer properties of an underlying distribution of probability. Inferential statistical analysis infers properties of a population, for example by testing hypotheses and deriving estimates. It is assumed that the observed data set is sampled from a larger population. Inferential statistics can be contrasted with descriptive statistics. Descriptive statistics is solely concerned with properties of the observed data, and it does not rest on the assumption that the data come from a larger population.

Inference

Inferences are steps in reasoning, moving from premises to logical consequences; etymologically, the word infer means to "carry forward". Inference is theoretically traditionally divided into deduction and induction, a distinction that in Europe dates at least to Aristotle (300s BCE). Deduction is inference deriving logical conclusions from premises known or assumed to be true, with the laws of valid inference being studied in logic. Induction is inference from particular evidence to a universal conclusion.

Transformer (machine learning model)

A transformer is a deep learning architecture that relies on the parallel multi-head attention mechanism. The modern transformer was proposed in the 2017 paper titled 'Attention Is All You Need' by Ashish Vaswani et al., Google Brain team. It is notable for requiring less training time than previous recurrent neural architectures, such as long short-term memory (LSTM), and its later variation has been prevalently adopted for training large language models on large (language) datasets, such as the Wikipedia corpus and Common Crawl, by virtue of the parallelized processing of input sequence.

Induced representation

In group theory, the induced representation is a representation of a group, G, which is constructed using a known representation of a subgroup H. Given a representation of H, the induced representation is, in a sense, the "most general" representation of G that extends the given one. Since it is often easier to find representations of the smaller group H than of G, the operation of forming induced representations is an important tool to construct new representations.

Related lectures (64)

Generative Models: Self-Attention and Transformers

Covers generative models with a focus on self-attention and transformers, discussing sampling methods and empirical means.

Document Analysis and Topic Modeling

Covers document analysis, topic modeling, and deep generative models, including autoencoders and GANs.

Deep Generative Models

Covers deep generative models, including LDA, autoencoders, GANs, and DCGANs.

Deep Generative Models: Variational Autoencoders & GANs

Explores Variational Autoencoders and Generative Adversarial Networks for deep generative modeling.