Lecture

Dimensionality Reduction: PCA & LDA

Graph Chat

Chat with Graph Search

Ask any question about EPFL courses, lectures, exercises, research, news, etc. or try the example questions below.

DISCLAIMER: The Graph Chatbot is not programmed to provide explicit or categorical answers to your questions. Rather, it transforms your questions into API requests that are distributed across the various IT services officially administered by EPFL. Its purpose is solely to collect and recommend relevant references to content that you can explore to help you answer your questions.

Description

This lecture covers the concepts of Principal Component Analysis (PCA) and Linear Discriminant Analysis (LDA) for dimensionality reduction. Starting with the intuition behind PCA, it explains how PCA aims to retain important signal while removing noise. The lecture then delves into the mathematical details of PCA, including variance maximization and eigenvector problems. Moving on to LDA, it discusses the goal of clustering samples from the same class and separating different classes. The lecture concludes with a comparison of Kernel PCA and traditional PCA, highlighting the benefits of Kernel PCA for nonlinear data. Additionally, it introduces t-distributed stochastic neighbor embedding (t-SNE) as a popular technique for visualizing high-dimensional data.

Official source

Related concepts (32)

Dimensionality reduction

Dimensionality reduction, or dimension reduction, is the transformation of data from a high-dimensional space into a low-dimensional space so that the low-dimensional representation retains some meaningful properties of the original data, ideally close to its intrinsic dimension. Working in high-dimensional spaces can be undesirable for many reasons; raw data are often sparse as a consequence of the curse of dimensionality, and analyzing the data is usually computationally intractable (hard to control or deal with).

Principal component analysis

Principal component analysis (PCA) is a popular technique for analyzing large datasets containing a high number of dimensions/features per observation, increasing the interpretability of data while preserving the maximum amount of information, and enabling the visualization of multidimensional data. Formally, PCA is a statistical technique for reducing the dimensionality of a dataset. This is accomplished by linearly transforming the data into a new coordinate system where (most of) the variation in the data can be described with fewer dimensions than the initial data.

Nonlinear dimensionality reduction

Nonlinear dimensionality reduction, also known as manifold learning, refers to various related techniques that aim to project high-dimensional data onto lower-dimensional latent manifolds, with the goal of either visualizing the data in the low-dimensional space, or learning the mapping (either from the high-dimensional space to the low-dimensional embedding or vice versa) itself. The techniques described below can be understood as generalizations of linear decomposition methods used for dimensionality reduction, such as singular value decomposition and principal component analysis.

Kernel principal component analysis

In the field of multivariate statistics, kernel principal component analysis (kernel PCA) is an extension of principal component analysis (PCA) using techniques of kernel methods. Using a kernel, the originally linear operations of PCA are performed in a reproducing kernel Hilbert space. Recall that conventional PCA operates on zero-centered data; that is, where is one of the multivariate observations.

Curse of dimensionality

The curse of dimensionality refers to various phenomena that arise when analyzing and organizing data in high-dimensional spaces that do not occur in low-dimensional settings such as the three-dimensional physical space of everyday experience. The expression was coined by Richard E. Bellman when considering problems in dynamic programming. Dimensionally cursed phenomena occur in domains such as numerical analysis, sampling, combinatorics, machine learning, data mining and databases.

Related lectures (64)

Neural Networks Recap: Activation Functions

Covers the basics of neural networks, activation functions, training, image processing, CNNs, regularization, and dimensionality reduction methods.

Dimensionality Reduction: PCA & t-SNE

Explores PCA and t-SNE for reducing dimensions and visualizing high-dimensional data effectively.

Clustering & Density Estimation

Covers clustering, PCA, LDA, K-means, GMM, KDE, and Mean Shift algorithms for density estimation and clustering.

Clustering: K-means & LDA

Covers clustering using K-means and LDA, PCA, K-means properties, Fisher LDA, and spectral clustering.

Clustering: Unsupervised Learning