Mediaspace scheduled maintenance: Aug 25, 2026 07:00 - 12:00 AM. During this time, videos will be temporarily unavailable. Check status updates.

Lecture

PCA and Kernel PCA

In course

EE-411: Fundamentals of inference and learning

This is an introductory course in the theory of statistics, inference, and machine learning, with an emphasis on theoretical understanding & practical exercises. The course will combine, and alternate, between mathematical theoretical foundations and prac ...

Description

This lecture covers Principal Component Analysis (PCA) and Kernel PCA, explaining how PCA is used to eliminate dimensions by finding the principal components with the most variation. It also delves into the comparison between PCA and Kernel PCA, showcasing how Kernel PCA projects data into a higher-dimensional space to make it linearly separable.

Instructor

Florent Gérard Krzakala

Official source

Related concepts (32)

Kernel principal component analysis

In the field of multivariate statistics, kernel principal component analysis (kernel PCA) is an extension of principal component analysis (PCA) using techniques of kernel methods. Using a kernel, the originally linear operations of PCA are performed in a reproducing kernel Hilbert space. Recall that conventional PCA operates on zero-centered data; that is, where is one of the multivariate observations.

Dimensionality reduction

Dimensionality reduction, or dimension reduction, is the transformation of data from a high-dimensional space into a low-dimensional space so that the low-dimensional representation retains some meaningful properties of the original data, ideally close to its intrinsic dimension. Working in high-dimensional spaces can be undesirable for many reasons; raw data are often sparse as a consequence of the curse of dimensionality, and analyzing the data is usually computationally intractable (hard to control or deal with).

Principal component analysis

Principal component analysis (PCA) is a popular technique for analyzing large datasets containing a high number of dimensions/features per observation, increasing the interpretability of data while preserving the maximum amount of information, and enabling the visualization of multidimensional data. Formally, PCA is a statistical technique for reducing the dimensionality of a dataset. This is accomplished by linearly transforming the data into a new coordinate system where (most of) the variation in the data can be described with fewer dimensions than the initial data.

Nonlinear dimensionality reduction

Nonlinear dimensionality reduction, also known as manifold learning, refers to various related techniques that aim to project high-dimensional data onto lower-dimensional latent manifolds, with the goal of either visualizing the data in the low-dimensional space, or learning the mapping (either from the high-dimensional space to the low-dimensional embedding or vice versa) itself. The techniques described below can be understood as generalizations of linear decomposition methods used for dimensionality reduction, such as singular value decomposition and principal component analysis.

Linear separability

In Euclidean geometry, linear separability is a property of two sets of points. This is most easily visualized in two dimensions (the Euclidean plane) by thinking of one set of points as being colored blue and the other set of points as being colored red. These two sets are linearly separable if there exists at least one line in the plane with all of the blue points on one side of the line and all the red points on the other side. This idea immediately generalizes to higher-dimensional Euclidean spaces if the line is replaced by a hyperplane.

Related lectures (32)

Kernel PCA: Nonlinear Dimensionality Reduction

Explores Kernel Principal Component Analysis, a nonlinear method using kernels for linear problem solving and dimensionality reduction.

Principal Component Analysis: Dimensionality Reduction

Covers Principal Component Analysis for dimensionality reduction, exploring its applications, limitations, and importance of choosing the right components.

Linear Dimensionality Reduction: PCA and LDA

Explores PCA and LDA for linear dimensionality reduction in data, emphasizing clustering and class separation techniques.

Dimensionality Reduction: PCA & Autoencoders

Explores PCA, Autoencoders, and their applications in dimensionality reduction and data generation.

Unsupervised Learning: Dimensionality Reduction