Lecture

Dimensionality Reduction: PCA and LDA

Graph Chat

Chat with Graph Search

Ask any question about EPFL courses, lectures, exercises, research, news, etc. or try the example questions below.

DISCLAIMER: The Graph Chatbot is not programmed to provide explicit or categorical answers to your questions. Rather, it transforms your questions into API requests that are distributed across the various IT services officially administered by EPFL. Its purpose is solely to collect and recommend relevant references to content that you can explore to help you answer your questions.

Description

This lecture covers the concepts of dimensionality reduction through techniques like Principal Component Analysis (PCA) and Fisher Linear Discriminant Analysis (LDA). It explains how PCA aims to retain important data signal while removing noise by maximizing variance, and how LDA focuses on clustering samples from the same class and separating different classes. The lecture also introduces Kernel PCA for nonlinear data, t-SNE for visualization, and discusses clustering methods like K-means. It delves into Gaussian Mixture Models (GMM) for density estimation, KDE for smooth distribution estimates, and Mean Shift for clustering by finding density maxima. The presentation concludes with a comparison of KDE and histograms for data representation.

Official source

Ontological neighbourhood

Information engineering

Machine learning: Unsupervised learning

Related concepts (32)

Cluster analysis

Cluster analysis or clustering is the task of grouping a set of objects in such a way that objects in the same group (called a cluster) are more similar (in some sense) to each other than to those in other groups (clusters). It is a main task of exploratory data analysis, and a common technique for statistical data analysis, used in many fields, including pattern recognition, , information retrieval, bioinformatics, data compression, computer graphics and machine learning.

Hierarchical clustering

In data mining and statistics, hierarchical clustering (also called hierarchical cluster analysis or HCA) is a method of cluster analysis that seeks to build a hierarchy of clusters. Strategies for hierarchical clustering generally fall into two categories: Agglomerative: This is a "bottom-up" approach: Each observation starts in its own cluster, and pairs of clusters are merged as one moves up the hierarchy. Divisive: This is a "top-down" approach: All observations start in one cluster, and splits are performed recursively as one moves down the hierarchy.

K-means clustering

k-means clustering is a method of vector quantization, originally from signal processing, that aims to partition n observations into k clusters in which each observation belongs to the cluster with the nearest mean (cluster centers or cluster centroid), serving as a prototype of the cluster. This results in a partitioning of the data space into Voronoi cells. k-means clustering minimizes within-cluster variances (squared Euclidean distances), but not regular Euclidean distances, which would be the more difficult Weber problem: the mean optimizes squared errors, whereas only the geometric median minimizes Euclidean distances.

Principal component analysis

Principal component analysis (PCA) is a popular technique for analyzing large datasets containing a high number of dimensions/features per observation, increasing the interpretability of data while preserving the maximum amount of information, and enabling the visualization of multidimensional data. Formally, PCA is a statistical technique for reducing the dimensionality of a dataset. This is accomplished by linearly transforming the data into a new coordinate system where (most of) the variation in the data can be described with fewer dimensions than the initial data.

Kernel density estimation

In statistics, kernel density estimation (KDE) is the application of kernel smoothing for probability density estimation, i.e., a non-parametric method to estimate the probability density function of a random variable based on kernels as weights. KDE answers a fundamental data smoothing problem where inferences about the population are made, based on a finite data sample. In some fields such as signal processing and econometrics it is also termed the Parzen–Rosenblatt window method, after Emanuel Parzen and Murray Rosenblatt, who are usually credited with independently creating it in its current form.

Related lectures (64)

Clustering & Density Estimation

Covers dimensionality reduction, PCA, clustering techniques, and density estimation methods.

Dimensionality Reduction: PCA & t-SNE

Explores PCA and t-SNE for reducing dimensions and visualizing high-dimensional data effectively.

Clustering & Density Estimation

Covers clustering, PCA, LDA, K-means, GMM, KDE, and Mean Shift algorithms for density estimation and clustering.

Unsupervised Learning: Dimensionality Reduction and Clustering

Covers unsupervised learning, focusing on dimensionality reduction and clustering, explaining how it helps find patterns in data without labels.

Clustering & Density Estimation

Covers dimensionality reduction, clustering, and density estimation techniques, including PCA, K-means, GMM, and Mean Shift.

Related concepts (32)

Cluster analysis

Hierarchical clustering

K-means clustering

Principal component analysis

Kernel density estimation