Lecture

Dimensionality Reduction: Curse of Dimensionality

Graph Chat

Chat with Graph Search

Ask any question about EPFL courses, lectures, exercises, research, news, etc. or try the example questions below.

DISCLAIMER: The Graph Chatbot is not programmed to provide explicit or categorical answers to your questions. Rather, it transforms your questions into API requests that are distributed across the various IT services officially administered by EPFL. Its purpose is solely to collect and recommend relevant references to content that you can explore to help you answer your questions.

Description

This lecture covers the curse of dimensionality, which implies that in high dimensions, data points become increasingly isolated, requiring more training data. Methods for variable selection, such as filtering based on correlation or mutual information, are discussed. The coefficient of determination is introduced as a measure of how well predicted values correlate with actual values. Limitations of filtering methods are highlighted, using the example of explaining the output of an exclusive OR (XOR) operation. Overall, the lecture emphasizes the challenges and strategies for reducing dimensionality in machine learning.

Official source

Related concepts (32)

Pearson correlation coefficient

In statistics, the Pearson correlation coefficient (PCC) is a correlation coefficient that measures linear correlation between two sets of data. It is the ratio between the covariance of two variables and the product of their standard deviations; thus, it is essentially a normalized measurement of the covariance, such that the result always has a value between −1 and 1. As with covariance itself, the measure can only reflect a linear correlation of variables, and ignores many other types of relationships or correlations.

Dimensionality reduction

Dimensionality reduction, or dimension reduction, is the transformation of data from a high-dimensional space into a low-dimensional space so that the low-dimensional representation retains some meaningful properties of the original data, ideally close to its intrinsic dimension. Working in high-dimensional spaces can be undesirable for many reasons; raw data are often sparse as a consequence of the curse of dimensionality, and analyzing the data is usually computationally intractable (hard to control or deal with).

Correlation coefficient

A correlation coefficient is a numerical measure of some type of correlation, meaning a statistical relationship between two variables. The variables may be two columns of a given data set of observations, often called a sample, or two components of a multivariate random variable with a known distribution. Several types of correlation coefficient exist, each with their own definition and own range of usability and characteristics. They all assume values in the range from −1 to +1, where ±1 indicates the strongest possible agreement and 0 the strongest possible disagreement.

Curse of dimensionality

The curse of dimensionality refers to various phenomena that arise when analyzing and organizing data in high-dimensional spaces that do not occur in low-dimensional settings such as the three-dimensional physical space of everyday experience. The expression was coined by Richard E. Bellman when considering problems in dynamic programming. Dimensionally cursed phenomena occur in domains such as numerical analysis, sampling, combinatorics, machine learning, data mining and databases.

Correlation

In statistics, correlation or dependence is any statistical relationship, whether causal or not, between two random variables or bivariate data. Although in the broadest sense, "correlation" may indicate any type of association, in statistics it usually refers to the degree to which a pair of variables are linearly related. Familiar examples of dependent phenomena include the correlation between the height of parents and their offspring, and the correlation between the price of a good and the quantity the consumers are willing to purchase, as it is depicted in the so-called demand curve.

Related lectures (64)

Properties of Time-Frequency Domain Signals

Covers the main properties of time-frequency domain signals and their limitations.

Variable Selection Methods: Filtering and Correlation

Explores variable selection through filtering and correlation methods in machine learning, emphasizing relevance quantification and relationship measurement with the label.

Stochastic Processes: Basics & Stationarity

Covers signal generation, statistical relations, stationarity, white noise, and orthogonality in stochastic processes.

Quantifying Statistical Dependence: Covariance and Correlation

Explores covariance, correlation, and mutual information in quantifying statistical dependence between random variables.

Describing Data: Statistics & Uncertainty