Mediaspace scheduled maintenance: Aug 25, 2026 07:00 - 12:00 AM. During this time, videos will be temporarily unavailable. Check status updates.

Lecture

Covariance Cleaning and Estimators

In course

FIN-525: Financial big data

The course introduces modern methods to acquire, clean, and analyze large quantities of financial data efficiently. The second part expands on how to apply these techniques and robust statistics to financial analysis, in particular to intraday data and inv ...

Description

This lecture covers the rationale behind covariance matrix cleaning, emphasizing the drawbacks of using the sample covariance matrix for portfolio optimization. It delves into rotationally invariant estimators, hierarchical clustering, and bootstraps. The instructor discusses the empirical spectrum of correlation matrices, eigenvalue clipping, and a formal approach to correlation/covariance cleaning. The lecture explores the concept of a good covariance estimator, focusing on minimizing the distance with the true matrix and building an optimal estimator for portfolio optimization. Additionally, it introduces the linear shrinkage method and the concept of optimal rotationally invariant estimators.

Instructor

Damien Challet

Official source

Ontological neighbourhood

Statistics

Statistical inference: Mathematical statistics

Related concepts (32)

Covariance matrix

In probability theory and statistics, a covariance matrix (also known as auto-covariance matrix, dispersion matrix, variance matrix, or variance–covariance matrix) is a square matrix giving the covariance between each pair of elements of a given random vector. Any covariance matrix is symmetric and positive semi-definite and its main diagonal contains variances (i.e., the covariance of each element with itself). Intuitively, the covariance matrix generalizes the notion of variance to multiple dimensions.

Bias of an estimator

In statistics, the bias of an estimator (or bias function) is the difference between this estimator's expected value and the true value of the parameter being estimated. An estimator or decision rule with zero bias is called unbiased. In statistics, "bias" is an property of an estimator. Bias is a distinct concept from consistency: consistent estimators converge in probability to the true value of the parameter, but may be biased or unbiased; see bias versus consistency for more.

Sample mean and covariance

The sample mean (sample average) or empirical mean (empirical average), and the sample covariance or empirical covariance are statistics computed from a sample of data on one or more random variables. The sample mean is the average value (or mean value) of a sample of numbers taken from a larger population of numbers, where "population" indicates not number of people but the entirety of relevant data, whether collected or not. A sample of 40 companies' sales from the Fortune 500 might be used for convenience instead of looking at the population, all 500 companies' sales.

Invariant estimator

In statistics, the concept of being an invariant estimator is a criterion that can be used to compare the properties of different estimators for the same quantity. It is a way of formalising the idea that an estimator should have certain intuitively appealing qualities. Strictly speaking, "invariant" would mean that the estimates themselves are unchanged when both the measurements and the parameters are transformed in a compatible way, but the meaning has been extended to allow the estimates to change in appropriate ways with such transformations.

Estimator

In statistics, an estimator is a rule for calculating an estimate of a given quantity based on observed data: thus the rule (the estimator), the quantity of interest (the estimand) and its result (the estimate) are distinguished. For example, the sample mean is a commonly used estimator of the population mean. There are point and interval estimators. The point estimators yield single-valued results. This is in contrast to an interval estimator, where the result would be a range of plausible values.

Related lectures (32)

Principal Components: Properties & Applications

Explores principal components, covariance, correlation, choice, and applications in data analysis.

Dependence in Random Vectors

Explores dependence in random vectors, covering joint density, conditional independence, covariance, and moment generating functions.

Shrinkage Estimation of Large Covariance Matrices

Explores shrinkage estimation of high-dimensional covariance matrices, comparing linear and nonlinear approaches for improved accuracy.

Market Response Functions

Explores market response functions, flash crashes, correlation estimation, and noise filtering in finance.

Multivariate Statistics: Wishart and Hotelling T²