Lecture

Singular Value Decomposition: Ridge and LASSO

Graph Chat

Chat with Graph Search

Ask any question about EPFL courses, lectures, exercises, research, news, etc. or try the example questions below.

DISCLAIMER: The Graph Chatbot is not programmed to provide explicit or categorical answers to your questions. Rather, it transforms your questions into API requests that are distributed across the various IT services officially administered by EPFL. Its purpose is solely to collect and recommend relevant references to content that you can explore to help you answer your questions.

In course

EE-411: Fundamentals of inference and learning

This is an introductory course in the theory of statistics, inference, and machine learning, with an emphasis on theoretical understanding & practical exercises. The course will combine, and alternate, between mathematical theoretical foundations and prac ...

Description

This lecture covers the concept of Singular Value Decomposition (SVD) and its application in decomposing matrices, with remarks on Ridge and LASSO regression techniques, discussing the importance of rank, span, and singular values in solving regression problems.

Instructor

Florent Gérard Krzakala

Official source

Ontological neighbourhood

Mathematics

Algebra: Linear algebra

Related concepts (23)

Singular value decomposition

In linear algebra, the singular value decomposition (SVD) is a factorization of a real or complex matrix. It generalizes the eigendecomposition of a square normal matrix with an orthonormal eigenbasis to any matrix. It is related to the polar decomposition. Specifically, the singular value decomposition of an complex matrix M is a factorization of the form where U is an complex unitary matrix, is an rectangular diagonal matrix with non-negative real numbers on the diagonal, V is an complex unitary matrix, and is the conjugate transpose of V.

Linear regression

In statistics, linear regression is a linear approach for modelling the relationship between a scalar response and one or more explanatory variables (also known as dependent and independent variables). The case of one explanatory variable is called simple linear regression; for more than one, the process is called multiple linear regression. This term is distinct from multivariate linear regression, where multiple correlated dependent variables are predicted, rather than a single scalar variable.

Matrix decomposition

In the mathematical discipline of linear algebra, a matrix decomposition or matrix factorization is a factorization of a matrix into a product of matrices. There are many different matrix decompositions; each finds use among a particular class of problems. In numerical analysis, different decompositions are used to implement efficient matrix algorithms. For instance, when solving a system of linear equations , the matrix A can be decomposed via the LU decomposition.

Polar decomposition

In mathematics, the polar decomposition of a square real or complex matrix is a factorization of the form , where is a unitary matrix and is a positive semi-definite Hermitian matrix ( is an orthogonal matrix and is a positive semi-definite symmetric matrix in the real case), both square and of the same size. Intuitively, if a real matrix is interpreted as a linear transformation of -dimensional space , the polar decomposition separates it into a rotation or reflection of , and a scaling of the space along a set of orthogonal axes.

Singular value

In mathematics, in particular functional analysis, the singular values, or s-numbers of a compact operator acting between Hilbert spaces and , are the square roots of the (necessarily non-negative) eigenvalues of the self-adjoint operator (where denotes the adjoint of ). The singular values are non-negative real numbers, usually listed in decreasing order (σ1(T), σ2(T), ...). The largest singular value σ1(T) is equal to the operator norm of T (see Min-max theorem).

Related lectures (64)

Linear Regression: Statistical Inference and Regularization

Covers the probabilistic model for linear regression and the importance of regularization techniques.

Linear Regression: Least Squares Method

Explains the method of least squares in linear regression to find the best-fitting line to a set of data points.

Regularization in Machine Learning

Explores Ridge and Lasso Regression for regularization in machine learning models, emphasizing hyperparameter tuning and visualization of parameter coefficients.

Linear Regression: Absence or Presence of Covariates

Explores linear regression with and without covariates, covering models captured by independent distributions and tools like subspaces and orthogonal projections.

Probabilistic Linear Regression