Lecture

Geometry of the Lasso

Graph Chat

Chat with Graph Search

Ask any question about EPFL courses, lectures, exercises, research, news, etc. or try the example questions below.

DISCLAIMER: The Graph Chatbot is not programmed to provide explicit or categorical answers to your questions. Rather, it transforms your questions into API requests that are distributed across the various IT services officially administered by EPFL. Its purpose is solely to collect and recommend relevant references to content that you can explore to help you answer your questions.

In course

MATH-412: Statistical machine learning

A course on statistical machine learning for supervised and unsupervised learning ...

Description

This lecture explains the geometric intuition behind why Lasso solutions are sparse, focusing on a 2D example where the L1 norm constraint creates ellipsoidal contours, leading to a solution with a low-dimensional face. The instructor also discusses the behavior of coefficients as the regularization parameter changes in Lasso regression.

Instructor

Yoav Zemel

Official source

Related concepts (32)

Linear regression

In statistics, linear regression is a linear approach for modelling the relationship between a scalar response and one or more explanatory variables (also known as dependent and independent variables). The case of one explanatory variable is called simple linear regression; for more than one, the process is called multiple linear regression. This term is distinct from multivariate linear regression, where multiple correlated dependent variables are predicted, rather than a single scalar variable.

Lasso (statistics)

In statistics and machine learning, lasso (least absolute shrinkage and selection operator; also Lasso or LASSO) is a regression analysis method that performs both variable selection and regularization in order to enhance the prediction accuracy and interpretability of the resulting statistical model. It was originally introduced in geophysics, and later by Robert Tibshirani, who coined the term. Lasso was originally formulated for linear regression models. This simple case reveals a substantial amount about the estimator.

4-polytope

In geometry, a 4-polytope (sometimes also called a polychoron, polycell, or polyhedroid) is a four-dimensional polytope. It is a connected and closed figure, composed of lower-dimensional polytopal elements: vertices, edges, faces (polygons), and cells (polyhedra). Each face is shared by exactly two cells. The 4-polytopes were discovered by the Swiss mathematician Ludwig Schläfli before 1853. The two-dimensional analogue of a 4-polytope is a polygon, and the three-dimensional analogue is a polyhedron.

Regularized least squares

Regularized least squares (RLS) is a family of methods for solving the least-squares problem while using regularization to further constrain the resulting solution. RLS is used for two main reasons. The first comes up when the number of variables in the linear system exceeds the number of observations. In such settings, the ordinary least-squares problem is ill-posed and is therefore impossible to fit because the associated optimization problem has infinitely many solutions.

Regularization (mathematics)

In mathematics, statistics, finance, computer science, particularly in machine learning and inverse problems, regularization is a process that changes the result answer to be "simpler". It is often used to obtain results for ill-posed problems or to prevent overfitting. Although regularization procedures can be divided in many ways, the following delineation is particularly helpful: Explicit regularization is regularization whenever one explicitly adds a term to the optimization problem.

Related lectures (64)

Linear Regression: Statistical Inference and Regularization

Covers the probabilistic model for linear regression and the importance of regularization techniques.

Linear and Ridge Regression

Covers linear and ridge regression, overfitting, hyperparameters, and test sets.

Comparing L1 and L0 + Greedy algorithms

Compares L1 and L0 penalization in linear regression with orthogonal designs using greedy algorithms and empirical comparisons.

Support Vector Regression: Principles and Optimization

Covers Support Vector Regression principles, optimization, and hyperparameters' influence on the fit.

Probabilistic Models for Linear Regression