Mediaspace scheduled maintenance: Aug 25, 2026 07:00 - 12:00 AM. During this time, videos will be temporarily unavailable. Check status updates.

Lecture

Comparing L1 and L0 + Greedy algorithms

In course

MATH-412: Statistical machine learning

A course on statistical machine learning for supervised and unsupervised learning ...

Description

This lecture delves into comparing L1 and L0 penalization in linear regression with orthogonal designs, emphasizing soft-thresholding vs hard-thresholding. It also explores greedy algorithms like Forward selection and Orthogonal Matching Pursuit, along with an empirical comparison of L0, L1, and L2 regularization methods.

Instructor

Yoav Zemel

Official source

Ontological neighbourhood

Statistics

Data analysis: Regression analysis, Cross-validation

Related concepts (32)

Regularized least squares

Regularized least squares (RLS) is a family of methods for solving the least-squares problem while using regularization to further constrain the resulting solution. RLS is used for two main reasons. The first comes up when the number of variables in the linear system exceeds the number of observations. In such settings, the ordinary least-squares problem is ill-posed and is therefore impossible to fit because the associated optimization problem has infinitely many solutions.

Linear regression

In statistics, linear regression is a linear approach for modelling the relationship between a scalar response and one or more explanatory variables (also known as dependent and independent variables). The case of one explanatory variable is called simple linear regression; for more than one, the process is called multiple linear regression. This term is distinct from multivariate linear regression, where multiple correlated dependent variables are predicted, rather than a single scalar variable.

Lasso (statistics)

In statistics and machine learning, lasso (least absolute shrinkage and selection operator; also Lasso or LASSO) is a regression analysis method that performs both variable selection and regularization in order to enhance the prediction accuracy and interpretability of the resulting statistical model. It was originally introduced in geophysics, and later by Robert Tibshirani, who coined the term. Lasso was originally formulated for linear regression models. This simple case reveals a substantial amount about the estimator.

Ridge regression

Ridge regression is a method of estimating the coefficients of multiple-regression models in scenarios where the independent variables are highly correlated. It has been used in many fields including econometrics, chemistry, and engineering. Also known as Tikhonov regularization, named for Andrey Tikhonov, it is a method of regularization of ill-posed problems. It is particularly useful to mitigate the problem of multicollinearity in linear regression, which commonly occurs in models with large numbers of parameters.

Regularization (mathematics)

In mathematics, statistics, finance, computer science, particularly in machine learning and inverse problems, regularization is a process that changes the result answer to be "simpler". It is often used to obtain results for ill-posed problems or to prevent overfitting. Although regularization procedures can be divided in many ways, the following delineation is particularly helpful: Explicit regularization is regularization whenever one explicitly adds a term to the optimization problem.

Related lectures (48)

Kernel Methods: Machine Learning

Explores kernel methods in machine learning, emphasizing their application in regression tasks and the prevention of overfitting.

Probabilistic Linear Regression

Explores probabilistic linear regression, covering joint and conditional probability, ridge regression, and overfitting mitigation.

Nonlinear ML Algorithms

Introduces nonlinear ML algorithms, covering nearest neighbor, k-NN, polynomial curve fitting, model complexity, overfitting, and regularization.

Model Selection Criteria: AIC, BIC, Cp

Explores model selection criteria like AIC, BIC, and Cp in statistics for data science.

Regularization: Tikhonov and Ridge