Mediaspace scheduled maintenance: Aug 25, 2026 07:00 - 12:00 AM. During this time, videos will be temporarily unavailable. Check status updates.

Lecture

Other regularizations + the Lasso

In course

MATH-412: Statistical machine learning

A course on statistical machine learning for supervised and unsupervised learning ...

Description

This lecture covers a diverse set of regularization approaches, including the L0 quasi-norm and the Lasso method. It discusses best subset selection, the trade-off between fitting and the number of variables, and the NP-hardness of the problem. The Lasso, or Least Absolute Shrinkage and Selection Operator, is introduced as a convex but non-differentiable optimization problem with efficient algorithms for solution. The lecture also explores the extension to quasi-norms and the difference between constrained and regularized Lasso regression problems.

Instructor

Yoav Zemel

Official source

Related concepts (32)

Regularization (mathematics)

In mathematics, statistics, finance, computer science, particularly in machine learning and inverse problems, regularization is a process that changes the result answer to be "simpler". It is often used to obtain results for ill-posed problems or to prevent overfitting. Although regularization procedures can be divided in many ways, the following delineation is particularly helpful: Explicit regularization is regularization whenever one explicitly adds a term to the optimization problem.

Feature selection

Feature selection is the process of selecting a subset of relevant features (variables, predictors) for use in model construction. Stylometry and DNA microarray analysis are two cases where feature selection is used. It should be distinguished from feature extraction. Feature selection techniques are used for several reasons: simplification of models to make them easier to interpret by researchers/users, shorter training times, to avoid the curse of dimensionality, improve data's compatibility with a learning model class, encode inherent symmetries present in the input space.

Regularized least squares

Regularized least squares (RLS) is a family of methods for solving the least-squares problem while using regularization to further constrain the resulting solution. RLS is used for two main reasons. The first comes up when the number of variables in the linear system exceeds the number of observations. In such settings, the ordinary least-squares problem is ill-posed and is therefore impossible to fit because the associated optimization problem has infinitely many solutions.

Ridge regression

Ridge regression is a method of estimating the coefficients of multiple-regression models in scenarios where the independent variables are highly correlated. It has been used in many fields including econometrics, chemistry, and engineering. Also known as Tikhonov regularization, named for Andrey Tikhonov, it is a method of regularization of ill-posed problems. It is particularly useful to mitigate the problem of multicollinearity in linear regression, which commonly occurs in models with large numbers of parameters.

Convex optimization

Convex optimization is a subfield of mathematical optimization that studies the problem of minimizing convex functions over convex sets (or, equivalently, maximizing concave functions over convex sets). Many classes of convex optimization problems admit polynomial-time algorithms, whereas mathematical optimization is in general NP-hard.

Related lectures (48)

Comparing L1 and L0 + Greedy algorithms

Compares L1 and L0 penalization in linear regression with orthogonal designs using greedy algorithms and empirical comparisons.

Linear Regression: Statistical Inference and Regularization

Covers the probabilistic model for linear regression and the importance of regularization techniques.

Solving Parity Games in Practice

Explores practical aspects of solving parity games, including winning strategies, algorithms, complexity, determinism, and heuristic approaches.

L1 Regularization: Sparse Solutions and Dimensionality Reduction

Delves into L1 regularization, sparse solutions, and dimensionality reduction in the context of machine learning.

LASSO Regression: Sparse Signal Induction