Mediaspace scheduled maintenance: Aug 25, 2026 07:00 - 12:00 AM. During this time, videos will be temporarily unavailable. Check status updates.

Lecture

Optimization in Machine Learning: Gradient Descent

Description

This lecture introduces optimization in machine learning, focusing on gradient descent. It covers the analytical solution for linear regression, the lack of analytical solutions for logistic regression, and the use of specialized optimizers. The concept of convexity, stochastic gradient descent, and early stopping are explained. Practical considerations such as choosing a good learning rate and preprocessing data are discussed. The lecture also delves into solving the XOR problem without feature engineering using logistic regression and neural networks.

Official source

Ontological neighbourhood

Information engineering

Machine learning: Artificial neural networks

Statistics

Data analysis: Regression analysis

Related concepts (32)

Linear regression

In statistics, linear regression is a linear approach for modelling the relationship between a scalar response and one or more explanatory variables (also known as dependent and independent variables). The case of one explanatory variable is called simple linear regression; for more than one, the process is called multiple linear regression. This term is distinct from multivariate linear regression, where multiple correlated dependent variables are predicted, rather than a single scalar variable.

Stochastic gradient descent

Stochastic gradient descent (often abbreviated SGD) is an iterative method for optimizing an objective function with suitable smoothness properties (e.g. differentiable or subdifferentiable). It can be regarded as a stochastic approximation of gradient descent optimization, since it replaces the actual gradient (calculated from the entire data set) by an estimate thereof (calculated from a randomly selected subset of the data).

Gradient descent

In mathematics, gradient descent (also often called steepest descent) is a iterative optimization algorithm for finding a local minimum of a differentiable function. The idea is to take repeated steps in the opposite direction of the gradient (or approximate gradient) of the function at the current point, because this is the direction of steepest descent. Conversely, stepping in the direction of the gradient will lead to a local maximum of that function; the procedure is then known as gradient ascent.

Logistic regression

In statistics, the logistic model (or logit model) is a statistical model that models the probability of an event taking place by having the log-odds for the event be a linear combination of one or more independent variables. In regression analysis, logistic regression (or logit regression) is estimating the parameters of a logistic model (the coefficients in the linear combination).

Nonlinear regression

In statistics, nonlinear regression is a form of regression analysis in which observational data are modeled by a function which is a nonlinear combination of the model parameters and depends on one or more independent variables. The data are fitted by a method of successive approximations. In nonlinear regression, a statistical model of the form, relates a vector of independent variables, , and its associated observed dependent variables, . The function is nonlinear in the components of the vector of parameters , but otherwise arbitrary.

Related lectures (51)

Neural Networks: Multilayer Learning

Covers the fundamentals of multilayer neural networks and deep learning, including back-propagation and network architectures like LeNet, AlexNet, and VGG-16.

Multilayer Neural Networks: Deep Learning

Covers the fundamentals of multilayer neural networks and deep learning.

Gradient Descent and Linear Regression

Covers stochastic gradient descent, linear regression, regularization, supervised learning, and the iterative nature of gradient descent.

Linear and Logistic Regression

Covers linear and logistic regression, including underfitting, overfitting, and performance metrics.

Generalized Linear Regression