Mediaspace scheduled maintenance: Aug 25, 2026 07:00 - 12:00 AM. During this time, videos will be temporarily unavailable. Check status updates.

Lecture

Descent methods and line search: Inexact line search

Description

This lecture covers the concept of descent methods and line search, focusing on the idea of updating a point based on a step direction to minimize a function. Examples in one dimension are provided to illustrate the process, along with the intuition that descent direction assumes small steps. The lecture also discusses the importance of maintaining small steps for Taylor's theorem to apply and how even ascent directions can decrease the function with long steps.

In MOOC

Optimization: principles and algorithms - Unconstrained nonlinear optimization

Official source

Related concepts (15)

Dimension

In physics and mathematics, the dimension of a mathematical space (or object) is informally defined as the minimum number of coordinates needed to specify any point within it. Thus, a line has a dimension of one (1D) because only one coordinate is needed to specify a point on it - for example, the point at 5 on a number line. A surface, such as the boundary of a cylinder or sphere, has a dimension of two (2D) because two coordinates are needed to specify a point on it - for example, both a latitude and longitude are required to locate a point on the surface of a sphere.

Minkowski–Bouligand dimension

In fractal geometry, the Minkowski–Bouligand dimension, also known as Minkowski dimension or box-counting dimension, is a way of determining the fractal dimension of a set in a Euclidean space , or more generally in a metric space . It is named after the Polish mathematician Hermann Minkowski and the French mathematician Georges Bouligand. To calculate this dimension for a fractal , imagine this fractal lying on an evenly spaced grid and count how many boxes are required to cover the set.

Gradient descent

In mathematics, gradient descent (also often called steepest descent) is a iterative optimization algorithm for finding a local minimum of a differentiable function. The idea is to take repeated steps in the opposite direction of the gradient (or approximate gradient) of the function at the current point, because this is the direction of steepest descent. Conversely, stepping in the direction of the gradient will lead to a local maximum of that function; the procedure is then known as gradient ascent.

Stochastic gradient descent

Stochastic gradient descent (often abbreviated SGD) is an iterative method for optimizing an objective function with suitable smoothness properties (e.g. differentiable or subdifferentiable). It can be regarded as a stochastic approximation of gradient descent optimization, since it replaces the actual gradient (calculated from the entire data set) by an estimate thereof (calculated from a randomly selected subset of the data).

Lebesgue covering dimension

In mathematics, the Lebesgue covering dimension or topological dimension of a topological space is one of several different ways of defining the dimension of the space in a topologically invariant way. For ordinary Euclidean spaces, the Lebesgue covering dimension is just the ordinary Euclidean dimension: zero for points, one for lines, two for planes, and so on. However, not all topological spaces have this kind of "obvious" dimension, and so a precise definition is needed in such cases.

Related lectures (33)

Optimization: Gradient Descent and Subgradients

Explores optimization methods like gradient descent and subgradients for training machine learning models, including advanced techniques like Adam optimization.

Optimization Techniques: Convexity in Machine Learning

Covers optimization techniques in machine learning, focusing on convexity and its implications for efficient problem-solving.

Optimization without Constraints: Gradient Method

Covers optimization without constraints using the gradient method to find the function's minimum.

Optimization Basics

Introduces optimization basics, covering logistic regression, derivatives, convex functions, gradient descent, and second-order methods.

Optimization methods