Lecture

Choosing a Step Size

Graph Chat

Chat with Graph Search

Ask any question about EPFL courses, lectures, exercises, research, news, etc. or try the example questions below.

DISCLAIMER: The Graph Chatbot is not programmed to provide explicit or categorical answers to your questions. Rather, it transforms your questions into API requests that are distributed across the various IT services officially administered by EPFL. Its purpose is solely to collect and recommend relevant references to content that you can explore to help you answer your questions.

In course

MATH-512: Optimization on manifolds

We develop, analyze and implement numerical algorithms to solve optimization problems of the form min f(x) where x is a point on a smooth manifold. To this end, we first study differential and Riemannian geometry (with a focus dictated by computational con ...

Description

This lecture covers the concept of choosing a step size in optimization on manifolds, focusing on making a Lipschitz-like assumption about the objective function and the retraction operator. It discusses the backtracking line-search method and the conditions for sufficient decrease, emphasizing the challenge of determining the Lipschitz constant. The instructor presents the Armijo backtracking for gradient descent and provides an algorithm for selecting the step size. The lecture concludes with a theorem relating the assumptions made to ensure sufficient decrease in the optimization process.

In MOOC

Introduction to optimization on smooth manifolds: first order methods

Optimization on manifolds is the result of smooth geometry and optimization merging into one elegant modern framework. We start the course at "What is a manifold?", and give the students a firm understanding of submanifolds embedded in real space. This covers numerous applications in engineering and the sciences. All definitions and theorems are motivated to build time-tested optimization algorithms. The math is precise, to promote understanding and enable computation. We build our way up to Riemannian g

Instructor

Nicolas Boumal

Teaching & PhD PhD Students Andreea-Alexandra Musat Past EPFL PhD Students Christopher Criscitiello, Quentin Rebjock Courses Continuous optimization MATH-329 This course introduces students to continuous, nonlinear optimization. We study the theory of optimization with continuous variables (with full proofs), and we analyze and implement important algorithms to solve constrained and unconstrained problems. Linear Algebra MATH-111(e) The purpose of the course is to introduce

Official source

Related concepts (32)

Gradient descent

In mathematics, gradient descent (also often called steepest descent) is a iterative optimization algorithm for finding a local minimum of a differentiable function. The idea is to take repeated steps in the opposite direction of the gradient (or approximate gradient) of the function at the current point, because this is the direction of steepest descent. Conversely, stepping in the direction of the gradient will lead to a local maximum of that function; the procedure is then known as gradient ascent.

Manifold

In mathematics, a manifold is a topological space that locally resembles Euclidean space near each point. More precisely, an -dimensional manifold, or -manifold for short, is a topological space with the property that each point has a neighborhood that is homeomorphic to an open subset of -dimensional Euclidean space. One-dimensional manifolds include lines and circles, but not lemniscates. Two-dimensional manifolds are also called surfaces. Examples include the plane, the sphere, and the torus, and also the Klein bottle and real projective plane.

Multi-objective optimization

Multi-objective optimization or Pareto optimization (also known as multi-objective programming, vector optimization, multicriteria optimization, or multiattribute optimization) is an area of multiple-criteria decision making that is concerned with mathematical optimization problems involving more than one objective function to be optimized simultaneously. Multi-objective is a type of vector optimization that has been applied in many fields of science, including engineering, economics and logistics where optimal decisions need to be taken in the presence of trade-offs between two or more conflicting objectives.

Differentiable manifold

In mathematics, a differentiable manifold (also differential manifold) is a type of manifold that is locally similar enough to a vector space to allow one to apply calculus. Any manifold can be described by a collection of charts (atlas). One may then apply ideas from calculus while working within the individual charts, since each chart lies within a vector space to which the usual rules of calculus apply. If the charts are suitably compatible (namely, the transition from one chart to another is differentiable), then computations done in one chart are valid in any other differentiable chart.

Mathematical optimization

Mathematical optimization (alternatively spelled optimisation) or mathematical programming is the selection of a best element, with regard to some criterion, from some set of available alternatives. It is generally divided into two subfields: discrete optimization and continuous optimization. Optimization problems arise in all quantitative disciplines from computer science and engineering to operations research and economics, and the development of solution methods has been of interest in mathematics for centuries.

Related lectures (64)

Momentum Methods and Nonlinear Conjugate Gradients

The session explains how momentum accelerates gradient descent, presents the theory of nonlinear conjugate gradients on manifolds, compares several beta‑rule updates, discusses line‑search and preconditioner choices, and shows how to employ the Manopt library for Riemannian optimization without explicit Hessians.

Transporters: a proxy for parallel transport

Explores transporters as a practical alternative to parallel transport, discussing minimal requirements, examples with matrices, pragmatic choices, and optimization algorithms.

Computing the Newton Step: GD as a Matrix-Free Way

Explores matrix-based and matrix-free approaches for computing the Newton step in optimization on manifolds.

Comparing Tangent Vectors: Three Reasons Why

Explores the importance of comparing tangent vectors at different points using algorithms and finite differences.

Truncated Conjugate Gradients for Trust-Region Subproblem