Lecture

Optimization Techniques: Convexity in Machine Learning

Graph Chat

Chat with Graph Search

Ask any question about EPFL courses, lectures, exercises, research, news, etc. or try the example questions below.

DISCLAIMER: The Graph Chatbot is not programmed to provide explicit or categorical answers to your questions. Rather, it transforms your questions into API requests that are distributed across the various IT services officially administered by EPFL. Its purpose is solely to collect and recommend relevant references to content that you can explore to help you answer your questions.

In course

CS-439: Optimization for machine learning

This course teaches an overview of modern optimization methods, for applications in machine learning and data science. In particular, scalability of algorithms to large datasets will be discussed in theory and in implementation. ...

Description

This lecture introduces the fundamental concepts of optimization in machine learning, focusing on convexity. It begins with an overview of optimization problems, emphasizing the importance of convex functions and sets. The instructor explains the general optimization problem and the significance of mathematical modeling in defining and measuring machine learning models. Various optimization algorithms are discussed, including gradient descent and coordinate descent, along with their historical development. The lecture highlights the properties of convex functions, such as the Cauchy-Schwarz inequality and the definition of convex sets. The instructor elaborates on the motivation behind convex optimization, explaining that local minima in convex functions are also global minima, which is a crucial property for efficient problem-solving. The lecture concludes with a discussion on the convergence theory of optimization algorithms, reinforcing the importance of convexity in deriving general analysis and guarantees for convergence in optimization problems.

Official source

Related concepts (32)

Convex optimization

Convex optimization is a subfield of mathematical optimization that studies the problem of minimizing convex functions over convex sets (or, equivalently, maximizing concave functions over convex sets). Many classes of convex optimization problems admit polynomial-time algorithms, whereas mathematical optimization is in general NP-hard.

Convex set

In geometry, a subset of a Euclidean space, or more generally an affine space over the reals, is convex if, given any two points in the subset, the subset contains the whole line segment that joins them. Equivalently, a convex set or a convex region is a subset that intersects every line into a single line segment (possibly empty). For example, a solid cube is a convex set, but anything that is hollow or has an indent, for example, a crescent shape, is not convex. The boundary of a convex set is always a convex curve.

Convex function

In mathematics, a real-valued function is called convex if the line segment between any two distinct points on the graph of the function lies above the graph between the two points. Equivalently, a function is convex if its epigraph (the set of points on or above the graph of the function) is a convex set. A twice-differentiable function of a single variable is convex if and only if its second derivative is nonnegative on its entire domain.

Convex hull

In geometry, the convex hull or convex envelope or convex closure of a shape is the smallest convex set that contains it. The convex hull may be defined either as the intersection of all convex sets containing a given subset of a Euclidean space, or equivalently as the set of all convex combinations of points in the subset. For a bounded subset of the plane, the convex hull may be visualized as the shape enclosed by a rubber band stretched around the subset. Convex hulls of open sets are open, and convex hulls of compact sets are compact.

Mathematical optimization

Mathematical optimization (alternatively spelled optimisation) or mathematical programming is the selection of a best element, with regard to some criterion, from some set of available alternatives. It is generally divided into two subfields: discrete optimization and continuous optimization. Optimization problems arise in all quantitative disciplines from computer science and engineering to operations research and economics, and the development of solution methods has been of interest in mathematics for centuries.

Related lectures (64)

Stochastic Gradient Descent: Non-convex Optimization Techniques

Discusses Stochastic Gradient Descent and its application in non-convex optimization, focusing on convergence rates and challenges in machine learning.

Convex Optimization: Gradient Algorithms

Covers convex optimization problems and gradient-based algorithms to find the global minimum.

Gradient Descent: Principles and Applications

Covers gradient descent, its principles, applications, and convergence rates in optimization for machine learning.

Structures in Non-Convex Optimization

Covers non-convex optimization, deep learning training problems, stochastic gradient descent, adaptive methods, and neural network architectures.

Optimization: Gradient Descent and Subgradients