Mediaspace scheduled maintenance: Aug 25, 2026 07:00 - 12:00 AM. During this time, videos will be temporarily unavailable. Check status updates.

Lecture

Primal-dual Optimization II

In course

EE-556: Mathematics of data: from theory to computation

This course provides an overview of key advances in continuous optimization and statistical analysis for machine learning. We review recent learning formulations and models as well as their guarantees

Description

This lecture delves into primal-dual optimization methods, focusing on Lagrangian approaches. It covers the Swiss army knife of convex formulations, real-world applications, and the translation of unconstrained problems into constrained ones. Various primal-dual methods are explored, including penalty and augmented Lagrangian methods, Arrow-Hurwitz's method variants, splitting techniques, and second-order decomposition methods. The lecture also discusses the quadratic penalty and Lagrangian formulations, e-accuracy in constrained optimization, and the behavior of the AL dual function. Examples such as blind image deconvolution, maximum-weight cut of a graph, clustering, and neural networks are presented to illustrate the concepts.

Instructor

Volkan Cevher

Volkan Cevher received the B.Sc. (valedictorian) in electrical engineering from Bilkent University in Ankara, Turkey, in 1999 and the Ph.D. in electrical and computer engineering from the Georgia Institute of Technology in Atlanta, GA in 2005. He was a Research Scientist with the University of Maryland, College Park from 2006-2007 and also with Rice University in Houston, TX, from 2008-2009. Currently, he is an Associate Professor at the Swiss Federal Institute of Technology Lausanne and a Faculty Fellow in the Electrical and Computer Engineering Department at Rice University. His research interests include machine learning, signal processing theory, optimization theory and methods, and information theory. Dr. Cevher is an ELLIS fellow and was the recipient of the Google Faculty Research award in 2018, the IEEE Signal Processing Society Best Paper Award in 2016, a Best Paper Award at CAMSAP in 2015, a Best Paper Award at SPARS in 2009, and an ERC CG in 2016 as well as an ERC StG in 2011.

Official source

Ontological neighbourhood

Theoretical computer science

Algorithms and data structures: Combinatorial optimization

Related concepts (32)

Convex optimization

Convex optimization is a subfield of mathematical optimization that studies the problem of minimizing convex functions over convex sets (or, equivalently, maximizing concave functions over convex sets). Many classes of convex optimization problems admit polynomial-time algorithms, whereas mathematical optimization is in general NP-hard.

Mathematical optimization

Mathematical optimization (alternatively spelled optimisation) or mathematical programming is the selection of a best element, with regard to some criterion, from some set of available alternatives. It is generally divided into two subfields: discrete optimization and continuous optimization. Optimization problems arise in all quantitative disciplines from computer science and engineering to operations research and economics, and the development of solution methods has been of interest in mathematics for centuries.

Duality (optimization)

In mathematical optimization theory, duality or the duality principle is the principle that optimization problems may be viewed from either of two perspectives, the primal problem or the dual problem. If the primal is a minimization problem then the dual is a maximization problem (and vice versa). Any feasible solution to the primal (minimization) problem is at least as large as any feasible solution to the dual (maximization) problem.

Constrained optimization

In mathematical optimization, constrained optimization (in some contexts called constraint optimization) is the process of optimizing an objective function with respect to some variables in the presence of constraints on those variables. The objective function is either a cost function or energy function, which is to be minimized, or a reward function or utility function, which is to be maximized.

Interior-point method

Interior-point methods (also referred to as barrier methods or IPMs) are a certain class of algorithms that solve linear and nonlinear convex optimization problems. An interior point method was discovered by Soviet mathematician I. I. Dikin in 1967 and reinvented in the U.S. in the mid-1980s. In 1984, Narendra Karmarkar developed a method for linear programming called Karmarkar's algorithm, which runs in provably polynomial time and is also very efficient in practice.

Related lectures (48)

Semidefinite Programming: Formulations and Applications

Explores Semidefinite Programming formulations, SDP relaxations, and optimization strategies with convergence guarantees.

Primal-dual Optimization III: Lagrangian Gradient Methods

Explores primal-dual optimization methods, emphasizing Lagrangian gradient techniques and their applications in data optimization.

Primal-dual optimization: Theory and Computation

Explores primal-dual optimization, conjugation of functions, strong duality, and quadratic penalty methods in data mathematics.

Nonconvex Optimization: Challenges and Strategies

Explores nonconvex optimization challenges and strategies, including blind image deconvolution and SDP relaxations.

The Hidden Convex Optimization Landscape of Deep Neural Networks