Mediaspace scheduled maintenance: Aug 25, 2026 07:00 - 12:00 AM. During this time, videos will be temporarily unavailable. Check status updates.

Lecture

Stochastic Gradient Descent: Optimization and Convergence Analysis

In course

EE-556: Mathematics of data: from theory to computation

This course provides an overview of key advances in continuous optimization and statistical analysis for machine learning. We review recent learning formulations and models as well as their guarantees

Description

This lecture covers the optimization method of Stochastic Gradient Descent (SGD) for minimizing non-strongly convex functions. It explains the convergence rate of SGD iterates and the impact of step size on convergence. The lecture also discusses the modified Huber loss function and its computation, along with the optimal step size for running SGD to minimize this loss. Various convergence behaviors are compared and explained, shedding light on the differences observed in different scenarios.

Instructor

Volkan Cevher

Volkan Cevher received the B.Sc. (valedictorian) in electrical engineering from Bilkent University in Ankara, Turkey, in 1999 and the Ph.D. in electrical and computer engineering from the Georgia Institute of Technology in Atlanta, GA in 2005. He was a Research Scientist with the University of Maryland, College Park from 2006-2007 and also with Rice University in Houston, TX, from 2008-2009. Currently, he is an Associate Professor at the Swiss Federal Institute of Technology Lausanne and a Faculty Fellow in the Electrical and Computer Engineering Department at Rice University. His research interests include machine learning, signal processing theory, optimization theory and methods, and information theory. Dr. Cevher is an ELLIS fellow and was the recipient of the Google Faculty Research award in 2018, the IEEE Signal Processing Society Best Paper Award in 2016, a Best Paper Award at CAMSAP in 2015, a Best Paper Award at SPARS in 2009, and an ERC CG in 2016 as well as an ERC StG in 2011.

Official source

Related concepts (32)

Radius of convergence

In mathematics, the radius of convergence of a power series is the radius of the largest disk at the center of the series in which the series converges. It is either a non-negative real number or . When it is positive, the power series converges absolutely and uniformly on compact sets inside the open disk of radius equal to the radius of convergence, and it is the Taylor series of the analytic function to which it converges.

Rate of convergence

In numerical analysis, the order of convergence and the rate of convergence of a convergent sequence are quantities that represent how quickly the sequence approaches its limit. A sequence that converges to is said to have order of convergence and rate of convergence if The rate of convergence is also called the asymptotic error constant. Note that this terminology is not standardized and some authors will use rate where this article uses order (e.g., ).

Convergence of random variables

In probability theory, there exist several different notions of convergence of random variables. The convergence of sequences of random variables to some limit random variable is an important concept in probability theory, and its applications to statistics and stochastic processes. The same concepts are known in more general mathematics as stochastic convergence and they formalize the idea that a sequence of essentially random or unpredictable events can sometimes be expected to settle down into a behavior that is essentially unchanging when items far enough into the sequence are studied.

Bounded set

In mathematical analysis and related areas of mathematics, a set is called bounded if it is, in a certain sense, of finite measure. Conversely, a set which is not bounded is called unbounded. The word "bounded" makes no sense in a general topological space without a corresponding metric. Boundary is a distinct concept: for example, a circle in isolation is a boundaryless bounded set, while the half plane is unbounded yet has a boundary. A bounded set is not necessarily a closed set and vice versa.

Absolute convergence

In mathematics, an infinite series of numbers is said to converge absolutely (or to be absolutely convergent) if the sum of the absolute values of the summands is finite. More precisely, a real or complex series is said to converge absolutely if for some real number Similarly, an improper integral of a function, is said to converge absolutely if the integral of the absolute value of the integrand is finite—that is, if Absolute convergence is important for the study of infinite series because its definition is strong enough to have properties of finite sums that not all convergent series possess – a convergent series that is not absolutely convergent is called conditionally convergent, while absolutely convergent series behave "nicely".

Related lectures (46)

Nonlinear Equations: Fixed Point Method Convergence

Covers the convergence of fixed point methods for nonlinear equations, including global and local convergence theorems and the order of convergence.

Proximal and Subgradient Descent: Optimization Techniques

Discusses proximal and subgradient descent methods for optimization in machine learning.

Faster and Projected Gradient Descent: Optimization Techniques

Discusses advanced optimization techniques, focusing on faster and projected gradient descent methods in machine learning.

Real Analysis: Sequences and Limits

Covers real sequences, induction, limits, and convergence in mathematical analysis.

Numerics: semester project