Lecture

Stochastic Gradient Descent: Theory and Applications

Graph Chat

Chat with Graph Search

Ask any question about EPFL courses, lectures, exercises, research, news, etc. or try the example questions below.

DISCLAIMER: The Graph Chatbot is not programmed to provide explicit or categorical answers to your questions. Rather, it transforms your questions into API requests that are distributed across the various IT services officially administered by EPFL. Its purpose is solely to collect and recommend relevant references to content that you can explore to help you answer your questions.

Related lectures (64)

Optimization Methods: Convergence and Trade-offs

Covers optimization methods, convergence guarantees, trade-offs, and variance reduction techniques in numerical optimization.

Supervised Learning: Gradient Descent

Covers supervised learning with gradient descent algorithms and their statistical justification.

Gradient Descent: Principles and Applications

Covers gradient descent, its principles, applications, and convergence rates in optimization for machine learning.

Double Descent Curves: Overparametrization

Explores double descent curves and overparametrization in machine learning models, highlighting the risks and benefits.

Topological Data Science

Delves into Topological Data Analysis, emphasizing the mathematical foundations of neural networks and exploring the manifold hypothesis and persistent homology.

Linear Models for Classification: Logistic Regression and SVM

Covers linear models for classification, focusing on logistic regression and support vector machines.

Conformal Blocks: Understanding Young Diagrams and Weight States

Covers conformal blocks, Young diagrams, and their significance in weight states.

Variance Reduction in Deep Learning

Explores variance reduction techniques in deep learning, covering gradient descent, stochastic gradient descent, SVRG method, and performance comparison of algorithms.

Unconstrained Optimization Theory

Explores unconstrained optimization theory, covering global and local minima, convexity, and gradient concepts.

Adversarial Machine Learning: Fundamentals and Techniques

Explores adversarial machine learning, covering the generation of adversarial examples, robustness challenges, and techniques like Fast Gradient Sign Method.

Natural Transformations: Functors and Categories

Explores functors, natural transformations, and the theory of groups, emphasizing the importance of comparisons and structure preservation.

Gradient Descent

Covers the concept of gradient descent in scalar cases, focusing on finding the minimum of a function by iteratively moving in the direction of the negative gradient.

Optimization Methods: Theory Discussion

Explores optimization methods, including unconstrained problems, linear programming, and heuristic approaches.

Optimization without Constraints: Gradient Method

Covers optimization without constraints using the gradient method to find the function's minimum.

Momentum Methods and Nonlinear Conjugate Gradients

The session explains how momentum accelerates gradient descent, presents the theory of nonlinear conjugate gradients on manifolds, compares several beta‑rule updates, discusses line‑search and preconditioner choices, and shows how to employ the Manopt library for Riemannian optimization without explicit Hessians.

Gradient Descent and Linear Regression

Covers stochastic gradient descent, linear regression, regularization, supervised learning, and the iterative nature of gradient descent.

Deep Learning III

Delves into deep learning optimization, challenges, SGD variants, critical points, overparametrized networks, and adaptive methods.

Stochastic Optimization: Algorithms and Methods

Explores stochastic optimization algorithms and methods for convex problems with smooth and nonsmooth risks.

Continuous Functions: Theory and Applications

Explores continuous functions, Cauchy criterion, and function extension by continuity.

Model-Free Prediction in Reinforcement Learning: Key Methods

Covers model-free prediction methods in reinforcement learning, focusing on Monte Carlo and Temporal Differences for estimating value functions without transition dynamics knowledge.