Mediaspace scheduled maintenance: Aug 25, 2026 07:00 - 12:00 AM. During this time, videos will be temporarily unavailable. Check status updates.

Lecture

Memory-Efficient Adaptive Optimization

Description

This lecture covers memory-efficient adaptive optimization for humungous-scale learning, focusing on joint work with Rohan Anil, Tomer Koren, and Vineet Gupta. Topics include deep non-linear learning, convex optimization, and preconditioning techniques. The instructor discusses the challenges of memory overhead in training large models and the practical applications of adaptive regularization methods.

Official source

Related concepts (32)

Mathematical optimization

Mathematical optimization (alternatively spelled optimisation) or mathematical programming is the selection of a best element, with regard to some criterion, from some set of available alternatives. It is generally divided into two subfields: discrete optimization and continuous optimization. Optimization problems arise in all quantitative disciplines from computer science and engineering to operations research and economics, and the development of solution methods has been of interest in mathematics for centuries.

Preconditioner

In mathematics, preconditioning is the application of a transformation, called the preconditioner, that conditions a given problem into a form that is more suitable for numerical solving methods. Preconditioning is typically related to reducing a condition number of the problem. The preconditioned problem is then usually solved by an iterative method. In linear algebra and numerical analysis, a preconditioner of a matrix is a matrix such that has a smaller condition number than .

Machine translation

Machine translation is use of either rule-based or probabilistic (i.e. statistical and, most recently, neural network-based) machine learning approaches to translation of text or speech from one language to another, including the contextual, idiomatic and pragmatic nuances of both languages. History of machine translation The origins of machine translation can be traced back to the work of Al-Kindi, a ninth-century Arabic cryptographer who developed techniques for systemic language translation, including cryptanalysis, frequency analysis, and probability and statistics, which are used in modern machine translation.

Neural machine translation

Neural machine translation (NMT) is an approach to machine translation that uses an artificial neural network to predict the likelihood of a sequence of words, typically modeling entire sentences in a single integrated model. They require only a fraction of the memory needed by traditional statistical machine translation (SMT) models. Furthermore, unlike conventional translation systems, all parts of the neural translation model are trained jointly (end-to-end) to maximize the translation performance.

Educational technology

Educational technology (commonly abbreviated as edutech, or edtech) is the combined use of computer hardware, software, and educational theory and practice to facilitate learning. When referred to with its abbreviation, edtech, it often refers to the industry of companies that create educational technology. In addition to the practical educational experience, educational technology is based on theoretical knowledge from various disciplines such as communication, education, psychology, sociology, artificial intelligence, and computer science.

Related lectures (52)

Structures in Non-Convex Optimization

Covers non-convex optimization, deep learning training problems, stochastic gradient descent, adaptive methods, and neural network architectures.

Optimization Techniques: Convexity in Machine Learning

Covers optimization techniques in machine learning, focusing on convexity and its implications for efficient problem-solving.

Deep Learning Building Blocks

Covers tensors, loss functions, autograd, and convolutional layers in deep learning.

Optimality of Convergence Rates: Accelerated/Stochastic Gradient Descent

Covers the optimality of convergence rates in accelerated and stochastic gradient descent methods for non-convex optimization problems.

Gradient Descent Methods: Theory and Computation