Mediaspace scheduled maintenance: Aug 25, 2026 07:00 - 12:00 AM. During this time, videos will be temporarily unavailable. Check status updates.

Lecture

Optimization Basics: Norms, Convexity, Differentiability

In course

EE-556: Mathematics of data: from theory to computation

This course provides an overview of key advances in continuous optimization and statistical analysis for machine learning. We review recent learning formulations and models as well as their guarantees

Description

This lecture covers the fundamentals of optimization, including norms, convexity, differentiability, and smoothness. It delves into topics such as linear algebra, analysis, convex sets and functions, convergence rates, and gradient descent methods. The presentation progresses from defining metrics and vector norms to discussing quasi-norms, semi-norms, norm balls, and dual norms. It also explores matrix norms, Schatten q-norms, operator norms, and positive definite matrices. The lecture concludes with insights on continuity, Lipschitz continuity, differentiability, gradients, and linear approximations.

Instructor

Volkan Cevher

Volkan Cevher received the B.Sc. (valedictorian) in electrical engineering from Bilkent University in Ankara, Turkey, in 1999 and the Ph.D. in electrical and computer engineering from the Georgia Institute of Technology in Atlanta, GA in 2005. He was a Research Scientist with the University of Maryland, College Park from 2006-2007 and also with Rice University in Houston, TX, from 2008-2009. Currently, he is an Associate Professor at the Swiss Federal Institute of Technology Lausanne and a Faculty Fellow in the Electrical and Computer Engineering Department at Rice University. His research interests include machine learning, signal processing theory, optimization theory and methods, and information theory. Dr. Cevher is an ELLIS fellow and was the recipient of the Google Faculty Research award in 2018, the IEEE Signal Processing Society Best Paper Award in 2016, a Best Paper Award at CAMSAP in 2015, a Best Paper Award at SPARS in 2009, and an ERC CG in 2016 as well as an ERC StG in 2011.

Official source

Ontological neighbourhood

Mathematics

Analysis: Functional analysis

Algebra: Linear algebra

Related concepts (32)

Matrix norm

In mathematics, a matrix norm is a vector norm in a vector space whose elements (vectors) are matrices (of given dimensions). Given a field of either real or complex numbers, let be the K-vector space of matrices with rows and columns and entries in the field . A matrix norm is a norm on . This article will always write such norms with double vertical bars (like so: ).

Norm (mathematics)

In mathematics, a norm is a function from a real or complex vector space to the non-negative real numbers that behaves in certain ways like the distance from the origin: it commutes with scaling, obeys a form of the triangle inequality, and is zero only at the origin. In particular, the Euclidean distance in a Euclidean space is defined by a norm on the associated Euclidean vector space, called the Euclidean norm, the 2-norm, or, sometimes, the magnitude of the vector.

Operator norm

In mathematics, the operator norm measures the "size" of certain linear operators by assigning each a real number called its . Formally, it is a norm defined on the space of bounded linear operators between two given normed vector spaces. Informally, the operator norm of a linear map is the maximum factor by which it "lengthens" vectors. Given two normed vector spaces and (over the same base field, either the real numbers or the complex numbers ), a linear map is continuous if and only if there exists a real number such that The norm on the left is the one in and the norm on the right is the one in .

Normed vector space

In mathematics, a normed vector space or normed space is a vector space over the real or complex numbers on which a norm is defined. A norm is a generalization of the intuitive notion of "length" in the physical world. If is a vector space over , where is a field equal to or to , then a norm on is a map , typically denoted by , satisfying the following four axioms: Non-negativity: for every ,. Positive definiteness: for every , if and only if is the zero vector.

Banach space

In mathematics, more specifically in functional analysis, a Banach space (pronounced ˈbanax) is a complete normed vector space. Thus, a Banach space is a vector space with a metric that allows the computation of vector length and distance between vectors and is complete in the sense that a Cauchy sequence of vectors always converges to a well-defined limit that is within the space. Banach spaces are named after the Polish mathematician Stefan Banach, who introduced this concept and studied it systematically in 1920–1922 along with Hans Hahn and Eduard Helly.

Related lectures (40)

Mathematics of Data: Optimization Basics

Covers basics on optimization, including norms, Lipschitz continuity, and convexity concepts.

Mathematics of Data: Optimization Basics

Covers optimization basics, including metrics, norms, convexity, gradients, and logistic regression, with a focus on strong convexity and convergence rates.

Optimization Basics: Linear Algebra, Analysis, Convexity

Introduces optimization basics, covering linear algebra, analysis, and convexity principles.

Signal Representations

Covers the norm of a matrix, operator, singular values, and unitary matrices in linear algebra.

Vector Spaces and Scalar Products