Mediaspace scheduled maintenance: Aug 25, 2026 07:00 - 12:00 AM. During this time, videos will be temporarily unavailable. Check status updates.

Lecture

Learning Sparse Features: Overfitting in Neural Networks

In course

PHYS-754: Lecture series on scientific machine learning

This lecture presents ongoing work on how scientific questions can be tackled using machine learning. Machine learning enables extracting knowledge from data computationally and in an automatized way. We will learn on examples how this is influencing the v ...

Description

This lecture explores how learning sparse features can lead to overfitting in neural networks. Despite theoretical expectations, empirical evidence shows that generalization is possible due to learning meaningful features. The presentation discusses the impact of feature learning versus lazy training on generalization error scaling and the smoothness of image datasets.

This video is available exclusively on Mediaspace for a restricted audience. Please log in to MediaSpace to access it if you have the necessary permissions.

Watch on Mediaspace

Instructors (10)

Paolo De Los Rios

Paolo De Los Rios earned his master in Electronic Engineering at the Turin Institute of Technology (Politecnico di Torino) in May 1993. In November 1993 he moved to Trieste, Italy, to enter the PhD program in Theoretical Condensed Matter Theory at the International School for Advanced Studies (SISSA/ISAS) where he obtained the PhD degree in October 1996 for his work on the statistical physics of disordered systems. After a one year postdoc at the Max-Planck Institute for the Physics of Complex Systems in Dresden, Germany, in November 1997 he moved to the University of Fribourg, Switzerland, to join the group of Prof. Yi-Cheng Zhang. There he has worked on various applications of statistical physics to complex systems. In September 2000 he has been appointed Assistant Professor in Statistical Physics of Living Matter and Complex Systems at the Institute of Theoretical Physics of the University of Lausanne, Switzerland. Since April 2010 he is Associate Professor at the Institute of Theoretical Physics of the Ecole Polytechnique Fédérale de Lausanne (EPFL), Switzerland.

Michele Ceriotti

Michele Ceriotti received his Ph.D. in Physics from ETH Zürich in 2010. He spent three years in Oxford as a Junior Research Fellow at Merton College. Since 2013 he leads the laboratory for Computational Science and Modeling in the Institute of Materials at EPFL. His research revolves around the atomic-scale modelling of materials, based on the sampling of quantum and thermal fluctuations and on the use of machine learning to predict and rationalize structure-property relations. He has been awarded the IBM Research Forschungspreis in 2010, the Volker Heine Young Investigator Award in 2013, an ERC Starting Grant in 2016, and the IUPAP C10 Young Scientist Prize in 2018.

Anne-Florence Raphaëlle Bitbol

Lenka Zdeborová

Lenka Zdeborová is a Professor of Physics and of Computer Science in École Polytechnique Fédérale de Lausanne where she leads the Statistical Physics of Computation Laboratory. She received a PhD in physics from University Paris-Sud and from Charles University in Prague in 2008. She spent two years in the Los Alamos National Laboratory as the Director's Postdoctoral Fellow. Between 2010 and 2020 she was a researcher at CNRS working in the Institute of Theoretical Physics in CEA Saclay, France. In 2014, she was awarded the CNRS bronze medal, in 2016 Philippe Meyer prize in theoretical physics and an ERC Starting Grant, in 2018 the Irène Joliot-Curie prize, in 2021 the Gibbs lectureship of AMS. She is an editorial board member for Journal of Physics A, Physical Review E, Physical Review X, SIMODS, Machine Learning: Science and Technology, and Information and Inference. Lenka's expertise is in applications of concepts from statistical physics, such as advanced mean field methods, replica method and related message-passing algorithms, to problems in machine learning, signal processing, inference and optimization. She enjoys erasing the boundaries between theoretical physics, mathematics and computer science.

Official source

Ontological neighbourhood

Information engineering

Machine learning: Artificial neural networks

Related concepts (32)

Feature (machine learning)

In machine learning and pattern recognition, a feature is an individual measurable property or characteristic of a phenomenon. Choosing informative, discriminating and independent features is a crucial element of effective algorithms in pattern recognition, classification and regression. Features are usually numeric, but structural features such as strings and graphs are used in syntactic pattern recognition. The concept of "feature" is related to that of explanatory variable used in statistical techniques such as linear regression.

Machine learning

Machine learning (ML) is an umbrella term for solving problems for which development of algorithms by human programmers would be cost-prohibitive, and instead the problems are solved by helping machines 'discover' their 'own' algorithms, without needing to be explicitly told what to do by any human-developed algorithms. Recently, generative artificial neural networks have been able to surpass results of many previous approaches.

Generalization error

For supervised learning applications in machine learning and statistical learning theory, generalization error (also known as the out-of-sample error or the risk) is a measure of how accurately an algorithm is able to predict outcome values for previously unseen data. Because learning algorithms are evaluated on finite samples, the evaluation of a learning algorithm may be sensitive to sampling error. As a result, measurements of prediction error on the current data may not provide much information about predictive ability on new data.

Deep learning

Deep learning is part of a broader family of machine learning methods, which is based on artificial neural networks with representation learning. The adjective "deep" in deep learning refers to the use of multiple layers in the network. Methods used can be either supervised, semi-supervised or unsupervised.

Sphere

A sphere () is a geometrical object that is a three-dimensional analogue to a two-dimensional circle. Formally, a sphere is the set of points that are all at the same distance r from a given point in three-dimensional space. That given point is the centre of the sphere, and r is the sphere's radius. The earliest known mentions of spheres appear in the work of the ancient Greek mathematicians. The sphere is a fundamental object in many fields of mathematics. Spheres and nearly-spherical shapes also appear in nature and industry.

Related lectures (39)

Neural Networks

Explores neural networks, hidden layers, weight adjustments, activation functions, and the universal approximation theorem.

Neural Networks: Training and Activation

Explores neural networks, activation functions, backpropagation, and PyTorch implementation.

Perception: Data-Driven Approaches

Explores perception in deep learning for autonomous vehicles, covering image classification, optimization methods, and the role of representation in machine learning.

Landscape and Generalisation in Deep Learning

Explores the challenges and insights of deep learning, focusing on loss landscape, generalization, and feature learning.

Gradient-Based Algorithms in High-Dimensional Learning