Lecture

Learning Sparse Features: Overfitting in Neural Networks

Graph Chat

Chat with Graph Search

Ask any question about EPFL courses, lectures, exercises, research, news, etc. or try the example questions below.

DISCLAIMER: The Graph Chatbot is not programmed to provide explicit or categorical answers to your questions. Rather, it transforms your questions into API requests that are distributed across the various IT services officially administered by EPFL. Its purpose is solely to collect and recommend relevant references to content that you can explore to help you answer your questions.

In course

PHYS-754: Lecture series on scientific machine learning

This lecture presents ongoing work on how scientific questions can be tackled using machine learning. Machine learning enables extracting knowledge from data computationally and in an automatized way. We will learn on examples how this is influencing the v ...

Description

This lecture explores how learning sparse features can lead to overfitting in neural networks. Despite theoretical expectations, empirical evidence shows that generalization is possible due to learning meaningful features. The presentation discusses the impact of feature learning versus lazy training on generalization error scaling and the smoothness of image datasets.

Instructors (10)

Paolo De Los Rios

Paolo De Los Rios earned his master in Electronic Engineering at the Turin Institute of Technology (Politecnico di Torino) in May 1993. In November 1993 he moved to Trieste, Italy, to enter the PhD program in Theoretical Condensed Matter Theory at the International School for Advanced Studies (SISSA/ISAS) where he obtained the PhD degree in October 1996 for his work on the statistical physics of disordered systems. After a one year postdoc at the Max-Planck Institute for the Physics of Complex Systems in Dresden, Germany, in November 1997 he moved to the University of Fribourg, Switzerland, to join the group of Prof. Yi-Cheng Zhang. There he has worked on various applications of statistical physics to complex systems. In September 2000 he has been appointed Assistant Professor in Statistical Physics of Living Matter and Complex Systems at the Institute of Theoretical Physics of the University of Lausanne, Switzerland. Since April 2010 he is Associate Professor at the Institute of Theoretical Physics of the Ecole Polytechnique Fédérale de Lausanne (EPFL), Switzerland.

Anne-Clémence Corminboeuf

Teaching & PhD PhD Students Liam Oliver Marsh, Pratham Pushparaj Deshmukh, Roel Christiaan M Raike, Thanapat Worakul, Osvaldo Hernandez Cuellar, Juliette Veronique Schleicher, Yannick Calvino Alonso, Sara Bassetta, Yuri Cho, Kelian Dietmar Marc Gaedecke, Guoyuan Liu Past EPFL PhD Students Fabrice Avaltroni, Daniel Filipe Jana, Stephan Niklaus Steinmann, Jérôme Florian Gonthier, Riccardo Petraglia, Antonio Prlj, Laurent Alexis Clément Vannay, Alberto Fabrizio, Kun-Han Lin, Boodsarin Sawatlon, Raimon Fabreg

Philippe Schwaller

Michele Ceriotti

Michele Ceriotti received his Ph.D. in Physics from ETH Zürich in 2010. He spent three years in Oxford as a Junior Research Fellow at Merton College. Since 2013 he leads the laboratory for Computational Science and Modeling in the Institute of Materials at EPFL. His research revolves around the atomic-scale modelling of materials, based on the sampling of quantum and thermal fluctuations and on the use of machine learning to predict and rationalize structure-property relations. He has been awarded the IBM Research Forschungspreis in 2010, the Volker Heine Young Investigator Award in 2013, an ERC Starting Grant in 2016, and the IUPAP C10 Young Scientist Prize in 2018.

Official source

Ontological neighbourhood

Information engineering

Machine learning: Artificial neural networks

Related concepts (32)

Deep learning

Deep learning is part of a broader family of machine learning methods, which is based on artificial neural networks with representation learning. The adjective "deep" in deep learning refers to the use of multiple layers in the network. Methods used can be either supervised, semi-supervised or unsupervised.

Generalization error

For supervised learning applications in machine learning and statistical learning theory, generalization error (also known as the out-of-sample error or the risk) is a measure of how accurately an algorithm is able to predict outcome values for previously unseen data. Because learning algorithms are evaluated on finite samples, the evaluation of a learning algorithm may be sensitive to sampling error. As a result, measurements of prediction error on the current data may not provide much information about predictive ability on new data.

Sphere

A sphere () is a geometrical object that is a three-dimensional analogue to a two-dimensional circle. Formally, a sphere is the set of points that are all at the same distance r from a given point in three-dimensional space. That given point is the centre of the sphere, and r is the sphere's radius. The earliest known mentions of spheres appear in the work of the ancient Greek mathematicians. The sphere is a fundamental object in many fields of mathematics. Spheres and nearly-spherical shapes also appear in nature and industry.

Online machine learning

In computer science, online machine learning is a method of machine learning in which data becomes available in a sequential order and is used to update the best predictor for future data at each step, as opposed to batch learning techniques which generate the best predictor by learning on the entire training data set at once. Online learning is a common technique used in areas of machine learning where it is computationally infeasible to train over the entire dataset, requiring the need of out-of-core algorithms.

Feature (computer vision)

In computer vision and , a feature is a piece of information about the content of an image; typically about whether a certain region of the image has certain properties. Features may be specific structures in the image such as points, edges or objects. Features may also be the result of a general neighborhood operation or feature detection applied to the image. Other examples of features are related to motion in image sequences, or to shapes defined in terms of curves or boundaries between different image regions.

Related lectures (64)

Neural Networks: Training and Activation

Explores neural networks, activation functions, backpropagation, and PyTorch implementation.

Neural Networks

Explores neural networks, hidden layers, weight adjustments, activation functions, and the universal approximation theorem.

Landscape and Generalisation in Deep Learning

Explores the challenges and insights of deep learning, focusing on loss landscape, generalization, and feature learning.

Perception: Data-Driven Approaches

Explores perception in deep learning for autonomous vehicles, covering image classification, optimization methods, and the role of representation in machine learning.

Gradient-Based Algorithms in High-Dimensional Learning