Leveraging Continuous Time to Understand Momentum When Training Diagonal Linear Networks

Chat with Graph Search

Ask any question about EPFL courses, lectures, exercises, research, news, etc. or try the example questions below.

DISCLAIMER: The Graph Chatbot is not programmed to provide explicit or categorical answers to your questions. Rather, it transforms your questions into API requests that are distributed across the various IT services officially administered by EPFL. Its purpose is solely to collect and recommend relevant references to content that you can explore to help you answer your questions.

While momentum-based accelerated variants of stochastic gradient descent (SGD) are widely used when training machine learning models, there is little theoretical understanding on the generalization error of such methods. In this work, we first show that th ...

In this PhD manuscript, we explore optimisation phenomena which occur in complex neural networks through the lens of

2

-layer diagonal linear networks. This rudimentary architecture, which consists of a two layer feedforward linear network with a diagonal ...

In inverse problems, the task is to reconstruct an unknown signal from its possibly noise-corrupted measurements. Penalized-likelihood-based estimation and Bayesian estimation are two powerful statistical paradigms for the resolution of such problems. They ...

In the past few years, Machine Learning (ML) techniques have ushered in a paradigm shift, allowing the harnessing of ever more abundant sources of data to automate complex tasks. The technical workhorse behind these important breakthroughs arguably lies in ...

In this thesis, we study two closely related directions: robustness and generalization in modern deep learning. Deep learning models based on empirical risk minimization are known to be often non-robust to small, worst-case perturbations known as adversari ...

Magnonics is a research field that has gained an increasing interest in both the fundamental and applied sciences in recent years. This field aims to explore and functionalize collective spin excitations in magnetically ordered materials for modern informa ...

Efficient sampling and approximation of Boltzmann distributions involving large sets of binary variables, or spins, are pivotal in diverse scientific fields even beyond physics. Recent advances in generative neural networks have significantly impacted this ...

Modern neuroscience research is generating increasingly large datasets, from recording thousands of neurons over long timescales to behavioral recordings of animals spanning weeks, months, or even years. Despite a great variety in recording setups and expe ...

Distributed learning is the key for enabling training of modern large-scale machine learning models, through parallelising the learning process. Collaborative learning is essential for learning from privacy-sensitive data that is distributed across various ...

Modern optimization is tasked with handling applications of increasingly large scale, chiefly due to the massive amounts of widely available data and the ever-growing reach of Machine Learning. Consequently, this area of research is under steady pressure t ...

Throughout history, the pace of knowledge and information sharing has evolved into an unthinkable speed and media. At the end of the XVII century, in Europe, the ideas that would shape the "Age of Enlightenment" were slowly being developed in coffeehouses, ...

Despite the widespread empirical success of ResNet, the generalization properties of deep ResNet are rarely explored beyond the lazy training regime. In this work, we investigate scaled ResNet in the limit of infinitely deep and wide neural networks, of wh ...

Collaborative learning algorithms, such as distributed SGD (or D-SGD), are prone to faulty machines that may deviate from their prescribed algorithm because of software or hardware bugs, poisoned data or malicious behaviors. While many solutions have been ...

The variational approach is a cornerstone of computational physics, considering both conventional and quantum computing computational platforms. The variational quantum eigensolver algorithm aims to prepare the ground state of a Hamiltonian exploiting para ...

The monumental progress in the development of machine learning models has led to a plethora of applications with transformative effects in engineering and science. This has also turned the attention of the research community towards the pursuit of construc ...

Within the context of contemporary machine learning problems, efficiency of optimization process depends on the properties of the model and the nature of the data available, which poses a significant problem as the complexity of either increases ad infinit ...

Deep neural networks have become ubiquitous in today's technological landscape, finding their way in a vast array of applications. Deep supervised learning, which relies on large labeled datasets, has been particularly successful in areas such as image cla ...

The recent developments of deep learning cover a wide variety of tasks such as image classification, text translation, playing go, and folding proteins. All these successful methods depend on a gradient-based learning algorithm to train a model on massive ...

Deep Neural Networks (DNNs) training can be difficult due to vanishing and exploding gradients during weight optimization through backpropagation. To address this problem, we propose a general class of Hamiltonian DNNs (H-DNNs) that stem from the discretiz ...

This thesis focuses on two selected learning problems: 1) statistical inference on graphs models, and, 2) gradient descent on neural networks, with the common objective of defining and analysing the measures that characterize the fundamental limits. In th ...

Leveraging Continuous Time to Understand Momentum When Training Diagonal Linear Networks

Chat with Graph Search

On the Generalization of Stochastic Gradient Descent with Momentum

Deep Learning Theory Through the Lens of Diagonal Linear Networks

Statistical Inference for Inverse Problems: From Sparsity-Based Methods to Neural Networks

Topics in statistical physics of high-dimensional machine learning

Understanding generalization and robustness in modern deep learning

The 2024 Magnonics Roadmap

Sparse autoregressive neural networks for classical spin systems

Robust machine learning for neuroscientific inference

Optimization Algorithms for Decentralized, Distributed and Collaborative Machine Learning

Scalable constrained optimization

Machine Learning for Modeling Stock Returns

Generalization of Scaled Deep ResNets in the Mean-Field Regime

Robust Collaborative Learning with Linear Gradient Overhead

Phenomenological theory of variational quantum ground-state preparation

Probabilistic methods for neural combinatorial optimization

Universal and adaptive methods for robust stochastic optimization

Deep Learning Generalization with Limited and Noisy Labels

Transformer Models for Vision

Hamiltonian Deep Neural Networks Guaranteeing Non-Vanishing Gradients by Design

Fundamental Limits in Statistical Learning Problems: Block Models and Neural Networks