Implicit Bias of SGD for Diagonal Linear Networks: a Provable Benefit of Stochasticity

Chat with Graph Search

Ask any question about EPFL courses, lectures, exercises, research, news, etc. or try the example questions below.

DISCLAIMER: The Graph Chatbot is not programmed to provide explicit or categorical answers to your questions. Rather, it transforms your questions into API requests that are distributed across the various IT services officially administered by EPFL. Its purpose is solely to collect and recommend relevant references to content that you can explore to help you answer your questions.

Recent years have witnessed a growing integration of automatic control into a wide range of cyber-physical systems, including electrical grids and autonomous transportation networks. These systems operate in uncertain and dynamic environments, exhibit comp ...

We study a class of stochastic nonconvex optimization in the form of min(x is an element of X) F(x) := E-xi[f (phi(x, xi))], that is, F is a composition of a convex function f and a random function phi. Leveraging an (implicit) convex reformulation via a v ...

There is a bias in the inference pipeline of most diffusion models. This bias arises from a signal leak whose distribution deviates from the noise distribution, creating a discrepancy between training and inference processes. We demonstrate that this signa ...

In this PhD manuscript, we explore optimisation phenomena which occur in complex neural networks through the lens of

2

-layer diagonal linear networks. This rudimentary architecture, which consists of a two layer feedforward linear network with a diagonal ...

We consider the problem of comparing several samples of stochastic processes with respect to their second-order structure, and describing the main modes of variation in this second order structure, if present. These tasks can be seen as an Analysis of Vari ...

While momentum-based accelerated variants of stochastic gradient descent (SGD) are widely used when training machine learning models, there is little theoretical understanding on the generalization error of such methods. In this work, we first show that th ...

We conduct two survey experiments to study which information people choose to consume and how it affects their beliefs. In the first experiment, respondents choose between optimistic and pessimistic article headlines related to the COVID-19 pandemic and ar ...

In this work, we investigate the effect of momentum on the optimisation trajectory of gradient descent. We leverage a continuous-time approach in the analysis of momentum gradient descent with step size

\gamma

and momentum parameter

\beta

that allows u ...

In this thesis, we study two closely related directions: robustness and generalization in modern deep learning. Deep learning models based on empirical risk minimization are known to be often non-robust to small, worst-case perturbations known as adversari ...

Distributed learning is the key for enabling training of modern large-scale machine learning models, through parallelising the learning process. Collaborative learning is essential for learning from privacy-sensitive data that is distributed across various ...

Context. Gaia DR3 has offered the scientific community a remarkable dataset of approximately one million spectra acquired with the radial velocity spectrometer (RVS) in the calcium II triplet region, which is well suited to identify very metal-poor (VMP) s ...

We propose an interpretable model to score the subjective bias present in documents, based only on their textual content. Our model is trained on pairs of revisions of the same Wikipedia article, where one version is more biased than the other. Although pr ...

The space industry has experienced substantial growth in recent years, leading to rapid advancements in space exploration and space-based technologies. Consequently, the study of electronics and sensor performance in extreme environments has become crucial ...

In the past few years, Machine Learning (ML) techniques have ushered in a paradigm shift, allowing the harnessing of ever more abundant sources of data to automate complex tasks. The technical workhorse behind these important breakthroughs arguably lies in ...

An efficient characterization of scientifically significant locations is essential prior to the return of humans to the Moon. The highest resolution imagery acquired from orbit of south-polar shadowed regions and other relevant locations remains, at best, ...

Machine learning (ML) enables artificial intelligent (AI) agents to learn autonomously from data obtained from their environment to perform tasks. Modern ML systems have proven to be extremely effective, reaching or even exceeding human intelligence. Altho ...

The extent of littoral influence on lake gas dynamics remains debated in the aquatic science community due to the lack of direct quantification of lateral gas transport. The prevalent assumption of diffusive horizontal transport in gas budgets fails to exp ...

In this paper we will consider distributed Linear-Quadratic Optimal Control Problems dealing with Advection-Diffusion PDEs for high values of the Peclet number. In this situation, computational instabilities occur, both for steady and unsteady cases. A Str ...

Driven by the need for more efficient and seamless integration of physical models and data, physics -informed neural networks (PINNs) have seen a surge of interest in recent years. However, ensuring the reliability of their convergence and accuracy remains ...

We consider on the torus the scaling limit of stochastic 2D (inviscid) fluid dynamics equations with transport noise to deterministic viscous equations. Quantitative estimates on the convergence rates are provided by combining analytic and probabilistic ar ...

Implicit Bias of SGD for Diagonal Linear Networks: a Provable Benefit of Stochasticity

Chat with Graph Search

Optimal Control under Uncertainty: From Regret Minimization to Distributional Robustness

Efficient Algorithms for a Class of Stochastic Hidden Convex Optimization and Its Applications in Network Revenue Management

Exploiting the Signal-Leak Bias in Diffusion Models

Deep Learning Theory Through the Lens of Diagonal Linear Networks

Transportation-based functional ANOVA and PCA for covariance operators

On the Generalization of Stochastic Gradient Descent with Momentum

Biases in Information Selection and Processing: Survey Evidence from the Pandemic

Leveraging Continuous Time to Understand Momentum When Training Diagonal Linear Networks

Understanding generalization and robustness in modern deep learning

Optimization Algorithms for Decentralized, Distributed and Collaborative Machine Learning

Gaia's brightest very metal-poor (VMP) stars Metallicity catalogue of a thousand VMP stars from Gaia's radial velocity spectrometer spectra

It’s All Relative: Learning Interpretable Models for Scoring Subjective Bias in Documents from Pairwise Comparisons

Single-Photon Avalanche Diode Image Sensors for Harsh Radiation Environments

Topics in statistical physics of high-dimensional machine learning

A lunar reconnaissance drone for cooperative exploration and high-resolution mapping of extreme locations

Few-shot Learning for Efficient and Effective Machine Learning Model Adaptation

Lake surface cooling drives littoral-pelagic exchange of dissolved gases

A Streamline Upwind Petrov-Galerkin Reduced Order Method for Advection-Dominated Partial Differential Equations Under Optimal Control

Residual-based attention in physics-informed neural networks

Quantitative convergence rates for scaling limit of SPDEs with transport noise