Publications associées à Implicit Bias of SGD for Diagonal Linear Networks: a Provable Benefit of Stochasticity

Publications associées (34)

Optimal Control under Uncertainty: From Regret Minimization to Distributional Robustness

Recent years have witnessed a growing integration of automatic control into a wide range of cyber-physical systems, including electrical grids and autonomous transportation networks. These systems operate in uncertain and dynamic environments, exhibit comp ...

EPFL2025

Efficient Algorithms for a Class of Stochastic Hidden Convex Optimization and Its Applications in Network Revenue Management

Yifan Hu

We study a class of stochastic nonconvex optimization in the form of min(x is an element of X) F(x) := E-xi[f (phi(x, xi))], that is, F is a composition of a convex function f and a random function phi. Leveraging an (implicit) convex reformulation via a v ...

INFORMS2024

Exploiting the Signal-Leak Bias in Diffusion Models

Athanasios Fitsios, Sabine Süsstrunk, Radhakrishna Achanta, Mahmut Sami Arpa, Martin Nicolas Everaert

There is a bias in the inference pipeline of most diffusion models. This bias arises from a signal leak whose distribution deviates from the noise distribution, creating a discrepancy between training and inference processes. We demonstrate that this signa ...

2024

Deep Learning Theory Through the Lens of Diagonal Linear Networks

Scott William Pesme

In this PhD manuscript, we explore optimisation phenomena which occur in complex neural networks through the lens of

2

-layer diagonal linear networks. This rudimentary architecture, which consists of a two layer feedforward linear network with a diagonal ...

EPFL2024

Transportation-based functional ANOVA and PCA for covariance operators

Victor Panaretos, Yoav Zemel, Valentina Masarotto

We consider the problem of comparing several samples of stochastic processes with respect to their second-order structure, and describing the main modes of variation in this second order structure, if present. These tasks can be seen as an Analysis of Vari ...

Inst Mathematical Statistics-Ims2024

On the Generalization of Stochastic Gradient Descent with Momentum

Volkan Cevher, Kimon Antonakopoulos

While momentum-based accelerated variants of stochastic gradient descent (SGD) are widely used when training machine learning models, there is little theoretical understanding on the generalization error of such methods. In this work, we first show that th ...

Microtome Publishing2024

Biases in Information Selection and Processing: Survey Evidence from the Pandemic

Andreas Fuster

We conduct two survey experiments to study which information people choose to consume and how it affects their beliefs. In the first experiment, respondents choose between optimistic and pessimistic article headlines related to the COVID-19 pandemic and ar ...

2024

Leveraging Continuous Time to Understand Momentum When Training Diagonal Linear Networks

Nicolas Henri Bernard Flammarion, Scott William Pesme, Hristo Georgiev Papazov

In this work, we investigate the effect of momentum on the optimisation trajectory of gradient descent. We leverage a continuous-time approach in the analysis of momentum gradient descent with step size

\gamma

and momentum parameter

\beta

that allows u ...

2024

Understanding generalization and robustness in modern deep learning

Maksym Andriushchenko

In this thesis, we study two closely related directions: robustness and generalization in modern deep learning. Deep learning models based on empirical risk minimization are known to be often non-robust to small, worst-case perturbations known as adversari ...

EPFL2024

Optimization Algorithms for Decentralized, Distributed and Collaborative Machine Learning

Anastasiia Koloskova

Distributed learning is the key for enabling training of modern large-scale machine learning models, through parallelising the learning process. Collaborative learning is essential for learning from privacy-sensitive data that is distributed across various ...

EPFL2024

Gaia's brightest very metal-poor (VMP) stars Metallicity catalogue of a thousand VMP stars from Gaia's radial velocity spectrometer spectra

Nicolas Lawrence Etienne Longeard

Context. Gaia DR3 has offered the scientific community a remarkable dataset of approximately one million spectra acquired with the radial velocity spectrometer (RVS) in the calcium II triplet region, which is well suited to identify very metal-poor (VMP) s ...

Edp Sciences S A2024

It’s All Relative: Learning Interpretable Models for Scoring Subjective Bias in Documents from Pairwise Comparisons

Matthias Grossglauser, Aswin Suresh

We propose an interpretable model to score the subjective bias present in documents, based only on their textual content. Our model is trained on pairs of revisions of the same Wikipedia article, where one version is more biased than the other. Although pr ...

2024

Single-Photon Avalanche Diode Image Sensors for Harsh Radiation Environments

Ming-Lo Wu

The space industry has experienced substantial growth in recent years, leading to rapid advancements in space exploration and space-based technologies. Consequently, the study of electronics and sensor performance in extreme environments has become crucial ...

EPFL2024

Topics in statistical physics of high-dimensional machine learning

Hugo Chao Cui

In the past few years, Machine Learning (ML) techniques have ushered in a paradigm shift, allowing the harnessing of ever more abundant sources of data to automate complex tasks. The technical workhorse behind these important breakthroughs arguably lies in ...

EPFL2024

A lunar reconnaissance drone for cooperative exploration and high-resolution mapping of extreme locations

Hippolyte Pierre Antoine Gustave Charles Rauch, David Rodriguez Martinez

An efficient characterization of scientifically significant locations is essential prior to the return of humans to the Moon. The highest resolution imagery acquired from orbit of south-polar shadowed regions and other relevant locations remains, at best, ...

Pergamon-Elsevier Science Ltd2024

Few-shot Learning for Efficient and Effective Machine Learning Model Adaptation

Arnout Jan J Devos

Machine learning (ML) enables artificial intelligent (AI) agents to learn autonomously from data obtained from their environment to perform tasks. Modern ML systems have proven to be extremely effective, reaching or even exceeding human intelligence. Altho ...

EPFL2024

Lake surface cooling drives littoral-pelagic exchange of dissolved gases

Alfred Johny Wüest, Damien Bouffard, Hugo Nicolás Ulloa Sánchez, Tomy Doda

The extent of littoral influence on lake gas dynamics remains debated in the aquatic science community due to the lack of direct quantification of lateral gas transport. The prevalent assumption of diffusive horizontal transport in gas budgets fails to exp ...

Amer Assoc Advancement Science2024

A Streamline Upwind Petrov-Galerkin Reduced Order Method for Advection-Dominated Partial Differential Equations Under Optimal Control

Gianluigi Rozza, Fabio Zoccolan

In this paper we will consider distributed Linear-Quadratic Optimal Control Problems dealing with Advection-Diffusion PDEs for high values of the Peclet number. In this situation, computational instabilities occur, both for steady and unsteady cases. A Str ...

Walter De Gruyter Gmbh2024

Residual-based attention in physics-informed neural networks

Nikolaos Stergiopoulos, Sokratis Anagnostopoulos

Driven by the need for more efficient and seamless integration of physical models and data, physics -informed neural networks (PINNs) have seen a surge of interest in recent years. However, ensuring the reliability of their convergence and accuracy remains ...

Elsevier Science Sa2024

Quantitative convergence rates for scaling limit of SPDEs with transport noise

Lucio Galeati

We consider on the torus the scaling limit of stochastic 2D (inviscid) fluid dynamics equations with transport noise to deterministic viscous equations. Quantitative estimates on the convergence rates are provided by combining analytic and probabilistic ar ...

Academic Press Inc Elsevier Science2024