(S)GD over Diagonal Linear Networks: Implicit Regularisation, Large Stepsizes and Edge of Stability

Chat with Graph Search

Ask any question about EPFL courses, lectures, exercises, research, news, etc. or try the example questions below.

DISCLAIMER: The Graph Chatbot is not programmed to provide explicit or categorical answers to your questions. Rather, it transforms your questions into API requests that are distributed across the various IT services officially administered by EPFL. Its purpose is solely to collect and recommend relevant references to content that you can explore to help you answer your questions.

Rectified-linear-unit (ReLU) neural networks, which play a prominent role in deep learning, generate continuous and piecewise-linear (CPWL) functions. While they provide a powerful parametric representation, the mapping between the parameter and function s ...

Robustness and stability of image-reconstruction algorithms have recently come under scrutiny. Their importance to medical imaging cannot be overstated. We review the known results for the topical variational regularization strategies ( ℓ2 and ℓ1 regulariz ...

BackgroundTinnitus is a heterogeneous condition which may be associated with moderate to severe disability, but the reasons why only a subset of individuals is burdened by the condition are not fully clear. Ecological momentary assessment (EMA) allows a be ...

A computer-implemented method for reconstructing/recovering high-resolution visible light spectral data at a target resolution d, that comprises obtaining a configuration of a low- resolution multi-channel imaging sensor of resolution p, the configuration ...

Neural-network quantum states (NQS) have been shown to be a suitable variational ansatz to simulate out-of-equilibrium dynamics in two-dimensional systems using timedependent variational Monte Carlo (t-VMC). In particular, stable and accurate time propagat ...

We consider the problem of learning a target function corresponding to a deep, extensive-width, non-linear neural network with random Gaussian weights. We consider the asymptotic limit where the number of samples, the input dimension and the network width ...

Neural machine translation (MT) and text generation have recently reached very high levels of quality. However, both areas share a problem: in order to reach these levels, they require massive amounts of data. When this is not present, they lack generaliza ...

This paper considers the multi-agent linear least-squares problem in a server-agent network architecture. The system comprises multiple agents, each with a set of local data points. The agents are connected to a server, and there is no inter-agent communic ...

In network semi-supervised learning problems, only a subset of the network nodes is able to access the data labeling. This paper formulates a decentralized optimization problem where agents have individual decision rules to estimate, subject to the conditi ...

In data-parallel optimization of machine learning models, workers collaborate to improve their estimates of the model: more accurate gradients allow them to use larger learning rates and optimize faster. In the decentralized setting, in which workers commu ...

In this paper we fully describe the trajectory of gradient flow over diagonal linear networks in the limit of vanishing initialisation. We show that the limiting flow successively jumps from a saddle of the training loss to another until reaching the minim ...

Sharpness-Aware Minimization (SAM) is a recent training method that relies on worst-case weight perturbations which significantly improves generalization in various settings. We argue that the existing justifications for the success of SAM which are based ...

Recently there has been a surge of interest in understanding implicit regularization properties of iterative gradient-based optimization algorithms. In this paper, we study the statistical guarantees on the excess risk achieved by early-stopped unconstrain ...

The goal of this thesis is to study continuous-domain inverse problems for the reconstruction of sparse signals and to develop efficient algorithms to solve such problems computationally. The task is to recover a signal of interest as a continuous function ...

Deep Neural Networks (DNNs) have obtained impressive performance across tasks, however they still remain as black boxes, e.g., hard to theoretically analyze. At the same time, Polynomial Networks (PNs) have emerged as an alternative method with a promising ...

Countless signal processing applications include the reconstruction of signals from few indirect linear measurements. The design of effective measurement operators is typically constrained by the underlying hardware and physics, posing a challenging and of ...

The emergence of deep-learning-based methods to solve image-reconstruction problems has enabled a significant increase in quality. Unfortunately, these new methods often lack reliability and explainability, and there is a growing interest to address these ...

We develop a new Newton Frank-Wolfe algorithm to solve a class of constrained self-concordant minimization problems using linear minimization oracles (LMO). Unlike L-smooth convex functions, where the Lipschitz continuity of the objective gradient holds gl ...

(S)GD over Diagonal Linear Networks: Implicit Regularisation, Large Stepsizes and Edge of Stability

Chat with Graph Search

Stable parameterization of continuous and piecewise-linear functions

Stability of Image-Reconstruction Algorithms

Daily Contributors of Tinnitus Loudness and Distress: An Ecological Momentary Assessment Study

System and method for transcoding spectral data with task-based optimization via symmetric non-negative matrix factorization

Role of stochastic noise and generalization error in the time propagation of neural-network quantum states

Bayes-optimal Learning of Deep Random Networks of Extensive-width

Regularization Techniques for Low-Resource Machine Translation

Iterative pre-conditioning for expediting the distributed gradient-descent method: The case of linear least-squares problem

Decentralized Semi-supervised Learning over Multitask Graphs

Beyond Spectral Gap: The Role of the Topology in Decentralized Learning

Saddle-to-Saddle Dynamics in Diagonal Linear Networks

Towards Understanding Sharpness-Aware Minimization

The statistical complexity of early-stopped mirror descent

Variational Methods For Continuous-Domain Inverse Problems: the Quest for the Sparsest Solution

Regularization of polynomial networks for image recognition

Gradient-Based Learning of Discrete Structured Measurement Operators for Signal Recovery

Regularization of polynomial networks for image recognition

A Neural-Network-Based Convex Regularizer for Inverse Problems

A Newton Frank-Wolfe method for constrained self-concordant minimization