Recherche

Afficher tous les résultats pour

Accueil

Publication

(S)GD over Diagonal Linear Networks: Implicit Regularisation, Large Stepsizes and Edge of Stability

À propos
Confidentialité
Mentions légales

Graph Chatbot

Publications associées (41)

Page 2 sur 3

Stable parameterization of continuous and piecewise-linear functions

Michaël Unser, Alexis Marie Frederic Goujon, Joaquim Gonçalves Garcia Barreto Campos

Rectified-linear-unit (ReLU) neural networks, which play a prominent role in deep learning, generate continuous and piecewise-linear (CPWL) functions. While they provide a powerful parametric representation, the mapping between the parameter and function s ...

2023

Stability of Image-Reconstruction Algorithms

Michaël Unser, Sebastian Jonas Neumayer, Pol del Aguila Pla

Robustness and stability of image-reconstruction algorithms have recently come under scrutiny. Their importance to medical imaging cannot be overstated. We review the known results for the topical variational regularization strategies ( ℓ2 and ℓ1 regulariz ...

2023

Daily Contributors of Tinnitus Loudness and Distress: An Ecological Momentary Assessment Study

Patrick Karl Alois Neff

BackgroundTinnitus is a heterogeneous condition which may be associated with moderate to severe disability, but the reasons why only a subset of individuals is burdened by the condition are not fully clear. Ecological momentary assessment (EMA) allows a be ...

FRONTIERS MEDIA SA2022

System and method for transcoding spectral data with task-based optimization via symmetric non-negative matrix factorization

Forrest Simon Webler

A computer-implemented method for reconstructing/recovering high-resolution visible light spectral data at a target resolution d, that comprises obtaining a configuration of a low- resolution multi-channel imaging sensor of resolution p, the configuration ...

2022

Role of stochastic noise and generalization error in the time propagation of neural-network quantum states

Giuseppe Carleo

Neural-network quantum states (NQS) have been shown to be a suitable variational ansatz to simulate out-of-equilibrium dynamics in two-dimensional systems using timedependent variational Monte Carlo (t-VMC). In particular, stable and accurate time propagat ...

SCIPOST FOUNDATION2022

Bayes-optimal Learning of Deep Random Networks of Extensive-width

Lenka Zdeborová, Florent Gérard Krzakala, Hugo Chao Cui

We consider the problem of learning a target function corresponding to a deep, extensive-width, non-linear neural network with random Gaussian weights. We consider the asymptotic limit where the number of samples, the input dimension and the network width ...

2023

Regularization Techniques for Low-Resource Machine Translation

Alejandro Ramírez Atrio

Neural machine translation (MT) and text generation have recently reached very high levels of quality. However, both areas share a problem: in order to reach these levels, they require massive amounts of data. When this is not present, they lack generaliza ...

EPFL2023

Iterative pre-conditioning for expediting the distributed gradient-descent method: The case of linear least-squares problem

Nirupam Gupta

This paper considers the multi-agent linear least-squares problem in a server-agent network architecture. The system comprises multiple agents, each with a set of local data points. The agents are connected to a server, and there is no inter-agent communic ...

PERGAMON-ELSEVIER SCIENCE LTD2022

Decentralized Semi-supervised Learning over Multitask Graphs

Ali H. Sayed, Roula Nassif, Elsa Rizk

In network semi-supervised learning problems, only a subset of the network nodes is able to access the data labeling. This paper formulates a decentralized optimization problem where agents have individual decision rules to estimate, subject to the conditi ...

IEEE2022

Beyond Spectral Gap: The Role of the Topology in Decentralized Learning

Martin Jaggi, Thijs Vogels, Hadrien Hendrikx

In data-parallel optimization of machine learning models, workers collaborate to improve their estimates of the model: more accurate gradients allow them to use larger learning rates and optimize faster. In the decentralized setting, in which workers commu ...

Microtome Publishing2023

Saddle-to-Saddle Dynamics in Diagonal Linear Networks

Nicolas Henri Bernard Flammarion, Scott William Pesme

In this paper we fully describe the trajectory of gradient flow over diagonal linear networks in the limit of vanishing initialisation. We show that the limiting flow successively jumps from a saddle of the training loss to another until reaching the minim ...

2023

Towards Understanding Sharpness-Aware Minimization

Nicolas Henri Bernard Flammarion, Maksym Andriushchenko

Sharpness-Aware Minimization (SAM) is a recent training method that relies on worst-case weight perturbations which significantly improves generalization in various settings. We argue that the existing justifications for the success of SAM which are based ...

JMLR-JOURNAL MACHINE LEARNING RESEARCH2022

The statistical complexity of early-stopped mirror descent

Tomas Vaskevicius

Recently there has been a surge of interest in understanding implicit regularization properties of iterative gradient-based optimization algorithms. In this paper, we study the statistical guarantees on the excess risk achieved by early-stopped unconstrain ...

Oxford University Press2023

Variational Methods For Continuous-Domain Inverse Problems: the Quest for the Sparsest Solution

Thomas Jean Debarre

The goal of this thesis is to study continuous-domain inverse problems for the reconstruction of sparse signals and to develop efficient algorithms to solve such problems computationally. The task is to recover a signal of interest as a continuous function ...

EPFL2022

Prediction of post-stroke motor recovery benefits from measures of sub-acute widespread network damages

Dimitri Nestor Alice Van De Ville, Maria Giulia Preti, Elvira Pirondini

Rivier et al. report that global measures of brain connectivity estimated by introducing virtual lesions in healthy structural connectomes significantly increase the accuracy of motor recovery prediction in stroke patients when added to focal and clinical ...

OXFORD UNIV PRESS2023

Regularization of polynomial networks for image recognition

Volkan Cevher, Grigorios Chrysos, Bohan Wang

Deep Neural Networks (DNNs) have obtained impressive performance across tasks, however they still remain as black boxes, e.g., hard to theoretically analyze. At the same time, Polynomial Networks (PNs) have emerged as an alternative method with a promising ...

2023

Gradient-Based Learning of Discrete Structured Measurement Operators for Signal Recovery

Jonathan Paul Sauder

Countless signal processing applications include the reconstruction of signals from few indirect linear measurements. The design of effective measurement operators is typically constrained by the underlying hardware and physics, posing a challenging and of ...

2022

Sparsest piecewise-linear regression of one-dimensional data

Michaël Unser, Julien René Fageot, Thomas Jean Debarre, Quentin Alain Denoyelle

We study the problem of one-dimensional regression of data points with total-variation (TV) regularization (in the sense of measures) on the second derivative, which is known to promote piecewise-linear solutions with few knots. While there are efficient a ...

Elsevier2021

Regularization of polynomial networks for image recognition

Volkan Cevher, Grigorios Chrysos, Bohan Wang

Ieee Computer Soc2023

A Neural-Network-Based Convex Regularizer for Inverse Problems

Michaël Unser, Alexis Marie Frederic Goujon, Pakshal Narendra Bohra, Sebastian Jonas Neumayer, Stanislas Ducotterd

The emergence of deep-learning-based methods to solve image-reconstruction problems has enabled a significant increase in quality. Unfortunately, these new methods often lack reliability and explainability, and there is a growing interest to address these ...

2023