Publication

Saddle-to-Saddle Dynamics in Diagonal Linear Networks

Related publications (28)

High-Dimensional Kernel Methods under Covariate Shift: Data-Dependent Implicit Regularization

This paper studies kernel ridge regression in high dimensions under covariate shifts and analyzes the role of importance re-weighting. We first derive the asymptotic expansion of high dimensional kernels under covariate shifts. By a bias-variance decomposi ...

2024

Statistical Inference for Inverse Problems: From Sparsity-Based Methods to Neural Networks

Pakshal Narendra Bohra

In inverse problems, the task is to reconstruct an unknown signal from its possibly noise-corrupted measurements. Penalized-likelihood-based estimation and Bayesian estimation are two powerful statistical paradigms for the resolution of such problems. They ...

EPFL2024

(S)GD over Diagonal Linear Networks: Implicit Regularisation, Large Stepsizes and Edge of Stability

Scott William Pesme, Nicolas Henri Bernard Flammarion, Mathieu Even

In this paper, we investigate the impact of stochasticity and large stepsizes on the implicit regularisation of gradient descent (GD) and stochastic gradient descent (SGD) over diagonal linear networks. We prove the convergence of GD and SGD with macroscop ...

2023

A Neural-Network-Based Convex Regularizer for Inverse Problems

Michaël Unser, Stanislas Ducotterd, Pakshal Narendra Bohra, Alexis Marie Frederic Goujon, Sebastian Jonas Neumayer

The emergence of deep-learning-based methods to solve image-reconstruction problems has enabled a significant increase in quality. Unfortunately, these new methods often lack reliability and explainability, and there is a growing interest to address these ...

2023

Regularization Techniques for Low-Resource Machine Translation

Alejandro Ramírez Atrio

Neural machine translation (MT) and text generation have recently reached very high levels of quality. However, both areas share a problem: in order to reach these levels, they require massive amounts of data. When this is not present, they lack generaliza ...

EPFL2023

Incremental Learning in Diagonal Linear Networks

Raphaël Jean Berthier

Diagonal linear networks (DLNs) are a toy simplification of artificial neural networks; they consist in a quadratic reparametrization of linear regression inducing a sparse implicit regularization. In this paper, we describe the trajectory of the gradient ...

Microtome Publishing2023

System and method for transcoding spectral data with task-based optimization via symmetric non-negative matrix factorization

Forrest Simon Webler

A computer-implemented method for reconstructing/recovering high-resolution visible light spectral data at a target resolution d, that comprises obtaining a configuration of a low- resolution multi-channel imaging sensor of resolution p, the configuration ...

2022

Role of stochastic noise and generalization error in the time propagation of neural-network quantum states

Giuseppe Carleo

Neural-network quantum states (NQS) have been shown to be a suitable variational ansatz to simulate out-of-equilibrium dynamics in two-dimensional systems using timedependent variational Monte Carlo (t-VMC). In particular, stable and accurate time propagat ...

SCIPOST FOUNDATION2022

A Least-Squares Method for the Solution of the Non-smooth Prescribed Jacobian Equation

Alexandre Caboussat, Dimitrios Gourzoulidis

We consider a least-squares/relaxation finite element method for the numerical solution of the prescribed Jacobian equation. We look for its solution via a least-squares approach. We introduce a relaxation algorithm that decouples this least-squares proble ...

SPRINGER/PLENUM PUBLISHERS2022

Compact atomic descriptors enable accurate predictions via linear models

Kevin Rossi

We probe the accuracy of linear ridge regression employing a three-body local density representation derived from the atomic cluster expansion. We benchmark the accuracy of this framework in the prediction of formation energies and atomic forces in molecul ...

AIP Publishing2021

Sparsest piecewise-linear regression of one-dimensional data

Michaël Unser, Julien René Pierre Fageot, Thomas Jean Debarre, Quentin Alain Denoyelle

We study the problem of one-dimensional regression of data points with total-variation (TV) regularization (in the sense of measures) on the second derivative, which is known to promote piecewise-linear solutions with few knots. While there are efficient a ...

Elsevier2021

Travel Time Prediction for Congested Freeways With a Dynamic Linear Model

Nikolaos Geroliminis, Semin Kwak

Accurate prediction of travel time is an essential feature to support Intelligent Transportation Systems (ITS). The non-linearity of traffic states, however, makes this prediction a challenging task. Here we propose to use dynamic linear models (DLMs) to a ...

IEEE Institute of Electrical and Electronics Engineers2021

TV-based reconstruction of periodic functions

Julien René Pierre Fageot, Matthieu Martin Jean-André Simeoni

We introduce a general framework for the reconstruction of periodic multivariate functions from finitely many and possibly noisy linear measurements. The reconstruction task is formulated as a penalized convex optimization problem, taking the form of a sum ...

IOP PUBLISHING LTD2020

Generalization error in high-dimensional perceptrons: Approaching Bayes error with convex optimization

Florent Gérard Krzakala, Lenka Zdeborová

We consider a commonly studied supervised classification of a synthetic dataset whose labels are generated by feeding a one-layer non-linear neural network with random iid inputs. We study the generalization performances of standard classifiers in the high ...

Curran Associates, Inc.2020

Efficient Greedy Coordinate Descent for Composite Problems

Sai Praneeth Reddy Karimireddy, Martin Jaggi, Sebastian Urban Stich, Anastasiia Koloskova

Coordinate descent with random coordinate selection is the current state of the art for many large scale optimization problems. However, greedy selection of the steepest coordinate on smooth problems can yield convergence rates independent of the dimension ...

2019

Hybrid-Spline Dictionaries for Continuous-Domain Inverse Problems

Michaël Unser, Shayan Aziznejad, Thomas Jean Debarre

We study one-dimensional continuous-domain inverse problems with multiple generalized total-variation regularization, which involves the joint use of several regularization operators. Our starting point is a new representer theorem that states that such in ...

2019

Sparse wars: A survey and comparative study of spherical deconvolution algorithms for diffusion MRI

Jean-Philippe Thiran, David Paul Roger Romascano, Alessandro Daducci, Muhamed Barakovic, Gabriel Girard, Gaëtan Olivier D Rensonnet, Jonathan Rafael Patino Lopez, Erick Jorge Canales Rodriguez, Marco Pizzolato

Spherical deconvolution methods are widely used to estimate the brain's white-matter fiber orientations from diffusion MRI data. In this study, eight spherical deconvolution algorithms were implemented and evaluated. These included two model selection tech ...

2019

Sparse Dictionaries for Continuous-Domain Inverse Problems

Michaël Unser, Shayan Aziznejad, Thomas Jean Debarre

We study 1D continuous-domain inverse problems for multicomponent signals. The prior assumption on these signals is that each component is sparse in a different dictionary specified by a regularization operators. We introduce a hybrid regularization functi ...

2019

Automatic L2 Regularization for Multiple Generalized Additive Models

Yousra El Bachir

Multiple generalized additive models are a class of statistical regression models wherein parameters of probability distributions incorporate information through additive smooth functions of predictors. The functions are represented by basis function expan ...

EPFL2019

B-Spline-Based Exact Discretization of Continuous-Domain Inverse Problems With Generalized TV Regularization

Michaël Unser, Julien René Pierre Fageot, Harshit Gupta, Thomas Jean Debarre

We study continuous-domain linear inverse problems with generalized total-variation (gTV) regularization, expressed in terms of a regularization operator L. It has recently been proved that such inverse problems have sparse spline solutions, with fewer jum ...

IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC2019