Publications associées à Saddle-to-Saddle Dynamics in Diagonal Linear Networks

Publications associées (42)

High-Dimensional Kernel Methods under Covariate Shift: Data-Dependent Implicit Regularization

This paper studies kernel ridge regression in high dimensions under covariate shifts and analyzes the role of importance re-weighting. We first derive the asymptotic expansion of high dimensional kernels under covariate shifts. By a bias-variance decomposi ...

2024

Deep learning approach for identification of H II regions during reionization in 21-cm observations - II. Foreground contamination

Jean-Paul Richard Kneib, Emma Elizabeth Tolley, Michele Bianco, Tianyue Chen

The upcoming Square Kilometre Array Observatory will produce images of neutral hydrogen distribution during the epoch of reionization by observing the corresponding 21-cm signal. However, the 21-cm signal will be subject to instrumental limitations such as ...

Oxford Univ Press2024

Statistical Inference for Inverse Problems: From Sparsity-Based Methods to Neural Networks

Pakshal Narendra Bohra

In inverse problems, the task is to reconstruct an unknown signal from its possibly noise-corrupted measurements. Penalized-likelihood-based estimation and Bayesian estimation are two powerful statistical paradigms for the resolution of such problems. They ...

EPFL2024

MaskCLR: Attention-Guided Contrastive Learning for Robust Action Representation Learning

Alexandre Massoud Alahi, Mohamed Ossama Ahmed Abdelfattah, Mariam Ahmed Mahmoud Hegazy Hassan

Current transformer-based skeletal action recognition models tend to focus on a limited set of joints and low-level motion patterns to predict action classes. This results in significant performance degradation under small skeleton perturbations or changin ...

2024

Spectral Phase Transition and Optimal PCA in Block-Structured Spiked Models

Florent Gérard Krzakala, Pierre Mergny

We discuss the inhomogeneous Wigner spike model, a theoretical framework recently introduced to study structured noise in various learning scenarios, through the prism of random matrix theory, with a specific focus on its spectral properties. Our primary o ...

2024

Robust Collaborative Learning with Linear Gradient Overhead

Rachid Guerraoui, Nirupam Gupta, Rafaël Benjamin Pinot, Le Nguyen Hoang, Sadegh Farhadkhani, John Stephan

Collaborative learning algorithms, such as distributed SGD (or D-SGD), are prone to faulty machines that may deviate from their prescribed algorithm because of software or hardware bugs, poisoned data or malicious behaviors. While many solutions have been ...

PLMR2023

A Statistical Framework to Investigate the Optimality of Signal-Reconstruction Methods

Michaël Unser, Pakshal Narendra Bohra, Pol del Aguila Pla

We present a statistical framework to benchmark the performance of reconstruction algorithms for linear inverse problems, in particular, neural-network-based methods that require large quantities of training data. We generate synthetic signals as realizati ...

2023

(S)GD over Diagonal Linear Networks: Implicit Regularisation, Large Stepsizes and Edge of Stability

Nicolas Henri Bernard Flammarion, Scott William Pesme

In this paper, we investigate the impact of stochasticity and large stepsizes on the implicit regularisation of gradient descent (GD) and stochastic gradient descent (SGD) over diagonal linear networks. We prove the convergence of GD and SGD with macroscop ...

2023

Compact atomic descriptors enable accurate predictions via linear models

Kevin Rossi

We probe the accuracy of linear ridge regression employing a three-body local density representation derived from the atomic cluster expansion. We benchmark the accuracy of this framework in the prediction of formation energies and atomic forces in molecul ...

AIP Publishing2021

A Neural-Network-Based Convex Regularizer for Inverse Problems

Michaël Unser, Alexis Marie Frederic Goujon, Pakshal Narendra Bohra, Sebastian Jonas Neumayer, Stanislas Ducotterd

The emergence of deep-learning-based methods to solve image-reconstruction problems has enabled a significant increase in quality. Unfortunately, these new methods often lack reliability and explainability, and there is a growing interest to address these ...

2023

Sparsest piecewise-linear regression of one-dimensional data

Michaël Unser, Julien René Fageot, Thomas Jean Debarre, Quentin Alain Denoyelle

We study the problem of one-dimensional regression of data points with total-variation (TV) regularization (in the sense of measures) on the second derivative, which is known to promote piecewise-linear solutions with few knots. While there are efficient a ...

Elsevier2021

Travel Time Prediction for Congested Freeways With a Dynamic Linear Model

Nikolaos Geroliminis, Semin Kwak

Accurate prediction of travel time is an essential feature to support Intelligent Transportation Systems (ITS). The non-linearity of traffic states, however, makes this prediction a challenging task. Here we propose to use dynamic linear models (DLMs) to a ...

IEEE Institute of Electrical and Electronics Engineers2021

Regularization Techniques for Low-Resource Machine Translation

Alejandro Ramírez Atrio

Neural machine translation (MT) and text generation have recently reached very high levels of quality. However, both areas share a problem: in order to reach these levels, they require massive amounts of data. When this is not present, they lack generaliza ...

EPFL2023

TV-based reconstruction of periodic functions

Julien René Fageot, Matthieu Martin Jean-André Simeoni

We introduce a general framework for the reconstruction of periodic multivariate functions from finitely many and possibly noisy linear measurements. The reconstruction task is formulated as a penalized convex optimization problem, taking the form of a sum ...

IOP PUBLISHING LTD2020

Incremental Learning in Diagonal Linear Networks

Raphaël Jean Berthier

Diagonal linear networks (DLNs) are a toy simplification of artificial neural networks; they consist in a quadratic reparametrization of linear regression inducing a sparse implicit regularization. In this paper, we describe the trajectory of the gradient ...

Microtome Publishing2023

Generalization error in high-dimensional perceptrons: Approaching Bayes error with convex optimization

Lenka Zdeborová, Florent Gérard Krzakala

We consider a commonly studied supervised classification of a synthetic dataset whose labels are generated by feeding a one-layer non-linear neural network with random iid inputs. We study the generalization performances of standard classifiers in the high ...

Curran Associates, Inc.2020

Efficient Greedy Coordinate Descent for Composite Problems

Martin Jaggi, Sebastian Urban Stich, Sai Praneeth Reddy Karimireddy, Anastasiia Koloskova

Coordinate descent with random coordinate selection is the current state of the art for many large scale optimization problems. However, greedy selection of the steepest coordinate on smooth problems can yield convergence rates independent of the dimension ...

2019

Democratizing Machine Learning

Nirupam Gupta, Rafaël Benjamin Pinot, Alexandre David Olivier Maurer

The increasing prevalence of personal devices motivates the design of algorithms that can leverage their computing power, together with the data they generate, in order to build privacy-preserving and effective machine learning models. However, traditional ...

IEEE2022

Manifold Learning-Based Polynomial Chaos Expansions For High-Dimensional Surrogate Models

Ketson Roberto Maximiano Dos Santos

In this work we introduce a manifold learning-based method for uncertainty quantification (UQ) in systems describing complex spatiotemporal processes. Our first objective is to identify the embedding of a set of high-dimensional data representing quantitie ...

2022

Hybrid-Spline Dictionaries for Continuous-Domain Inverse Problems

Michaël Unser, Shayan Aziznejad, Thomas Jean Debarre

We study one-dimensional continuous-domain inverse problems with multiple generalized total-variation regularization, which involves the joint use of several regularization operators. Our starting point is a new representer theorem that states that such in ...

2019