Saddle-to-Saddle Dynamics in Diagonal Linear Networks

Chat with Graph Search

Ask any question about EPFL courses, lectures, exercises, research, news, etc. or try the example questions below.

DISCLAIMER: The Graph Chatbot is not programmed to provide explicit or categorical answers to your questions. Rather, it transforms your questions into API requests that are distributed across the various IT services officially administered by EPFL. Its purpose is solely to collect and recommend relevant references to content that you can explore to help you answer your questions.

This paper studies kernel ridge regression in high dimensions under covariate shifts and analyzes the role of importance re-weighting. We first derive the asymptotic expansion of high dimensional kernels under covariate shifts. By a bias-variance decomposi ...

The upcoming Square Kilometre Array Observatory will produce images of neutral hydrogen distribution during the epoch of reionization by observing the corresponding 21-cm signal. However, the 21-cm signal will be subject to instrumental limitations such as ...

In inverse problems, the task is to reconstruct an unknown signal from its possibly noise-corrupted measurements. Penalized-likelihood-based estimation and Bayesian estimation are two powerful statistical paradigms for the resolution of such problems. They ...

Current transformer-based skeletal action recognition models tend to focus on a limited set of joints and low-level motion patterns to predict action classes. This results in significant performance degradation under small skeleton perturbations or changin ...

We discuss the inhomogeneous Wigner spike model, a theoretical framework recently introduced to study structured noise in various learning scenarios, through the prism of random matrix theory, with a specific focus on its spectral properties. Our primary o ...

Collaborative learning algorithms, such as distributed SGD (or D-SGD), are prone to faulty machines that may deviate from their prescribed algorithm because of software or hardware bugs, poisoned data or malicious behaviors. While many solutions have been ...

We present a statistical framework to benchmark the performance of reconstruction algorithms for linear inverse problems, in particular, neural-network-based methods that require large quantities of training data. We generate synthetic signals as realizati ...

In this paper, we investigate the impact of stochasticity and large stepsizes on the implicit regularisation of gradient descent (GD) and stochastic gradient descent (SGD) over diagonal linear networks. We prove the convergence of GD and SGD with macroscop ...

We probe the accuracy of linear ridge regression employing a three-body local density representation derived from the atomic cluster expansion. We benchmark the accuracy of this framework in the prediction of formation energies and atomic forces in molecul ...

The emergence of deep-learning-based methods to solve image-reconstruction problems has enabled a significant increase in quality. Unfortunately, these new methods often lack reliability and explainability, and there is a growing interest to address these ...

Accurate prediction of travel time is an essential feature to support Intelligent Transportation Systems (ITS). The non-linearity of traffic states, however, makes this prediction a challenging task. Here we propose to use dynamic linear models (DLMs) to a ...

Neural machine translation (MT) and text generation have recently reached very high levels of quality. However, both areas share a problem: in order to reach these levels, they require massive amounts of data. When this is not present, they lack generaliza ...

We introduce a general framework for the reconstruction of periodic multivariate functions from finitely many and possibly noisy linear measurements. The reconstruction task is formulated as a penalized convex optimization problem, taking the form of a sum ...

Diagonal linear networks (DLNs) are a toy simplification of artificial neural networks; they consist in a quadratic reparametrization of linear regression inducing a sparse implicit regularization. In this paper, we describe the trajectory of the gradient ...

We consider a commonly studied supervised classification of a synthetic dataset whose labels are generated by feeding a one-layer non-linear neural network with random iid inputs. We study the generalization performances of standard classifiers in the high ...

Coordinate descent with random coordinate selection is the current state of the art for many large scale optimization problems. However, greedy selection of the steepest coordinate on smooth problems can yield convergence rates independent of the dimension ...

The increasing prevalence of personal devices motivates the design of algorithms that can leverage their computing power, together with the data they generate, in order to build privacy-preserving and effective machine learning models. However, traditional ...

In this work we introduce a manifold learning-based method for uncertainty quantification (UQ) in systems describing complex spatiotemporal processes. Our first objective is to identify the embedding of a set of high-dimensional data representing quantitie ...

We study one-dimensional continuous-domain inverse problems with multiple generalized total-variation regularization, which involves the joint use of several regularization operators. Our starting point is a new representer theorem that states that such in ...

Spherical deconvolution methods are widely used to estimate the brain's white-matter fiber orientations from diffusion MRI data. In this study, eight spherical deconvolution algorithms were implemented and evaluated. These included two model selection tech ...

Saddle-to-Saddle Dynamics in Diagonal Linear Networks

Chat with Graph Search

High-Dimensional Kernel Methods under Covariate Shift: Data-Dependent Implicit Regularization

Deep learning approach for identification of H II regions during reionization in 21-cm observations - II. Foreground contamination

Statistical Inference for Inverse Problems: From Sparsity-Based Methods to Neural Networks

MaskCLR: Attention-Guided Contrastive Learning for Robust Action Representation Learning

Spectral Phase Transition and Optimal PCA in Block-Structured Spiked Models

Robust Collaborative Learning with Linear Gradient Overhead

A Statistical Framework to Investigate the Optimality of Signal-Reconstruction Methods

(S)GD over Diagonal Linear Networks: Implicit Regularisation, Large Stepsizes and Edge of Stability

Compact atomic descriptors enable accurate predictions via linear models

A Neural-Network-Based Convex Regularizer for Inverse Problems

Travel Time Prediction for Congested Freeways With a Dynamic Linear Model

Regularization Techniques for Low-Resource Machine Translation

TV-based reconstruction of periodic functions

Incremental Learning in Diagonal Linear Networks

Generalization error in high-dimensional perceptrons: Approaching Bayes error with convex optimization

Efficient Greedy Coordinate Descent for Composite Problems

Democratizing Machine Learning

Manifold Learning-Based Polynomial Chaos Expansions For High-Dimensional Surrogate Models

Hybrid-Spline Dictionaries for Continuous-Domain Inverse Problems

Sparse wars: A survey and comparative study of spherical deconvolution algorithms for diffusion MRI