Infinite-width limit of deep linear neural networks

Chat with Graph Search

Ask any question about EPFL courses, lectures, exercises, research, news, etc. or try the example questions below.

DISCLAIMER: The Graph Chatbot is not programmed to provide explicit or categorical answers to your questions. Rather, it transforms your questions into API requests that are distributed across the various IT services officially administered by EPFL. Its purpose is solely to collect and recommend relevant references to content that you can explore to help you answer your questions.

This paper investigates the fundamental regression task of learning k neurons (a.k.a. teachers) from Gaussian input, using two-layer ReLU neural networks with width m (a.k.a. students) and m, k = O(1), trained via gradient descent under proper initializati ...

Animals have to navigate complex environments and perform intricate swimming maneuvers in the real world. To conquer these challenges, animals evolved a variety of motion control strategies. While it is known that many factors contribute to motion control, ...

Deploying energy-efficient deep neural networks on energy-constrained edge devices is an important research topic in both machine learning and circuit design communities. Both artificial neural networks (ANNs) and spiking neural networks (SNNs) have been p ...

Deep neural networks (DNNs) are receiving increasing attention in wind power forecasting due to their ability to effectively capture complex patterns in wind data. However, their forecast errors are severely limited by the local optimal weight issue in opt ...

We present a reinforcement learning framework for constructing assemblies composed of rigid parts, which are commonly seen in many historical masonry buildings and bridges. Traditional construction methods for such structures often depend on dense scaffold ...

The ability to train ever-larger neural networks brings artificial intelligence to the forefront of scientific and technical discoveries. However, their exponentially increasing size creates a proportionally greater demand for energy and computational hard ...

In recent years, neural quantum states have emerged as a powerful variational approach, achieving state-of-the-art accuracy when representing the ground-state wave function of a great variety of quantum many-body systems, including spin lattices, interacti ...

Now is an exciting time for the domain of Earth observation (EO), with a multitude of diverse sensors looking at the planet from satellites, airplanes or drones. The volume of imagery acquired is massive, and hold great potential for a variety of applicati ...

Contemporary AI faces important limitations that remain unresolved despite significant successes. Chief among these are the inability to acquire new knowledge without destroying old, the incomprehensible and nonengineerable nature of internal representatio ...

Many empirical studies have provided evidence for the emergence of algorithmic mechanisms (abilities) in the learning of language models, that lead to qualitative improvements in the model capabilities. Yet, a theoretical characterization of how such mecha ...

Deep heteroscedastic regression models the mean and covariance of the target distribution through neural networks. The challenge arises from heteroscedasticity, which implies that the covariance is sample dependent and is often unknown. Consequently, recen ...

Multi-frequency and Doppler spectral radars are valuable tools for retrieving information on the microphysics of precipitation, particularly snowfall. However, estimating microphysical properties from radar measurements is an ill-posed problem, i.e., diffe ...

Neurons in the human amygdala and hippocampus are classically thought to encode a person’s identity invariant to visual features. However, it remains largely unknown how visual information from higher visual cortical areas is translated into such a semanti ...

High-dimensional data must be highly structured to be learnable. Although the compositional and hierarchical nature of data is often put forward to explain learnability, quantitative measurements establishing these properties are scarce. Likewise, accessin ...

Many feedforward neural networks (NNs) generate continuous and piecewise-linear (CPWL) mappings. Specifically, they partition the input domain into regions on which the mapping is affine. The number of these so-called linear regions offers a natural metric ...

In this paper, we study sampling from a posterior derived from a neural network. We propose a new probabilistic model consisting of adding noise at every pre- and post-activation in the network, arguing that the resulting posterior can be sampled using an ...

Deep digital twins (DDTs) are deep neural networks that encode the behavior of complex physical systems. DDTs are excellent system representations due to their ability to continuously adapt to operational changes and their capability to capture complex rel ...

The capabilities of deep learning systems have advanced much faster than our ability to understand them. Whilst the gains from deep neural networks (DNNs) are significant, they are accompanied by a growing risk and gravity of a bad outcome. This is troubli ...

Dexterous in-hand manipulation in robotics, particularly with multi-fingered robotic hands, poses significant challenges due to the intricate avoidance of collisions among fingers and the object being manipulated. Collision-free paths for all fingers must ...

The work presented in this thesis lies at the intersection of Machine Learning (ML) and control theory. The first part is dedicated to the design of Deep Neural Network (DNN) architectures with specific numerical properties, inspired by dynamical systems m ...

Infinite-width limit of deep linear neural networks

Chat with Graph Search

How Gradient Descent Balances Features: A Dynamical Analysis For Two-Layer Neural Networks

Using deep reinforcement learning to investigate stretch feedback during swimming of the lamprey

Eflop: a Sparsity-aware Metric for Evaluating Computational Cost in Spiking and Non-spiking Neural Networks

Improving the accuracy and interpretability of neural networks for wind power forecasting

Learning to Assemble with Alternative Plans

Training hybrid neural networks with multimode optical nonlinearities using digital twins

Accurate neural quantum states for interacting lattice bosons

Interact with Earth Observation Images using AI: Transparent Methods and Evaluation for Visual Question Answering

Foundations of a New Learning Paradigm in AI Grounded in the Principles of Evolutionary Developmental Biology

A phase transition between positional and semantic learning in a solvable model of dot-product attention

Towards Self-Supervised Covariance Estimation in Deep Heteroscedastic Regression

What can radars tell us about snowfall microphysics? Insights from a Markov chain Monte-Carlo approach

Feature-based encoding of face identity by single neurons in the human amygdala and hippocampus

Probing the latent hierarchical structure of data via diffusion models

On the number of regions of piecewise linear neural networks

Gibbs sampling the posterior of neural networks

Digital Twin Generalization with Meta and Geometric Deep Learning

Safe Deep Neural Networks

Enhancing Dexterity in Confined Spaces: Real-Time Motion Planning for Multifingered In-Hand Manipulation

Navigating the Loop: From Control Systems to Neural Networks and Back