Mediaspace scheduled maintenance: Aug 25, 2026 07:00 - 12:00 AM. During this time, videos will be temporarily unavailable. Check status updates.

Concept

Vanishing gradient problem

Related publications (31)

Rank-Adaptive Time Integration Of Tree Tensor Networks

Gianluca Ceruti

A rank-adaptive integrator for the approximate solution of high-order tensor differential equations by tree tensor networks is proposed and analyzed. In a recursion from the leaves to the root, the integrator updates bases and then evolves connection tenso ...

SIAM PUBLICATIONS2023

ADAGRAD Avoids Saddle Points

Kimon Antonakopoulos

Adaptive first-order methods in optimization are prominent in machine learning and data science owing to their ability to automatically adapt to the landscape of the function being optimized. However, their convergence guarantees are typically stated in te ...

JMLR-JOURNAL MACHINE LEARNING RESEARCH2022

Distributed neural network control with dependability guarantees: a compositional port-Hamiltonian approach

Giancarlo Ferrari Trecate, Luca Furieri, Muhammad Zakwan, Clara Lucía Galimberti

Large-scale cyber-physical systems require that control policies are distributed, that is, that they only rely on local real-time measurements and communication with neighboring agents. Optimal Distributed Control (ODC) problems are, however, highly intrac ...

2022

Robust Binary Models by Pruning Randomly-initialized Networks

Sabine Süsstrunk, Mathieu Salzmann, Chen Liu

Robustness to adversarial attacks was shown to require a larger model capacity, and thus a larger memory footprint. In this paper, we introduce an approach to obtain robust yet compact models by pruning randomly-initialized binary networks. Unlike adversar ...

2022

Gradient-Based Learning of Discrete Structured Measurement Operators for Signal Recovery

Jonathan Paul Sauder

Countless signal processing applications include the reconstruction of signals from few indirect linear measurements. The design of effective measurement operators is typically constrained by the underlying hardware and physics, posing a challenging and of ...

2022

Generalization Properties of NAS under Activation and Skip Connection Search

Volkan Cevher, Grigorios Chrysos, Fanghui Liu, Zhenyu Zhu

Neural Architecture Search (NAS) has fostered the automatic discovery of stateof- the-art neural architectures. Despite the progress achieved with NAS, so far there is little attention to theoretical guarantees on NAS. In this work, we study the generaliza ...

2022

Language Transformers for Remote Sensing Visual Question Answering

Devis Tuia, Vincent Alexandre Mendez, Christel Marie Tartini-Chappuis

Remote sensing visual question answering (RSVQA) opens new avenues to promote the use of satellites data, by interfacing satellite image analysis with natural language processing. Capitalizing on the remarkable advances in natural language processing and c ...

IEEE2022

Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputs

Nicolas Henri Bernard Flammarion, Loucas Pillaud-Vivien, Etienne Patrice Boursier

The training of neural networks by gradient descent methods is a cornerstone of the deep learning revolution. Yet, despite some recent progress, a complete theory explaining its success is still missing. This article presents, for orthogonal input vectors, ...

NeurIPS2022

Phase diagram of Stochastic Gradient Descent in high-dimensional two-layer neural networks

Lenka Zdeborová, Florent Gérard Krzakala, Bruno Loureiro, Ludovic Théo Stephan, Rodrigo Soares Veiga

Despite the non-convex optimization landscape, over-parametrized shallow networks are able to achieve global convergence under gradient descent. The picture can be radically different for narrow net-works, which tend to get stuck in badly-generalizing loca ...

2022

Adaptive Laser Welding Control: A Reinforcement Learning Approach

Giulio Masinelli, Silvio Zanoli

Despite extensive research efforts in the field of laser welding, the imperfect repeatability of the weld quality still represents an open topic. Indeed, the inherent complexity of the underlying physical phenomena prevents the implementation of an effecti ...

2020

Approximating Relu Networks By Single-Spike Computation

Wulfram Gerstner, Stanislaw Andrzej Wozniak, Angeliki Pantazi, Ana Stanojevic

Developing energy-saving neural network models is a topic of rapidly increasing interest in the artificial intelligence community. Spiking neural networks (SNNs) are biologically inspired models that strive to leverage the energy efficiency stemming from a ...

IEEE2022