Concept

Vanishing gradient problem

Publications associées (31)

Revisiting Character-level Adversarial Attacks for Language Models

Volkan Cevher, Grigorios Chrysos, Fanghui Liu, Yongtao Wu, Elias Abad Rocamora

Adversarial attacks in Natural Language Processing apply perturbations in the character or token levels. Token-level attacks, gaining prominence for their use of gradient-based methods, are susceptible to altering sentence semantics, leading to invalid adv ...

2024

Understanding generalization and robustness in modern deep learning

Maksym Andriushchenko

In this thesis, we study two closely related directions: robustness and generalization in modern deep learning. Deep learning models based on empirical risk minimization are known to be often non-robust to small, worst-case perturbations known as adversari ...

EPFL2024

Safe Deep Neural Networks

Kyle Michael Matoba

The capabilities of deep learning systems have advanced much faster than our ability to understand them. Whilst the gains from deep neural networks (DNNs) are significant, they are accompanied by a growing risk and gravity of a bad outcome. This is troubli ...

EPFL2024

Topics in statistical physics of high-dimensional machine learning

Hugo Chao Cui

In the past few years, Machine Learning (ML) techniques have ushered in a paradigm shift, allowing the harnessing of ever more abundant sources of data to automate complex tasks. The technical workhorse behind these important breakthroughs arguably lies in ...

EPFL2024

Hamiltonian Deep Neural Networks Guaranteeing Non-Vanishing Gradients by Design

Giancarlo Ferrari Trecate, Luca Furieri, Liang Xu, Clara Lucía Galimberti

Deep Neural Networks (DNNs) training can be difficult due to vanishing and exploding gradients during weight optimization through backpropagation. To address this problem, we propose a general class of Hamiltonian DNNs (H-DNNs) that stem from the discretiz ...

2023

NMDA-driven dendritic modulation enables multitask representation learning in hierarchical sensory processing pathways

Willem Anna Mark Wybo, Bernd Albert Illing

While sensory representations in the brain depend on context, it remains unclear how such modulations are implemented at the biophysical level, and how processing layers further in the hierarchy can extract useful features for each possible contex-tual sta ...

National Academy of Sciences2023

Fast Adversarial Training With Adaptive Step Size

Sabine Süsstrunk, Mathieu Salzmann, Chen Liu

While adversarial training and its variants have shown to be the most effective algorithms to defend against adversarial attacks, their extremely slow training process makes it hard to scale to large datasets like ImageNet. The key idea of recent works to ...

Ieee-Inst Electrical Electronics Engineers Inc2023

Supervised learning and inference of spiking neural networks with temporal coding

Ana Stanojevic

The way biological brains carry out advanced yet extremely energy efficient signal processing remains both fascinating and unintelligible. It is known however that at least some areas of the brain perform fast and low-cost processing relying only on a smal ...

EPFL2023

From Kernel Methods to Neural Networks: A Unifying Variational Formulation

Michaël Unser

The minimization of a data-fidelity term and an additive regularization functional gives rise to a powerful framework for supervised learning. In this paper, we present a unifying regularization functional that depends on an operator L\documentclass[12pt]{ ...

Springer2023

Contextual Stochastic Bilevel Optimization

Daniel Kuhn, Andreas Krause, Yifan Hu

We introduce contextual stochastic bilevel optimization (CSBO) -- a stochastic bilevel optimization framework with the lower-level problem minimizing an expectation conditioned on some contextual information and the upper-level decision variable. This fram ...

2023

Uncertainty quantification in machine learning for engineering design and health prognostics: A tutorial

Olga Fink, Luca Biggio

On top of machine learning (ML) models, uncertainty quantification (UQ) functions as an essential layer of safety assurance that could lead to more principled decision making by enabling sound risk assessment and management. The safety and reliability impr ...

2023

Polynomial-time universality and limitations of deep learning

Emmanuel Abbé

The goal of this paper is to characterize function distributions that general neural networks trained by descent algorithms (GD/SGD), can or cannot learn in polytime. The results are: (1) The paradigm of general neural networks trained by SGD is poly-time ...

WILEY2023

Privacy-preserving federated neural network training and inference

Sinem Sav

Training accurate and robust machine learning models requires a large amount of data that is usually scattered across data silos. Sharing, transferring, and centralizing the data from silos, however, is difficult due to current privacy regulations (e.g., H ...

EPFL2023

Implicit Distance Functions: Learning and Applications in Robotics

Mikhail Koptev

In this thesis, we address the complex issue of collision avoidance in the joint space of robots. Avoiding collisions with both the robot's own body parts and obstacles in the environment is a critical constraint in motion planning and is crucial for ensur ...

EPFL2023

Predicting nonlinear optical scattering with physics-driven neural networks

Demetri Psaltis, Carlo Gigli

Deep neural networks trained on physical losses are emerging as promising surrogates for nonlinear numerical solvers. These tools can predict solutions to Maxwell's equations and compute gradients of output fields with respect to the material and geometric ...

AIP Publishing2023

Near-Minimax Optimal Estimation With Shallow ReLU Neural Networks

Rahul Parhi

We study the problem of estimating an unknown function from noisy data using shallow ReLU neural networks. The estimators we study minimize the sum of squared data-fitting errors plus a regularization term proportional to the squared Euclidean norm of the ...

IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC2023

Regularization Techniques for Low-Resource Machine Translation

Alejandro Ramírez Atrio

Neural machine translation (MT) and text generation have recently reached very high levels of quality. However, both areas share a problem: in order to reach these levels, they require massive amounts of data. When this is not present, they lack generaliza ...

EPFL2023

Phenomenological theory of variational quantum ground-state preparation

Ivano Tavernelli, Giuseppe Carleo

The variational approach is a cornerstone of computational physics, considering both conventional and quantum computing computational platforms. The variational quantum eigensolver algorithm aims to prepare the ground state of a Hamiltonian exploiting para ...

Amer Physical Soc2023

Robust Training and Verification of Deep Neural Networks

Fabian Ricardo Latorre Gomez

According to the proposed Artificial Intelligence Act by the European Comission (expected to pass at the end of 2023), the class of High-Risk AI Systems (Title III) comprises several important applications of Deep Learning like autonomous driving vehicles ...

EPFL2023

A dynamic attractor network model of memory formation, reinforcement and forgetting

Wulfram Gerstner, Chiara Gastaldi

Empirical evidence shows that memories that are frequently revisited are easy to recall, and that familiar items involve larger hippocampal representations than less familiar ones. In line with these observations, here we develop a modelling approach to pr ...

Public Library Science2023