Publications associées à Attention is not all you need: pure attention loses rank doubly exponentially with depth

Publications associées (18)

Drone-Based Urban Traffic Monitoring and User Behavior Prediction

As urban populations continue to expand and vehicle density increases, the need for effective traffic monitoring and management becomes increasingly critical. Urban areas face significant challenges such as growing congestion, safety concerns, and parking ...

EPFL2025

High-resolution predictions of the coolant properties for the 3D PWR core with artificial neural networks based on CTF

Marianna Papadionysiou

PSI and North Carolina State University are developing a high-resolution multi-physics core solver for Pressurized Water Reactor (PWR) analysis in Cartesian geometry, using the neutron transport code nTRACER and two Machine Learning (ML) models providing t ...

2025

Simplifying Source-Free Domain Adaptation for Object Detection: Effective Self-training Strategies and Performance Insights

Olga Fink, Florent Evariste Forest

This paper focuses on source-free domain adaptation for object detection in computer vision. This task is challenging and of great practical interest, due to the cost of obtaining annotated data sets for every new domain. Recent research has proposed vario ...

Springer Science and Business Media Deutschland GmbH2025

Attention with Markov: A Curious Case of Single-layer Transformers

Michael Christoph Gastpar, Martin Jaggi, Marco Bondaschi, Adway Girish, Ashok Vardhan Makkuva

Attention-based transformers have achieved tremendous success across a variety of disciplines including natural languages. To deepen our understanding of their sequential modeling capabilities, there is a growing interest in using Markov input processes to ...

ICLR2025

Differences in Visual Perception in Humans and Deep Neural Networks

Ben Henrik Lönnqvist

Visual grouping is the ability to combine parts of objects to form whole objects, and it is crucial for general and robust vision. Despite this, a comprehensive model of how the process works in humans remains elusive. In my thesis, I investigate how well ...

EPFL2025

Hierarchical Training of Deep Neural Networks Using Early Exiting

Pascal Frossard, Liza Carol Andrea Dunbar, Ahmet Caner Yüzügüler, Yamin Sepehri

Deep neural networks (DNNs) provide state-of-the-art accuracy for vision tasks, but they require significant resources for training. Thus, they are trained on cloud servers far from the edge devices that acquire the data. This issue increases communication ...

Ieee-Inst Electrical Electronics Engineers Inc2024

Online Learning and Information Exponents: On The Importance of Batch size, and Time/Complexity Tradeoffs

Florent Gérard Krzakala, Bruno Loureiro, Ludovic Théo Stephan, Yatin Dandi, Luca Pesce, Luca Arnaboldi

We study the impact of the batch

n_b

on the iteration time

T

of training two-layer neural networks with one-pass stochastic gradient descent (SGD) on multi-index target functions of isotropic covariates. We characterize the optimal batch size minimizin ...

JMLR2024

Programming nonlinear propagation for efficient optical learning machines

Demetri Psaltis, Christophe Moser, Niyazi Ulas Dinç, Ugur Tegin, Ilker Oguz, Mustafa Yildirim, Carlo Gigli, Jih-Liang Hsieh

The ever-increasing demand for training and inferring with larger machine-learning models requires more efficient hardware solutions due to limitations such as power dissipation and scalability. Optics is a promising contender for providing lower power com ...

Spie-Soc Photo-Optical Instrumentation Engineers2024

NeuroMechFly v2: simulating embodied sensorimotor control in adult Drosophila

Victor Alfred Stimpfling, Louise Genoud, Sibo Wang, Pavan P Ramdya, Thomas Ka Chung Lam, Femke Imreh Hurtak, Pembe Gizem Ozdil

Discovering principles underlying the control of animal behavior requires a tight dialogue between experiments and neuromechanical models. Such models have primarily been used to investigate motor control with less emphasis on how the brain and motor syste ...

Nature Publishing Group2024

∇2DFT: A Universal Quantum Chemistry Dataset of Drug-Like Molecules and a Benchmark for Neural Network Potentials

Egor Rumiantsev

Methods of computational quantum chemistry provide accurate approximations of molecular properties crucial for computer-aided drug discovery and other areas of chemical science. However, high computational complexity limits the scalability of their applica ...

Neural information processing systems foundation2024

From STEM-EDXS data to phase separation and quantification using physics-guided NMF

Thomas Tim Holvoet, Cécile Hébert, Duncan Thomas Lindsay Alexander, Nathanaël Perraudin, Guillaume Romain Obozinski

We present the development of a new algorithm which combines state-of-the-art energy-dispersive x-ray (EDX) spectroscopy theory and a suitable machine learning formulation for the hyperspectral unmixing of scanning transmission electron microscope EDX spec ...

2024

Forward-forward training of an optical neural network

Demetri Psaltis, Christophe Moser, Niyazi Ulas Dinç, Ilker Oguz, Mustafa Yildirim, Jih-Liang Hsieh

Neural networks (NNs) have demonstrated remarkable capabilities in various tasks, but their computation-intensive nature demands faster and more energy-efficient hardware implementations. Optics-based platforms, using technologies such as silicon photonics ...

Optica Publishing Group2023

AmadeusGPT: a natural language interface for interactive animal behavioral analysis

Alexander Mathis, Mackenzie Mathis, Shaokai Ye, Jessy Lauer, Mu Zhou

The process of quantifying and analyzing animal behavior involves translating the naturally occurring descriptive language of their actions into machine-readable code. Yet, codifying behavior analysis is often challenging without deep understanding of anim ...

2023

Stochastic Approximation of Variational Quantum Imaginary Time Evolution

Giuseppe Carleo

The imaginary-time evolution of quantum states is integral to various fields, ranging from natural sciences to classical optimization or machine learning. Since simulating quantum imaginary-time evolution generally requires storing an exponentially large w ...

Institute of Electrical and Electronics Engineers Inc.2023

Low-temperature non-equilibrium synthesis of anisotropic multimetallic nanosurface alloys for electrochemical CO2 reduction

Andreas Züttel, Emad Oveisi, Mo Li, Jie Zhang, Cedric David Koolen

Multimetallic nanoparticles are of interest as functional materials due to their highly tunable properties. However, synthesizing congruent mixtures of immiscible components is limited by the need for high-temperature procedures followed by rapid quenching ...

Springernature2023

High-speed identification of suspended carbon nanotubes using Raman spectroscopy and deep learning

Martin Jaggi, Luis Felipe Barba Flores

The identification of nanomaterials with the properties required for energy-efficient electronic systems is usually a tedious human task. A workflow to rapidly localize and characterize nanomaterials at the various stages of their integration into large-sc ...

SPRINGERNATURE2022

Indirect Local Attacks for Context-aware Semantic Segmentation Networks

Mathieu Salzmann, Krishna Kanth Nakka

Recently, deep networks have achieved impressive semantic segmentation performance, in particular thanks to their use of larger contextual information. In this paper, we show that the resulting networks are sensitive not only to global adversarial attacks, ...

Springer Nature2020

Capsule networks explain complex spatial processing

Michael Herzog, Adrien Christophe Doerig

Classically, visual processing is described as a cascade of local feedforward computations. This view has received striking support from the success of convolutional neural networks (CNNs). However, CNNs only roughly mimic human vision. For example, CNNs d ...

SAGE PUBLICATIONS LTD2019