Attention is not all you need: pure attention loses rank doubly exponentially with depth

Chattez avec Graph Search

Posez n’importe quelle question sur les cours, conférences, exercices, recherches, actualités, etc. de l’EPFL ou essayez les exemples de questions ci-dessous.

AVERTISSEMENT : Le chatbot Graph n'est pas programmé pour fournir des réponses explicites ou catégoriques à vos questions. Il transforme plutôt vos questions en demandes API qui sont distribuées aux différents services informatiques officiellement administrés par l'EPFL. Son but est uniquement de collecter et de recommander des références pertinentes à des contenus que vous pouvez explorer pour vous aider à répondre à vos questions.

As urban populations continue to expand and vehicle density increases, the need for effective traffic monitoring and management becomes increasingly critical. Urban areas face significant challenges such as growing congestion, safety concerns, and parking ...

PSI and North Carolina State University are developing a high-resolution multi-physics core solver for Pressurized Water Reactor (PWR) analysis in Cartesian geometry, using the neutron transport code nTRACER and two Machine Learning (ML) models providing t ...

This paper focuses on source-free domain adaptation for object detection in computer vision. This task is challenging and of great practical interest, due to the cost of obtaining annotated data sets for every new domain. Recent research has proposed vario ...

Attention-based transformers have achieved tremendous success across a variety of disciplines including natural languages. To deepen our understanding of their sequential modeling capabilities, there is a growing interest in using Markov input processes to ...

Visual grouping is the ability to combine parts of objects to form whole objects, and it is crucial for general and robust vision. Despite this, a comprehensive model of how the process works in humans remains elusive. In my thesis, I investigate how well ...

Deep neural networks (DNNs) provide state-of-the-art accuracy for vision tasks, but they require significant resources for training. Thus, they are trained on cloud servers far from the edge devices that acquire the data. This issue increases communication ...

We study the impact of the batch

n_b

on the iteration time

T

of training two-layer neural networks with one-pass stochastic gradient descent (SGD) on multi-index target functions of isotropic covariates. We characterize the optimal batch size minimizin ...

The ever-increasing demand for training and inferring with larger machine-learning models requires more efficient hardware solutions due to limitations such as power dissipation and scalability. Optics is a promising contender for providing lower power com ...

Discovering principles underlying the control of animal behavior requires a tight dialogue between experiments and neuromechanical models. Such models have primarily been used to investigate motor control with less emphasis on how the brain and motor syste ...

Methods of computational quantum chemistry provide accurate approximations of molecular properties crucial for computer-aided drug discovery and other areas of chemical science. However, high computational complexity limits the scalability of their applica ...

We present the development of a new algorithm which combines state-of-the-art energy-dispersive x-ray (EDX) spectroscopy theory and a suitable machine learning formulation for the hyperspectral unmixing of scanning transmission electron microscope EDX spec ...

The process of quantifying and analyzing animal behavior involves translating the naturally occurring descriptive language of their actions into machine-readable code. Yet, codifying behavior analysis is often challenging without deep understanding of anim ...

The imaginary-time evolution of quantum states is integral to various fields, ranging from natural sciences to classical optimization or machine learning. Since simulating quantum imaginary-time evolution generally requires storing an exponentially large w ...

Multimetallic nanoparticles are of interest as functional materials due to their highly tunable properties. However, synthesizing congruent mixtures of immiscible components is limited by the need for high-temperature procedures followed by rapid quenching ...

The identification of nanomaterials with the properties required for energy-efficient electronic systems is usually a tedious human task. A workflow to rapidly localize and characterize nanomaterials at the various stages of their integration into large-sc ...

Recently, deep networks have achieved impressive semantic segmentation performance, in particular thanks to their use of larger contextual information. In this paper, we show that the resulting networks are sensitive not only to global adversarial attacks, ...

Attention is not all you need: pure attention loses rank doubly exponentially with depth

Chattez avec Graph Search

Drone-Based Urban Traffic Monitoring and User Behavior Prediction

High-resolution predictions of the coolant properties for the 3D PWR core with artificial neural networks based on CTF

Simplifying Source-Free Domain Adaptation for Object Detection: Effective Self-training Strategies and Performance Insights

Attention with Markov: A Curious Case of Single-layer Transformers

Differences in Visual Perception in Humans and Deep Neural Networks

Hierarchical Training of Deep Neural Networks Using Early Exiting

Online Learning and Information Exponents: On The Importance of Batch size, and Time/Complexity Tradeoffs

Programming nonlinear propagation for efficient optical learning machines

NeuroMechFly v2: simulating embodied sensorimotor control in adult Drosophila

∇2DFT: A Universal Quantum Chemistry Dataset of Drug-Like Molecules and a Benchmark for Neural Network Potentials

From STEM-EDXS data to phase separation and quantification using physics-guided NMF

AmadeusGPT: a natural language interface for interactive animal behavioral analysis

Stochastic Approximation of Variational Quantum Imaginary Time Evolution

Low-temperature non-equilibrium synthesis of anisotropic multimetallic nanosurface alloys for electrochemical CO2 reduction

High-speed identification of suspended carbon nanotubes using Raman spectroscopy and deep learning

Indirect Local Attacks for Context-aware Semantic Segmentation Networks