On the Relationship between Self-Attention and Convolutional Layers

Chat with Graph Search

Ask any question about EPFL courses, lectures, exercises, research, news, etc. or try the example questions below.

DISCLAIMER: The Graph Chatbot is not programmed to provide explicit or categorical answers to your questions. Rather, it transforms your questions into API requests that are distributed across the various IT services officially administered by EPFL. Its purpose is solely to collect and recommend relevant references to content that you can explore to help you answer your questions.

Self-attention mechanisms and non-local blocks have become crucial building blocks for state-of-the-art neural architectures thanks to their unparalleled ability in capturing long-range dependencies in the input. However their cost is quadratic with the nu ...

Communication by rare, binary spikes is a key factor for the energy efficiency of biological brains. However, it is harder to train biologically-inspired spiking neural networks than artificial neural networks. This is puzzling given that theoretical resul ...

Natural language processing has experienced significant improvements with the development of Transformer-based models, which employ self-attention mechanism and pre-training strategies. However, these models still present several obstacles. A notable issue ...

Fluorescence lifetime imaging (FLI) has been receiving increased attention in recent years as a powerful diagnostic technique in biological and medical research. However, existing FLI systems often suffer from a tradeoff between processing speed, accuracy, ...

Driven by the need for more efficient and seamless integration of physical models and data, physics -informed neural networks (PINNs) have seen a surge of interest in recent years. However, ensuring the reliability of their convergence and accuracy remains ...

A central question of machine learning is how deep nets manage to learn tasks in high dimensions. An appealing hypothesis is that they achieve this feat by building a representation of the data where information irrelevant to the task is lost. For image da ...

Efficient sampling and approximation of Boltzmann distributions involving large sets of binary variables, or spins, are pivotal in diverse scientific fields even beyond physics. Recent advances in generative neural networks have significantly impacted this ...

Author summaryIn recent years, the application of deep learning represented a breakthrough in the mass spectrometry (MS) field by improving the assignment of the correct sequence of amino acids from observable MS spectra without prior knowledge, also known ...

In this thesis, we study two closely related directions: robustness and generalization in modern deep learning. Deep learning models based on empirical risk minimization are known to be often non-robust to small, worst-case perturbations known as adversari ...

Dense image-based prediction methods have advanced tremendously in recent years. Their remarkable development has been possible due to the ample availability of real-world imagery. While these methods work well on photographs, their abilities do not genera ...

Machine learning models have been shown to leak sensitive information about their training datasets. Models are increasingly deployed on devices, raising concerns that white-box access to the model parameters increases the attack surface compared to black- ...

Artificial intelligence, particularly the subfield of machine learning, has seen a paradigm shift towards data-driven models that learn from and adapt to data. This has resulted in unprecedented advancements in various domains such as natural language proc ...

On the Relationship between Self-Attention and Convolutional Layers

Chat with Graph Search

Linear Complexity Self-Attention With 3rd Order Polynomials

High-performance deep spiking neural networks with 0.3 spikes per neuron

Modeling Structured Data in Attention-based Models

Coupling a recurrent neural network to SPAD TCSPC systems for real-time fluorescence lifetime imaging

Residual-based attention in physics-informed neural networks

How deep convolutional neural networks lose spatial information with training

Sparse autoregressive neural networks for classical spin systems

Multienzyme deep learning models improve peptide de novo sequencing by mass spectrometry proteomics

Understanding generalization and robustness in modern deep learning

Dense Image-based Predictions for Comics Analysis

Investigating the Effect of Misalignment on Membership Privacy in the White-box Setting

Breaking the Curse of Dimensionality in Deep Neural Networks by Learning Invariant Representations