On the Relationship between Self-Attention and Convolutional Layers

Chat with Graph Search

Ask any question about EPFL courses, lectures, exercises, research, news, etc. or try the example questions below.

DISCLAIMER: The Graph Chatbot is not programmed to provide explicit or categorical answers to your questions. Rather, it transforms your questions into API requests that are distributed across the various IT services officially administered by EPFL. Its purpose is solely to collect and recommend relevant references to content that you can explore to help you answer your questions.

Proteins are the molecular machines of life, driving essential biological processes such as enzymatic catalysis, cellular signaling, and immune responses. The ability to design entirely new proteins with desired structures and functions has immense implica ...

The current machine learning paradigm relies on continuous representations and fixed neural network architectures to approximate environmental structures, leading to challenges with continual learning, internal structure design, and goal-directed behavior ...

Recent progress in computer vision has been driven by a simple yet transformative observation: the potential of 30-year-old neural networks can be unlocked by massively scaling their parameters and the amount of human-labeled training data. This breakthrou ...

Machine learning models (e.g., neural networks) achieve high accuracy in wind power forecasting, but they are usually regarded as black boxes that lack interpretability. To address this issue, the paper proposes a glass-box model that combines high accurac ...

This thesis focuses on understanding and improving the reasoning capabilities of neural networks. It develops theoretical results and empirical analyses to uncover reasoning potential and limitations, leveraging these insights to guide the design of improv ...

Motivation Many machine learning (ML) models developed to classify phenotype from gene expression data provide interpretations for their decisions, with the aim of understanding biological processes. For many models, including neural networks, interpretati ...

In this PhD manuscript, we explore optimisation phenomena which occur in complex neural networks through the lens of

2

-layer diagonal linear networks. This rudimentary architecture, which consists of a two layer feedforward linear network with a diagonal ...

When arranged in a crossbar configuration, resistive memory devices can be used to execute Matrix-Vector Multiplications (MVMs), the most dominant operation of many Machine Learning (ML) algorithms, in constant time complexity. Nonetheless, when performing ...

Multi-Object Tracking (MOT) is a fundamental computer vision task that involves detecting objects of interest in video frames and associating detections of the same object across time to form trajectories. For association, appearance cues, extracted throug ...

Visual affordance segmentation identifies image regions of an object an agent can interact with. Existing methods re-use and adapt learning-based architectures for semantic segmentation to the affordance segmentation task and evaluate on small-size dataset ...

Mitigating the energy requirements of artificial intelligence requires novel physical substrates for computation. Phononic metamaterials have vanishingly low power dissipation and hence are a prime candidate for green, always-on computers. However, their u ...

Decentralized learning (DL) is an emerging approach that enables nodes to collaboratively train a machine learning model without sharing raw data. In many application domains, such as healthcare, this approach faces challenges due to the high level of hete ...

Monitoring forests, in particular their response to climate and land use change, requires studying long time scales. While efficient deep learning methods have been developed to process short time series of satellite imagery, leveraging long time series of ...

Semi-supervised semantic segmentation methods leverage unlabeled data by pseudo-labeling them. Thus the success of these methods hinges on the reliability of the pseudo-labels. Existing methods mostly choose high-confidence pixels in an effort to avoid err ...

We present a finite elements-neural network approach for the numerical approximation of parametric partial differential equations. The algorithm generates training data from finite element simulations, and uses a data -driven (supervised) feedforward neura ...

Photometric stereo, a computer vision technique for estimating the 3D shape of objects through images captured under varying illumination conditions, has been a topic of research for nearly four decades. In its general formulation, photometric stereo is an ...

Deep learning has shown remarkable potential for industrial applications, particularly in predictive maintenance and condition monitoring. A key challenge in this domain is prognostics, specifically predicting the Remaining Useful Life (RUL) of assets - a ...

Throughout history, the pace of knowledge and information sharing has evolved into an unthinkable speed and media. At the end of the XVII century, in Europe, the ideas that would shape the "Age of Enlightenment" were slowly being developed in coffeehouses, ...

Data augmentation is vital in deep learning for enhancing model robustness by artificially expanding training datasets. However, advanced methods like CutMix blend images and assign labels based on pixel ratios, often introducing label noise by neglecting ...

The upcoming Square Kilometre Array Observatory will produce images of neutral hydrogen distribution during the epoch of reionization by observing the corresponding 21-cm signal. However, the 21-cm signal will be subject to instrumental limitations such as ...

On the Relationship between Self-Attention and Convolutional Layers

Chat with Graph Search

Computational design of complex protein folds and soluble analogs of membrane proteins

Modelleyen: Continual Learning and Planning via Structured Modelling of Environment Dynamics

Exploiting Representation Similarities in Self-Supervised Learning for Vision Tasks

Explainable modeling for wind power forecasting: A Glass-Box model with high accuracy

Learning to Reason with Neural Networks: Principles and Methods

A comparative analysis of gene expression profiling by statistical and machine learning approaches

Deep Learning Theory Through the Lens of Diagonal Linear Networks

LIONHEART: A Layer-based Mapping Framework for Heterogeneous Systems with Analog In-Memory Computing Tiles

Person Re-Identification and its Application to Multi-Object Tracking

Segmenting Object Affordances: Reproducibility and Sensitivity to Scale

In-Sensor Passive Speech Classification with Phononic Metamaterials

Fair Decentralized Learning

Multi-temporal forest monitoring in the Swiss Alps with knowledge-guided deep learning

Weighting Pseudo-labels via High-Activation Feature Index Similarity and Object Detection for Semi-supervised Segmentation

Error assessment of an adaptive finite elements-neural networks method for an elliptic parametric PDE

Aggregating Spatial and Photometric Context for Photometric Stereo

Overcoming Distribution Shifts and Imbalance Challenges in Representation Learning for Deep Regression Models

Machine Learning for Modeling Stock Returns

SaliencyMix plus : Noise-Minimized Image Mixing Method With Saliency Map in Data Augmentation

Deep learning approach for identification of H II regions during reionization in 21-cm observations - II. Foreground contamination