Home

Publication

On the Relationship between Self-Attention and Convolutional Layers

About
Privacy
Disclaimer

Graph Chatbot

Related publications (54)

Page 3 of 3

Advancing Self-Supervised Deep Learning for 3D Scene Understanding

Seyed Mohammad Mahdi Johari

Recent advancements in deep learning have revolutionized 3D computer vision, enabling the extraction of intricate 3D information from 2D images and video sequences. This thesis explores the application of deep learning in three crucial challenges of 3D com ...

EPFL2024

Linear Complexity Self-Attention With 3rd Order Polynomials

Grigorios Chrysos

Self-attention mechanisms and non-local blocks have become crucial building blocks for state-of-the-art neural architectures thanks to their unparalleled ability in capturing long-range dependencies in the input. However their cost is quadratic with the nu ...

Ieee Computer Soc2023

High-performance deep spiking neural networks with 0.3 spikes per neuron

Wulfram Gerstner, Stanislaw Andrzej Wozniak, Guillaume Emmanuel Fernand Bellec, Ana Stanojevic, Angeliki Pantazi

Communication by rare, binary spikes is a key factor for the energy efficiency of biological brains. However, it is harder to train biologically-inspired spiking neural networks than artificial neural networks. This is puzzling given that theoretical resul ...

Springer Science and Business Media LLC2024

The autoregressive neural network architecture of the Boltzmann distribution of pairwise interacting spins systems

Indaco Biazzo

Autoregressive Neural Networks (ARNNs) have shown exceptional results in generation tasks across image, language, and scientific domains. Despite their success, ARNN architectures often operate as black boxes without a clear connection to underlying physic ...

Nature Portfolio2023

Modeling Structured Data in Attention-based Models

Alireza Mohammadshahi

Natural language processing has experienced significant improvements with the development of Transformer-based models, which employ self-attention mechanism and pre-training strategies. However, these models still present several obstacles. A notable issue ...

EPFL2023

Coupling a recurrent neural network to SPAD TCSPC systems for real-time fluorescence lifetime imaging

Edoardo Charbon, Claudio Bruschini, Andrei Ardelean, Paul Mos, Yang Lin

Fluorescence lifetime imaging (FLI) has been receiving increased attention in recent years as a powerful diagnostic technique in biological and medical research. However, existing FLI systems often suffer from a tradeoff between processing speed, accuracy, ...

Nature Portfolio2024

Residual-based attention in physics-informed neural networks

Nikolaos Stergiopoulos, Sokratis Anagnostopoulos

Driven by the need for more efficient and seamless integration of physical models and data, physics -informed neural networks (PINNs) have seen a surge of interest in recent years. However, ensuring the reliability of their convergence and accuracy remains ...

Elsevier Science Sa2024

How deep convolutional neural networks lose spatial information with training

Matthieu Wyart, Leonardo Petrini, Francesco Cagnetta, Umberto Maria Tomasini

A central question of machine learning is how deep nets manage to learn tasks in high dimensions. An appealing hypothesis is that they achieve this feat by building a representation of the data where information irrelevant to the task is lost. For image da ...

Iop Publishing Ltd2023

Sparse autoregressive neural networks for classical spin systems

Giuseppe Carleo, Dian Wu, Indaco Biazzo

Efficient sampling and approximation of Boltzmann distributions involving large sets of binary variables, or spins, are pivotal in diverse scientific fields even beyond physics. Recent advances in generative neural networks have significantly impacted this ...

Iop Publishing Ltd2024

Multienzyme deep learning models improve peptide de novo sequencing by mass spectrometry proteomics

Hamed Khakzad

Author summaryIn recent years, the application of deep learning represented a breakthrough in the mass spectrometry (MS) field by improving the assignment of the correct sequence of amino acids from observable MS spectra without prior knowledge, also known ...

PUBLIC LIBRARY SCIENCE2023

Understanding generalization and robustness in modern deep learning

Maksym Andriushchenko

In this thesis, we study two closely related directions: robustness and generalization in modern deep learning. Deep learning models based on empirical risk minimization are known to be often non-robust to small, worst-case perturbations known as adversari ...

EPFL2024

Dense Image-based Predictions for Comics Analysis

Deblina Bhattacharjee

Dense image-based prediction methods have advanced tremendously in recent years. Their remarkable development has been possible due to the ample availability of real-world imagery. While these methods work well on photographs, their abilities do not genera ...

EPFL2023

Investigating the Effect of Misalignment on Membership Privacy in the White-box Setting

Ana-Maria Cretu

Machine learning models have been shown to leak sensitive information about their training datasets. Models are increasingly deployed on devices, raising concerns that white-box access to the model parameters increases the attack surface compared to black- ...

Privacy Enhancing Technologies Symposium Advisory Board2024

Breaking the Curse of Dimensionality in Deep Neural Networks by Learning Invariant Representations

Leonardo Petrini

Artificial intelligence, particularly the subfield of machine learning, has seen a paradigm shift towards data-driven models that learn from and adapt to data. This has resulted in unprecedented advancements in various domains such as natural language proc ...

EPFL2023