Publications related to Fast Transformers with Clustered Attention

Related publications (31)

Interact with Earth Observation Images using AI: Transparent Methods and Evaluation for Visual Question Answering

Now is an exciting time for the domain of Earth observation (EO), with a multitude of diverse sensors looking at the planet from satellites, airplanes or drones. The volume of imagery acquired is massive, and hold great potential for a variety of applicati ...

EPFL2025

Graph generative deep learning models with an application to circuit topologies

Igor Krawczuk

Modern integrated circuits are tiny yet incredibly complex technological artifacts, composed of millions and billions of individual structures working in unison. Managing their complexity and facilitating their design drove part of the co-evolution of mode ...

EPFL2024

Infusing structured knowledge priors in neural models for sample-efficient symbolic reasoning

Mattia Atzeni

The ability to reason, plan and solve highly abstract problems is a hallmark of human intelligence. Recent advancements in artificial intelligence, propelled by deep neural networks, have revolutionized disciplines like computer vision and natural language ...

EPFL2024

The multimodality cell segmentation challenge: toward universal solutions

Maxime Emmanuel Scheder, Jun Ma, Sahand Jamal Rahi, Xin Yang, Vojislav Gligorovski, Marco Labagnara

Cell segmentation is a critical step for quantitative single-cell analysis in microscopy images. Existing cell segmentation methods are often tailored to specific modalities or require manual interventions to specify hyper-parameters in different experimen ...

Nature Publishing Group2024

Driving and suppressing the human language network using large language models

Martin Schrimpf

Transformer models such as GPT generate human-like language and are predictive of human brain responses to language. Here, using functional-MRI-measured brain responses to 1,000 diverse sentences, we first show that a GPT-based encoding model can predict t ...

Nature Portfolio2024

Performing and Detecting Backdoor Attacks on Face Recognition Algorithms

Alexander Carl Unnervik

The field of biometrics, and especially face recognition, has seen a wide-spread adoption the last few years, from access control on personal devices such as phones and laptops, to automated border controls such as in airports. The stakes are increasingly ...

EPFL2024

Random matrix methods for high-dimensional machine learning models

Antoine Philippe Michel Bodin

In the rapidly evolving landscape of machine learning research, neural networks stand out with their ever-expanding number of parameters and reliance on increasingly large datasets. The financial cost and computational resources required for the training p ...

EPFL2024

Quantum Protocols for ML, Physics, and Finance

Grzegorz Adam Gluch

In this thesis, we give new protocols that offer a quantum advantage for problems in ML, Physics, and Finance. Quantum mechanics gives predictions that are inconsistent with local realism. The experiment proving this fact (Bell, 1964) gives a quantum proto ...

EPFL2024

Post-correction of Historical Text Transcripts with Large Language Models: An Exploratory Study

Frédéric Kaplan, Maud Ehrmann, Matteo Romanello, Sven Nicolas Yoann Najem-Meyer, Emanuela Boros

The quality of automatic transcription of heritage documents, whether from printed, manuscripts or audio sources, has a decisive impact on the ability to search and process historical texts. Although significant progress has been made in text recognition ( ...

Association for Computational Linguistics2024

Identification of typical district configurations: A two-step global sensitivity analysis framework

François Maréchal, Jonas Schnidrig, Cédric Terrier, Arthur Antoine Chuat

The recent geopolitical conflicts in Europe have underscored the vulnerability of the current energy system to the volatility of energy carrier prices. In the prospect of defining robust energy systems ensuring sustainable energy supply in the future, the ...

Pergamon-Elsevier Science Ltd2024

Reactive collision-free motion generation in joint space via dynamical systems and sampling-based MPC

Aude Billard, Nadia Barbara Figueroa Fernandez, Mikhail Koptev

Dynamical system (DS) based motion planning offers collision-free motion, with closed-loop reactivity thanks to their analytical expression. It ensures that obstacles are not penetrated by reshaping a nominal DS through matrix modulation, which is construc ...

Sage Publications Ltd2024

Inhalation of Microplastics—A Toxicological Complexity

Florian Frédéric Vincent Breider

Humans are chronically exposed to airborne microplastics (MPs) by inhalation. Various types of polymer particles have been detected in lung samples, which could pose a threat to human health. Inhalation toxicological studies are crucial for assessing the e ...

2024

Interpret3C: Interpretable Student Clustering Through Individualized Feature Selection

Julian Thomas Blackwell, Tanja Christina Käser Jacober, Paola Mejia Domenzain, Vinitra Swamy

Clustering in education, particularly in large-scale online environments like MOOCs, is essential for understanding and adapting to diverse student needs. However, the effectiveness of clustering depends on its interpretability, which becomes challenging w ...

2024

Efficient Continual Finite-Sum Minimization

Ioannis Mavrothalassitis, Volkan Cevher, Leello Tadesse Dadi, Efstratios Panteleimon Skoulakis

Given a sequence of functions

f_1,\ldots,f_n

with

f_i:\mathcal{D}\mapsto \mathbb{R}

, finite-sum minimization seeks a point

{x}^\star \in \mathcal{D}

minimizing

\sum_{j=1}^nf_j(x)/n

. In this work, we propose a key twist into the finite-sum minimizat ...

2024

Quantifying the Unknown: Data-Driven Approaches and Applications in Energy Systems

Paul Scharnhorst

In light of the challenges posed by climate change and the goals of the Paris Agreement, electricity generation is shifting to a more renewable and decentralized pattern, while the operation of systems like buildings is increasingly electrified. This calls ...

EPFL2024

Advancing Self-Supervised Deep Learning for 3D Scene Understanding

Seyed Mohammad Mahdi Johari

Recent advancements in deep learning have revolutionized 3D computer vision, enabling the extraction of intricate 3D information from 2D images and video sequences. This thesis explores the application of deep learning in three crucial challenges of 3D com ...

EPFL2024

Energy Management of Price-Maker Community Energy Storage by Stochastic Dynamic Programming

Xue Zhang, Tianshu Yang

In this paper, we propose an analytical stochastic dynamic programming (SDP) algorithm to address the optimal management problem of price-maker community energy storage. As a price-maker, energy storage smooths price differences, thus decreasing energy arb ...

China Electric Power Research Inst2024

A Comparative Analysis of Tools & Task Types for Measuring Computational Problem-Solving

Richard Lee Davis, Engin Walter Bumbacher, Jérôme Guillaume Brender

How to measure students' Computational Problem-Solving (CPS) competencies is an ongoing research topic. Prevalent approaches vary by measurement tools (e.g., interactive programming, multiple-choice tests, or programming-independent tests) and task types ( ...

Association for Computing Machinery2024

Bayesian Parameter-Efficient Fine-Tuning for Overcoming Catastrophic Forgetting

Haolin Chen

We are motivated primarily by the adaptation of text-to-speech synthesis models; however we argue that more generic parameter-efficient fine-tuning (PEFT) is an appropriate framework to do such adaptation. Nevertheless, catastrophic forgetting remains an i ...

IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC2024

Fast and Future: Towards Efficient Forecasting in Video Semantic Segmentation

Evann Pierre Guy Courdier

Deep learning has revolutionized the field of computer vision, a success largely attributable to the growing size of models, datasets, and computational power. Simultaneously, a critical pain point arises as several computer vision applications are deploye ...

EPFL2024