On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselines

Chat with Graph Search

Ask any question about EPFL courses, lectures, exercises, research, news, etc. or try the example questions below.

DISCLAIMER: The Graph Chatbot is not programmed to provide explicit or categorical answers to your questions. Rather, it transforms your questions into API requests that are distributed across the various IT services officially administered by EPFL. Its purpose is solely to collect and recommend relevant references to content that you can explore to help you answer your questions.

Tensor trains are a versatile tool to compress and work with high-dimensional data and functions. In this work we introduce the streaming tensor train approximation (STTA), a new class of algorithms for approximating a given tensor ' in the tensor train fo ...

Forecasting is a capability inherent in humans when navigating. Humans routinely plan their paths, considering the potential future movements of those around them. Similarly, to achieve comparable sophistication and safety, autonomous systems must embrace ...

Machine learning (ML) models for molecules and materials commonly rely on a decomposition of the global target quantity into local, atom-centered contributions. This approach is convenient from a computational perspective, enabling large-scale ML-driven si ...

We propose a principled method for projecting an arbitrary square matrix to the non- convex set of asymptotically stable matrices. Leveraging ideas from large deviations theory, we show that this projection is optimal in an information-theoretic sense and ...

Text-to-image models, such as Stable Diffusion, can generate high-quality images from simple textual prompts. With methods such as Textual Inversion, it is possible to expand the vocabulary of these models with additional concepts, by learning the vocabula ...

Training deep neural network based Automatic Speech Recognition (ASR) models often requires thousands of hours of transcribed data, limiting their use to only a few languages. Moreover, current state-of-the-art acoustic models are based on the Transformer ...

Large language models (LLMs) have demonstrated human-level performance on a vast spectrum of natural language tasks. However, it is largely unexplored whether they can better internalize knowledge from a structured data, such as a knowledge graph, or from ...

The way biological brains carry out advanced yet extremely energy efficient signal processing remains both fascinating and unintelligible. It is known however that at least some areas of the brain perform fast and low-cost processing relying only on a smal ...

Learning-based outlier (mismatched correspondence) rejection for robust 3D registration generally formulates the outlier removal as an inlier/outlier classification problem. The core for this to be successful is to learn the discriminative inlier/outlier f ...

Term extraction is an information extraction task at the root of knowledge discovery platforms. Developing term extractors that are able to generalize across very diverse and potentially highly technical domains is challenging, as annotations for domains r ...

Flood prediction in ungauged catchments is usually conducted by hydrological models that are parameterized based on nearby and similar gauged catchments. As an alternative to this process-based modelling, deep learning (DL) models have demonstrated their a ...

Natural language processing and other artificial intelligence fields have witnessed impressive progress over the past decade. Although some of this progress is due to algorithmic advances in deep learning, the majority has arguably been enabled by scaling ...

We propose a physics-informed neural network (PINN) as the forward model for tomographic reconstructions of biological samples. We demonstrate that by training this network with the Helmholtz equation as a physical loss, we can predict the scattered field ...

Occupant behavior is a highly stochastic phenomenon, which is known as a key challenge for the optimal control of residential energy systems. With the increasing share of renewable energy in the building sector, the volatile nature of renewable energy is a ...

Test-time domain adaptation aims to adapt a source pretrained model to a target domain without using any source data. Existing works mainly consider the case where the target domain is static. However, real-world machine perception systems are running in n ...

As distribution shifts are inescapable in realistic clinical scenarios due to inconsistencies in imaging protocols, scanner vendors, and across different centers, well-trained deep models incur a domain generalization problem in unseen environments. Despit ...

In this thesis, we reveal that supervised learning and inverse problems share similar mathematical foundations. Consequently, we are able to present a unified variational view of these tasks that we formulate as optimization problems posed over infinite-di ...

Remote sensing visual question answering (RSVQA) opens new avenues to promote the use of satellites data, by interfacing satellite image analysis with natural language processing. Capitalizing on the remarkable advances in natural language processing and c ...

On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselines

Chat with Graph Search

STREAMING TENSOR TRAIN APPROXIMATION

Deep Generative Models for Autonomous Driving: from Motion Forecasting to Realistic Image Synthesis

Robustness of Local Predictions in Atomistic Machine Learning Models

Efficient Learning of a Linear Dynamical System with Stability Guarantees

VETIM: Expanding the Vocabulary of Text-to-Image Models only with Text

Efficient Transformer-Based Speech Recognition

SKILL: Structured Knowledge Infusion for Large Language Models

Supervised learning and inference of spiking neural networks with temporal coding

Robust Outlier Rejection for 3D Registration with Variational Bayes

Unsupervised Term Extraction for Highly Technical Domains

Generalization of an Encoder-Decoder LSTM model for flood prediction in ungauged catchments

Text Representation Learning for Low Cost Natural Language Understanding

Physics-informed neural networks for diffraction tomography

Reinforcement learning for occupant-centric operation of residential energy system

Continual Test-Time Domain Adaptation

OptTTA: Learnable Test-Time Augmentation for Source-Free Medical Image Segmentation Under Domain Shift

Optimization Over Banach Spaces: A Unified View on Supervised Learning and Inverse Problems

Language Transformers for Remote Sensing Visual Question Answering