On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselines

Chat with Graph Search

Ask any question about EPFL courses, lectures, exercises, research, news, etc. or try the example questions below.

DISCLAIMER: The Graph Chatbot is not programmed to provide explicit or categorical answers to your questions. Rather, it transforms your questions into API requests that are distributed across the various IT services officially administered by EPFL. Its purpose is solely to collect and recommend relevant references to content that you can explore to help you answer your questions.

Effective Prognostics and Health Management (PHM) relies on accurate prediction of the Remaining Useful Life (RUL). Data-driven RUL prediction techniques rely heavily on the representativeness of the available time-to-failure trajectories. Therefore, these ...

This dataset contains a collection of ultrafast ultrasound acquisitions from nine volunteers and the CIRS 054G phantom. For a comprehensive understanding of the dataset, please refer to the paper: Viñals, R.; Thiran, J.-P. A KL Divergence-Based Loss for In ...

Cell segmentation is a critical step for quantitative single-cell analysis in microscopy images. Existing cell segmentation methods are often tailored to specific modalities or require manual interventions to specify hyper-parameters in different experimen ...

The ability to reason, plan and solve highly abstract problems is a hallmark of human intelligence. Recent advancements in artificial intelligence, propelled by deep neural networks, have revolutionized disciplines like computer vision and natural language ...

The increasing complexity of transformer models in artificial intelligence expands their computational costs, memory usage, and energy consumption. Hardware acceleration tackles the ensuing challenges by designing processors and accelerators tailored for t ...

Current transformer-based skeletal action recognition models tend to focus on a limited set of joints and low-level motion patterns to predict action classes. This results in significant performance degradation under small skeleton perturbations or changin ...

In recent years, new emerging immersive imaging modalities, e.g. light fields, have been receiving growing attention, becoming increasingly widespread over the years. Light fields are often captured through multi-camera arrays or plenoptic cameras, with th ...

In this dissertation, we propose multiple methods to improve transfer learning for pretrained language models (PLMs). Broadly, transfer learning is a powerful technique in natural language processing, where a language model is first pre-trained on a data-r ...

Metal-organic frameworks (MOFs) are a class of crystalline porous materials that exhibit a vast chemical space owing to their tunable molecular building blocks with diverse topologies. An unlimited number of MOFs can, in principle, be synthesized. Machine ...

In this work, we propose an approach to aid in mapping small settlements, which are often misclassified by models trained on a large-scale context (global or regional). We leverage pre-trained land cover models and few-shot learning to enhance the detectio ...

Current machine learning models for vision are often highly specialized and limited to a single modality and task. In contrast, recent large language models exhibit a wide range of capabilities, hinting at a possibility for similarly versatile models in co ...

Vision-Language Pre-training (VLP) has advanced the performance of many visionlanguage tasks, such as image-text retrieval, visual entailment, and visual reasoning. The pre-training mostly utilizes lexical databases and image queries in English. Previous w ...

Most recent test-time adaptation methods focus on only classification tasks, use specialized network architectures, destroy model calibration or rely on lightweight information from the source domain. To tackle these issues, this paper proposes a novel Tes ...

Abstractive summarization has seen big improvements in recent years, mostly due to advances in neural language modeling, language model pretraining, and scaling models and datasets. While large language models generate summaries that are fluent, coherent, ...

Recent transformer language models achieve outstanding results in many natural language processing (NLP) tasks. However, their enormous size often makes them impractical on memory-constrained devices, requiring practitioners to compress them to smaller net ...

Models trained with counterfactually augmented data learn representations of the causal structure of tasks, enabling robust generalization. However, high-quality counterfactual data is scarce for most tasks and not easily generated at scale. When crowdsour ...

The recent developments of deep learning cover a wide variety of tasks such as image classification, text translation, playing go, and folding proteins. All these successful methods depend on a gradient-based learning algorithm to train a model on massive ...

The recent rise in both popularity and performance of large language models has garnered considerable interest regarding their applicability to education. Technologies like ChatGPT, which can engage in human-like dialog, have already disrupted educational ...

The recent advance of large language models (LLMs) demonstrates that these large-scale foundation models achieve remarkable capabilities across a wide range of language tasks and domains. The success of the statistical learning approach challenges our unde ...

Author summaryIn recent years, the application of deep learning represented a breakthrough in the mass spectrometry (MS) field by improving the assignment of the correct sequence of amino acids from observable MS spectra without prior knowledge, also known ...

On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselines

Chat with Graph Search

Domain adaptation via alignment of operation profile for Remaining Useful Lifetime prediction

A KL Divergence-Based Loss for In Vivo Ultrafast Ultrasound Image Enhancement with Deep Learning: Dataset (6/6)

The multimodality cell segmentation challenge: toward universal solutions

Infusing structured knowledge priors in neural models for sample-efficient symbolic reasoning

Accelerator-driven Data Arrangement to Minimize Transformers Run-time on Multi-core Architectures

MaskCLR: Attention-Guided Contrastive Learning for Robust Action Representation Learning

Towards learning-based denoising of light fields

Improving Generalization of Pretrained Language Models

A multi-modal pre-training transformer for universal transfer learning in metal-organic frameworks

Detection of Settlements in Tanzania and Mozambique by Many Regional Few-Shot Models

4M: Massively Multimodal Masked Modeling

Stop Pre-Training: Adapt Visual-Language Models to Unseen Languages

TeSLA: Test-Time Self-Learning With Automatic Adversarial Augmentation

Interpretable Representation Learning and Evaluation for Abstractive Summarization

Revisiting Offline Compression: Going Beyond Factorization-based Methods for Transformer Language Models

DISCO: Distilling Counterfactuals with Large Language Models

Transformer Models for Vision

Prompting Large Language Models to Power Educational Chatbots

Monotonicity Reasoning in the Age of Neural Foundation Models

Multienzyme deep learning models improve peptide de novo sequencing by mass spectrometry proteomics