Lecture

Model Finetuning for NLI

Graph Chat

Chat with Graph Search

Ask any question about EPFL courses, lectures, exercises, research, news, etc. or try the example questions below.

DISCLAIMER: The Graph Chatbot is not programmed to provide explicit or categorical answers to your questions. Rather, it transforms your questions into API requests that are distributed across the various IT services officially administered by EPFL. Its purpose is solely to collect and recommend relevant references to content that you can explore to help you answer your questions.

Description

This lecture covers the second assignment for the CS-552: Modern NLP course, focusing on transfer learning and data augmentation. Students will implement training and evaluation of a pre-trained language model (DistilBERT) for natural language inference (NLI) tasks. They will identify model shortcuts, perform word-pair pattern extraction, annotate new data, and explore data augmentation methods. The lecture provides detailed instructions and code snippets for each part of the assignment, including model finetuning, identifying model shortcuts, and data augmentation.

Official source

Related concepts (32)

Language model

A language model is a probabilistic model of a natural language that can generate probabilities of a series of words, based on text corpora in one or multiple languages it was trained on. Large language models, as their most advanced form, are a combination of feedforward neural networks and transformers. They have superseded recurrent neural network-based models, which had previously superseded the pure statistical models, such as word n-gram language model.

Natural-language understanding

Natural-language understanding (NLU) or natural-language interpretation (NLI) is a subtopic of natural-language processing in artificial intelligence that deals with machine reading comprehension. Natural-language understanding is considered an AI-hard problem. There is considerable commercial interest in the field because of its application to automated reasoning, machine translation, question answering, news-gathering, text categorization, voice-activation, archiving, and large-scale content analysis.

Text mining

Text mining, text data mining (TDM) or text analytics is the process of deriving high-quality information from text. It involves "the discovery by computer of new, previously unknown information, by automatically extracting information from different written resources." Written resources may include websites, books, emails, reviews, and articles. High-quality information is typically obtained by devising patterns and trends by means such as statistical pattern learning. According to Hotho et al.

Transformer (machine learning model)

A transformer is a deep learning architecture that relies on the parallel multi-head attention mechanism. The modern transformer was proposed in the 2017 paper titled 'Attention Is All You Need' by Ashish Vaswani et al., Google Brain team. It is notable for requiring less training time than previous recurrent neural architectures, such as long short-term memory (LSTM), and its later variation has been prevalently adopted for training large language models on large (language) datasets, such as the Wikipedia corpus and Common Crawl, by virtue of the parallelized processing of input sequence.

Large language model

A large language model (LLM) is a language model characterized by its large size. Their size is enabled by AI accelerators, which are able to process vast amounts of text data, mostly scraped from the Internet. The artificial neural networks which are built can contain from tens of millions and up to billions of weights and are (pre-)trained using self-supervised learning and semi-supervised learning. Transformer architecture contributed to faster training.

Related lectures (64)

Binary Sentiment Classifier Training

Covers the training of a binary sentiment classifier using an RNN.

Logistic Regression: Vegetation Prediction

Explores logistic regression for predicting vegetation proportions in the Amazon region through remote sensing data analysis.

Data Annotation: Collection and Biases in NLP

Addresses data collection, annotation processes, and biases in natural language processing.

Unsupervised Learning: Clustering & Dimensionality Reduction

Introduces unsupervised learning through clustering with K-means and dimensionality reduction using PCA, along with practical examples.

Neural Networks: Training and Activation