Mediaspace scheduled maintenance: Aug 25, 2026 07:00 - 12:00 AM. During this time, videos will be temporarily unavailable. Check status updates.

Lecture

Modern NLP: Data Collection, Annotation & Biases

Description

This lecture covers the process of data annotation in Natural Language Processing, including the effects of biases on fine-tuning models. It discusses transfer learning, pretraining, and fine-tuning models on different datasets. The instructor also presents benchmark performance evaluations and the challenges in natural language understanding.

Official source

Related concepts (27)

Natural language processing

Natural language processing (NLP) is an interdisciplinary subfield of linguistics and computer science. It is primarily concerned with processing natural language datasets, such as text corpora or speech corpora, using either rule-based or probabilistic (i.e. statistical and, most recently, neural network-based) machine learning approaches. The goal is a computer capable of "understanding" the contents of documents, including the contextual nuances of the language within them.

Language model

A language model is a probabilistic model of a natural language that can generate probabilities of a series of words, based on text corpora in one or multiple languages it was trained on. Large language models, as their most advanced form, are a combination of feedforward neural networks and transformers. They have superseded recurrent neural network-based models, which had previously superseded the pure statistical models, such as word n-gram language model.

Generative pre-trained transformer

Generative pre-trained transformers (GPT) are a type of large language model (LLM) and a prominent framework for generative artificial intelligence. The first GPT was introduced in 2018 by OpenAI. GPT models are artificial neural networks that are based on the transformer architecture, pre-trained on large data sets of unlabelled text, and able to generate novel human-like content. As of 2023, most LLMs have these characteristics and are sometimes referred to broadly as GPTs.

Natural-language understanding

Natural-language understanding (NLU) or natural-language interpretation (NLI) is a subtopic of natural-language processing in artificial intelligence that deals with machine reading comprehension. Natural-language understanding is considered an AI-hard problem. There is considerable commercial interest in the field because of its application to automated reasoning, machine translation, question answering, news-gathering, text categorization, voice-activation, archiving, and large-scale content analysis.

Transformer (machine learning model)

A transformer is a deep learning architecture that relies on the parallel multi-head attention mechanism. The modern transformer was proposed in the 2017 paper titled 'Attention Is All You Need' by Ashish Vaswani et al., Google Brain team. It is notable for requiring less training time than previous recurrent neural architectures, such as long short-term memory (LSTM), and its later variation has been prevalently adopted for training large language models on large (language) datasets, such as the Wikipedia corpus and Common Crawl, by virtue of the parallelized processing of input sequence.

Related lectures (32)

Pretraining Sequence-to-Sequence Models: BART and T5

Covers the pretraining of sequence-to-sequence models, focusing on BART and T5 architectures.

Transformers: Pretraining and Decoding Techniques

Covers advanced transformer concepts, focusing on pretraining and decoding techniques in NLP.

Data Annotation: Collection and Biases in NLP

Addresses data collection, annotation processes, and biases in natural language processing.

Contextual Representations: ELMO and BERT Overview

Covers contextual representations in NLP, focusing on ELMO and BERT architectures and their applications in various tasks.

Coreference Resolution