Lecture

Modern NLP and Ethics in NLP

Graph Chat

Chat with Graph Search

Ask any question about EPFL courses, lectures, exercises, research, news, etc. or try the example questions below.

DISCLAIMER: The Graph Chatbot is not programmed to provide explicit or categorical answers to your questions. Rather, it transforms your questions into API requests that are distributed across the various IT services officially administered by EPFL. Its purpose is solely to collect and recommend relevant references to content that you can explore to help you answer your questions.

Description

This lecture explores the advancements in Natural Language Processing (NLP), including pretraining and scale, as well as the new challenges such as prompting, knowledge & reasoning, and robustness. It delves into the ethical considerations surrounding large language models, the implicit encoding of biases, and the potential harms like toxicity, disinformation, and privacy breaches.

This video is available exclusively on Mediaspace for a restricted audience. Please log in to MediaSpace to access it if you have the necessary permissions.

Watch on Mediaspace

Official source

Related concepts (32)

Language model

A language model is a probabilistic model of a natural language that can generate probabilities of a series of words, based on text corpora in one or multiple languages it was trained on. Large language models, as their most advanced form, are a combination of feedforward neural networks and transformers. They have superseded recurrent neural network-based models, which had previously superseded the pure statistical models, such as word n-gram language model.

Natural language processing

Natural language processing (NLP) is an interdisciplinary subfield of linguistics and computer science. It is primarily concerned with processing natural language datasets, such as text corpora or speech corpora, using either rule-based or probabilistic (i.e. statistical and, most recently, neural network-based) machine learning approaches. The goal is a computer capable of "understanding" the contents of documents, including the contextual nuances of the language within them.

Large language model

A large language model (LLM) is a language model characterized by its large size. Their size is enabled by AI accelerators, which are able to process vast amounts of text data, mostly scraped from the Internet. The artificial neural networks which are built can contain from tens of millions and up to billions of weights and are (pre-)trained using self-supervised learning and semi-supervised learning. Transformer architecture contributed to faster training.

Reason

Reason is the capacity of applying logic consciously by drawing conclusions from new or existing information, with the aim of seeking the truth. It is closely associated with such characteristically human activities as philosophy, science, language, mathematics, and art, and is normally considered to be a distinguishing ability possessed by humans. Reason is sometimes referred to as rationality. Reasoning is associated with the acts of thinking and cognition, and involves the use of one's intellect.

Generative pre-trained transformer

Generative pre-trained transformers (GPT) are a type of large language model (LLM) and a prominent framework for generative artificial intelligence. The first GPT was introduced in 2018 by OpenAI. GPT models are artificial neural networks that are based on the transformer architecture, pre-trained on large data sets of unlabelled text, and able to generate novel human-like content. As of 2023, most LLMs have these characteristics and are sometimes referred to broadly as GPTs.

Related lectures (64)

Neuro-symbolic Representations: Commonsense Knowledge & Reasoning

Delves into neuro-symbolic representations for commonsense knowledge and reasoning in natural language processing applications.

Language Models: From Theory to Computation

Explores the mathematics of language models, covering architecture design, pre-training, and fine-tuning, emphasizing the importance of pre-training and fine-tuning for various tasks.

Neuro-symbolic Representations: Commonsense Knowledge & Reasoning

Explores neuro-symbolic representations for understanding commonsense knowledge and reasoning, emphasizing the challenges and limitations of deep learning in natural language processing.

The Printing Press: A Revolution in France

Explores the introduction of the printing press in France and its societal impact.

Vision-Language-Action Models: Training and Applications