Lecture

Efficient Methods in Natural Language Processing

Graph Chat

Chat with Graph Search

Ask any question about EPFL courses, lectures, exercises, research, news, etc. or try the example questions below.

DISCLAIMER: The Graph Chatbot is not programmed to provide explicit or categorical answers to your questions. Rather, it transforms your questions into API requests that are distributed across the various IT services officially administered by EPFL. Its purpose is solely to collect and recommend relevant references to content that you can explore to help you answer your questions.

Description

This lecture covers the latest advancements in natural language processing, focusing on the efficiency of training large language models. Topics include the staggering costs of training state-of-the-art models, the environmental impact of AI, and the challenges of scaling language models. The lecture also discusses the evolution of model sizes over the years and the trade-offs between model accuracy and efficiency.

Official source

Related concepts (20)

Language model

A language model is a probabilistic model of a natural language that can generate probabilities of a series of words, based on text corpora in one or multiple languages it was trained on. Large language models, as their most advanced form, are a combination of feedforward neural networks and transformers. They have superseded recurrent neural network-based models, which had previously superseded the pure statistical models, such as word n-gram language model.

Natural language processing

Natural language processing (NLP) is an interdisciplinary subfield of linguistics and computer science. It is primarily concerned with processing natural language datasets, such as text corpora or speech corpora, using either rule-based or probabilistic (i.e. statistical and, most recently, neural network-based) machine learning approaches. The goal is a computer capable of "understanding" the contents of documents, including the contextual nuances of the language within them.

Machine ethics

Machine ethics (or machine morality, computational morality, or computational ethics) is a part of the ethics of artificial intelligence concerned with adding or ensuring moral behaviors of man-made machines that use artificial intelligence, otherwise known as artificial intelligent agents. Machine ethics differs from other ethical fields related to engineering and technology. Machine ethics should not be confused with computer ethics, which focuses on human use of computers.

Robot ethics

Robot ethics, sometimes known as "roboethics", concerns ethical problems that occur with robots, such as whether robots pose a threat to humans in the long or short run, whether some uses of robots are problematic (such as in healthcare or as 'killer robots' in war), and how robots should be designed such that they act 'ethically' (this last concern is also called machine ethics). Alternatively, roboethics refers specifically to the ethics of human behavior towards robots, as robots become increasingly advanced.

Ethics of artificial intelligence

The ethics of artificial intelligence is the branch of the ethics of technology specific to artificially intelligent systems. It is sometimes divided into a concern with the moral behavior of humans as they design, make, use and treat artificially intelligent systems, and a concern with the behavior of machines, in machine ethics. Robot ethics The term "robot ethics" (sometimes "roboethics") refers to the morality of how humans design, construct, use and treat robots. Robot ethics intersect with the ethics of AI.

Related lectures (32)

Ethics in Natural Language Processing: Addressing Bias and Misinformation

Covers ethical considerations in NLP, focusing on bias, toxicity, and misinformation.

Prompting and Alignment

Explores prompting, alignment, and the capabilities of large language models for natural language processing tasks.

Ethical Considerations in Natural Language Processing

Explores ethical challenges in NLP systems, including biases, toxicity, privacy, and disinformation.

Pretraining Sequence-to-Sequence Models: BART and T5

Covers the pretraining of sequence-to-sequence models, focusing on BART and T5 architectures.

Data-Driven Insights: NLP and AI Applications