Lecture

Modern NLP: Classical Language Models

Graph Chat

Chat with Graph Search

Ask any question about EPFL courses, lectures, exercises, research, news, etc. or try the example questions below.

DISCLAIMER: The Graph Chatbot is not programmed to provide explicit or categorical answers to your questions. Rather, it transforms your questions into API requests that are distributed across the various IT services officially administered by EPFL. Its purpose is solely to collect and recommend relevant references to content that you can explore to help you answer your questions.

Description

This lecture introduces language models, including count-based models and fixed-context neural models. It covers evaluating language models, smoothing techniques, and the impact of language models on downstream applications. The instructor explains the concept of language models as probabilistic models of token sequences and discusses the chain rule for computing joint probabilities. The lecture also delves into n-gram models, smoothing methods like Laplace smoothing and absolute discounting, and the challenges faced by n-gram models. Finally, it explores fixed-context neural language models, their advantages, and limitations compared to traditional n-gram models.

Official source

Related concepts (32)

Language model

A language model is a probabilistic model of a natural language that can generate probabilities of a series of words, based on text corpora in one or multiple languages it was trained on. Large language models, as their most advanced form, are a combination of feedforward neural networks and transformers. They have superseded recurrent neural network-based models, which had previously superseded the pure statistical models, such as word n-gram language model.

N-gram

n-gram is a series of n adjacent letters (including punctuation marks and blanks), syllables, or rarely whole words found in a language dataset; or adjacent phonemes extracted from a speech-recording dataset, or adjacent base pairs extracted from a genome. They are collected from a text or speech corpus. If Latin numerical prefixes are used, then n-gram of size 1 is called a "unigram", size 2 a "bigram" (or, less commonly, a "digram") etc. If, instead of the Latin ones, the English cardinal numbers are furtherly used, then they are called "four-gram", "five-gram", etc.

Cache language model

A cache language model is a type of statistical language model. These occur in the natural language processing subfield of computer science and assign probabilities to given sequences of words by means of a probability distribution. Statistical language models are key components of speech recognition systems and of many machine translation systems: they tell such systems which possible output word sequences are probable and which are improbable.

Artificial neural network

Artificial neural networks (ANNs, also shortened to neural networks (NNs) or neural nets) are a branch of machine learning models that are built using principles of neuronal organization discovered by connectionism in the biological neural networks constituting animal brains. An ANN is based on a collection of connected units or nodes called artificial neurons, which loosely model the neurons in a biological brain. Each connection, like the synapses in a biological brain, can transmit a signal to other neurons.

Models of neural computation

Models of neural computation are attempts to elucidate, in an abstract and mathematical fashion, the core principles that underlie information processing in biological nervous systems, or functional components thereof. This article aims to provide an overview of the most definitive models of neuro-biological computation as well as the tools commonly used to construct and analyze them.

Related lectures (64)

Language Models: Fixed-context and Recurrent Neural Networks

Discusses language models, focusing on fixed-context neural models and recurrent neural networks.

State Space Models: Expressivity of Transformers

Covers state space models and the expressivity of transformers in sequence copying tasks.

Probability and Statistics

Covers Simpson's paradox, probability distributions, and real-life examples in probability and statistics.

Conditional Density and Expectation

Explores conditional density, expectations, and independence of random variables with practical examples.

Elements of Statistics: Probability and Random Variables