Lecture

Decoding Sequence Models: Insights and Beam Search

Graph Chat

Chat with Graph Search

Ask any question about EPFL courses, lectures, exercises, research, news, etc. or try the example questions below.

DISCLAIMER: The Graph Chatbot is not programmed to provide explicit or categorical answers to your questions. Rather, it transforms your questions into API requests that are distributed across the various IT services officially administered by EPFL. Its purpose is solely to collect and recommend relevant references to content that you can explore to help you answer your questions.

Description

This lecture covers two new insights into beam search, focusing on decoding sequence models in natural language processing. It explores the challenges of decoding neural language generators and the concept of beam search as an iterative subset optimization. The instructor discusses the importance of considering task-specific output spaces and the exponential complexity of exact decoding. Additionally, the lecture delves into the best-first beam search algorithm, which prioritizes hypotheses based on model scores. The presentation concludes with a discussion on the cognitive motivation behind beam search and the concept of uniform information density regularization in generated text.

Official source

Related concepts (32)

Time complexity

In computer science, the time complexity is the computational complexity that describes the amount of computer time it takes to run an algorithm. Time complexity is commonly estimated by counting the number of elementary operations performed by the algorithm, supposing that each elementary operation takes a fixed amount of time to perform. Thus, the amount of time taken and the number of elementary operations performed by the algorithm are taken to be related by a constant factor.

Computational complexity

In computer science, the computational complexity or simply complexity of an algorithm is the amount of resources required to run it. Particular focus is given to computation time (generally measured by the number of needed elementary operations) and memory storage requirements. The complexity of a problem is the complexity of the best algorithms that allow solving the problem. The study of the complexity of explicitly given algorithms is called analysis of algorithms, while the study of the complexity of problems is called computational complexity theory.

Randomized algorithm

A randomized algorithm is an algorithm that employs a degree of randomness as part of its logic or procedure. The algorithm typically uses uniformly random bits as an auxiliary input to guide its behavior, in the hope of achieving good performance in the "average case" over all possible choices of random determined by the random bits; thus either the running time, or the output (or both) are random variables.

Computational complexity theory

In theoretical computer science and mathematics, computational complexity theory focuses on classifying computational problems according to their resource usage, and relating these classes to each other. A computational problem is a task solved by a computer. A computation problem is solvable by mechanical application of mathematical steps, such as an algorithm. A problem is regarded as inherently difficult if its solution requires significant resources, whatever the algorithm used.

Best-first search

Best-first search is a class of search algorithms, which explores a graph by expanding the most promising node chosen according to a specified rule. Judea Pearl described the best-first search as estimating the promise of node n by a "heuristic evaluation function which, in general, may depend on the description of n, the description of the goal, the information gathered by the search up to that point, and most importantly, on any extra knowledge about the problem domain.

Related lectures (64)

Dijkstra's Algorithm: All-Pairs

Covers Dijkstra's algorithm and its application to the all-pairs shortest path problem.

Algorithmic Complexity: Travel Time Analysis

Covers algorithmic complexity and travel time analysis, focusing on measuring the time taken by algorithms and evaluating their performance.

Solving Parity Games in Practice

Explores practical aspects of solving parity games, including winning strategies, algorithms, complexity, determinism, and heuristic approaches.

Search Algorithms: Abductive Reasoning

Covers search algorithms, focusing on abductive reasoning and heuristic search strategies.

Complexity & Induction: Algorithms & Proofs