Mediaspace scheduled maintenance: Aug 25, 2026 07:00 - 12:00 AM. During this time, videos will be temporarily unavailable. Check status updates.

Lecture

Decoding from Neural Models

Description

This lecture covers the concept of decoding from neural models in modern NLP, focusing on encoder-decoder models and the decoding algorithm. It explains the main idea of decoding, issues with argmax decoding, and introduces beam search as an alternative. The instructor also discusses the effect of beam size and what to expect in the upcoming lectures.

Official source

Related concepts (24)

Transformer (machine learning model)

A transformer is a deep learning architecture that relies on the parallel multi-head attention mechanism. The modern transformer was proposed in the 2017 paper titled 'Attention Is All You Need' by Ashish Vaswani et al., Google Brain team. It is notable for requiring less training time than previous recurrent neural architectures, such as long short-term memory (LSTM), and its later variation has been prevalently adopted for training large language models on large (language) datasets, such as the Wikipedia corpus and Common Crawl, by virtue of the parallelized processing of input sequence.

Large language model

A large language model (LLM) is a language model characterized by its large size. Their size is enabled by AI accelerators, which are able to process vast amounts of text data, mostly scraped from the Internet. The artificial neural networks which are built can contain from tens of millions and up to billions of weights and are (pre-)trained using self-supervised learning and semi-supervised learning. Transformer architecture contributed to faster training.

Attention (machine learning)

Machine learning-based attention is a mechanism mimicking cognitive attention. It calculates "soft" weights for each word, more precisely for its embedding, in the context window. It can do it either in parallel (such as in transformers) or sequentially (such as recursive neural networks). "Soft" weights can change during each runtime, in contrast to "hard" weights, which are (pre-)trained and fine-tuned and remain frozen afterwards. Multiple attention heads are used in transformer-based large language models.

Generative pre-trained transformer

Generative pre-trained transformers (GPT) are a type of large language model (LLM) and a prominent framework for generative artificial intelligence. The first GPT was introduced in 2018 by OpenAI. GPT models are artificial neural networks that are based on the transformer architecture, pre-trained on large data sets of unlabelled text, and able to generate novel human-like content. As of 2023, most LLMs have these characteristics and are sometimes referred to broadly as GPTs.

Language model

A language model is a probabilistic model of a natural language that can generate probabilities of a series of words, based on text corpora in one or multiple languages it was trained on. Large language models, as their most advanced form, are a combination of feedforward neural networks and transformers. They have superseded recurrent neural network-based models, which had previously superseded the pure statistical models, such as word n-gram language model.

Related lectures (33)

Pretraining: Transformers & Models

Explores pretraining models like BERT, T5, and GPT, discussing their training objectives and applications in natural language processing.

Transformers: Pretraining and Decoding Techniques

Covers advanced transformer concepts, focusing on pretraining and decoding techniques in NLP.

Contextual Representations: ELMO and BERT Overview

Covers contextual representations in NLP, focusing on ELMO and BERT architectures and their applications in various tasks.

Sequence to Sequence Models: Overview and Applications

Covers sequence to sequence models, their architecture, applications, and the role of attention mechanisms in improving performance.

Coreference Resolution