Mediaspace scheduled maintenance: Aug 25, 2026 07:00 - 12:00 AM. During this time, videos will be temporarily unavailable. Check status updates.

Lecture

Contrastive losses: Word2Vec and Skip-gram

Description

This lecture introduces contrastive losses for representation learning, focusing on Word2Vec and Skip-gram models. It covers the training objectives, context windows, negative sampling, and Noise Contrastive Estimation. The lecture also explores InfoNCE/CPC and its applications in image and graph data, as well as related topics like Deep Metric Learning and Energy-based models.

Official source

Related concepts (32)

Machine learning

Machine learning (ML) is an umbrella term for solving problems for which development of algorithms by human programmers would be cost-prohibitive, and instead the problems are solved by helping machines 'discover' their 'own' algorithms, without needing to be explicitly told what to do by any human-developed algorithms. Recently, generative artificial neural networks have been able to surpass results of many previous approaches.

Feature learning

In machine learning, feature learning or representation learning is a set of techniques that allows a system to automatically discover the representations needed for feature detection or classification from raw data. This replaces manual feature engineering and allows a machine to both learn the features and use them to perform a specific task. Feature learning is motivated by the fact that machine learning tasks such as classification often require input that is mathematically and computationally convenient to process.

Self-supervised learning

Self-supervised learning (SSL) is a paradigm in machine learning for processing data of lower quality, rather than improving ultimate outcomes. Self-supervised learning more closely imitates the way humans learn to classify objects. The typical SSL method is based on an artificial neural network or other model such as a decision list. The model learns in two steps. First, the task is solved based on an auxiliary or pretext classification task using pseudo-labels which help to initialize the model parameters.

Word2vec

Word2vec is a technique for natural language processing (NLP) published in 2013. The word2vec algorithm uses a neural network model to learn word associations from a large corpus of text. Once trained, such a model can detect synonymous words or suggest additional words for a partial sentence. As the name implies, word2vec represents each distinct word with a particular list of numbers called a vector.

Deep learning

Deep learning is part of a broader family of machine learning methods, which is based on artificial neural networks with representation learning. The adjective "deep" in deep learning refers to the use of multiple layers in the network. Methods used can be either supervised, semi-supervised or unsupervised.

Related lectures (30)

Language Models: From Theory to Computation

Explores the mathematics of language models, covering architecture design, pre-training, and fine-tuning, emphasizing the importance of pre-training and fine-tuning for various tasks.

Self-supervised Learning for Autonomous Vehicles

Explores self-supervised learning for autonomous vehicles, deriving labels from data itself and discussing its applications and challenges.

Neural Word Embeddings: Learning Representations for Natural Language

Covers neural word embeddings and methods for learning word representations in natural language processing.

Deep Learning Modus Operandi

Explores the benefits of deeper networks in deep learning and the importance of over-parameterization and generalization.

Unsupervised Learning: Clustering