Mediaspace scheduled maintenance: Aug 25, 2026 07:00 - 12:00 AM. During this time, videos will be temporarily unavailable. Check status updates.

Lecture

Principled Reinforcement Learning with Human Feedback

Description

This lecture explores a theoretical framework for Reinforcement Learning with Human Feedback (RLHF) that deals with ordinal data, focusing on the convergence of estimators under different models. It discusses the challenges faced when training a policy based on learned reward models and introduces a pessimistic MLE for improved performance. The analysis validates the success of existing RLHF algorithms and provides insights for algorithm design, unifying RLHF and max-entropy Inverse Reinforcement Learning. The lecture also covers the formulation of RLHF, the Plackett-Luce model, and the connection with Inverse RL, along with experiments comparing different estimators and policies.

Official source

Related concepts (32)

Maximum likelihood estimation

In statistics, maximum likelihood estimation (MLE) is a method of estimating the parameters of an assumed probability distribution, given some observed data. This is achieved by maximizing a likelihood function so that, under the assumed statistical model, the observed data is most probable. The point in the parameter space that maximizes the likelihood function is called the maximum likelihood estimate. The logic of maximum likelihood is both intuitive and flexible, and as such the method has become a dominant means of statistical inference.

Reinforcement learning

Reinforcement learning (RL) is an area of machine learning concerned with how intelligent agents ought to take actions in an environment in order to maximize the notion of cumulative reward. Reinforcement learning is one of three basic machine learning paradigms, alongside supervised learning and unsupervised learning. Reinforcement learning differs from supervised learning in not needing labelled input/output pairs to be presented, and in not needing sub-optimal actions to be explicitly corrected.

Deep reinforcement learning

Deep reinforcement learning (deep RL) is a subfield of machine learning that combines reinforcement learning (RL) and deep learning. RL considers the problem of a computational agent learning to make decisions by trial and error. Deep RL incorporates deep learning into the solution, allowing agents to make decisions from unstructured input data without manual engineering of the state space. Deep RL algorithms are able to take in very large inputs (e.g.

Likelihood function

In statistical inference, the likelihood function quantifies the plausibility of parameter values characterizing a statistical model in light of observed data. Its most typical usage is to compare possible parameter values (under a fixed set of observations and a particular model), where higher values of likelihood are preferred because they correspond to more probable parameter values.

Bias of an estimator

In statistics, the bias of an estimator (or bias function) is the difference between this estimator's expected value and the true value of the parameter being estimated. An estimator or decision rule with zero bias is called unbiased. In statistics, "bias" is an property of an estimator. Bias is a distinct concept from consistency: consistent estimators converge in probability to the true value of the parameter, but may be biased or unbiased; see bias versus consistency for more.

Related lectures (35)

Deep Learning: No Free Lunch Theorem and Inductive Bias

Covers the No Free Lunch Theorem and the role of inductive bias in deep learning and reinforcement learning.

Bio-Inspired Learning: Neural Networks, Genetic Algorithms

Explores bio-inspired learning with neural networks and genetic algorithms, covering structure, training, and practical applications.

Confidence Intervals: Definition and Estimation

Explains confidence intervals, parameter estimation methods, and the central limit theorem in statistical inference.

Estimation and Confidence Intervals

Explores bias, variance, and confidence intervals in parameter estimation using examples and distributions.

Perception: Data-Driven Approaches