Mediaspace scheduled maintenance: Aug 25, 2026 07:00 - 12:00 AM. During this time, videos will be temporarily unavailable. Check status updates.

Lecture

Temporal difference learning

Description

This lecture covers the theory of Reinforcement Learning, the Exploration/Exploitation dilemma, Temporal Difference Learning, Eligibility Traces, and strategies for Continuous State/Action Spaces. It also introduces the Q-Learning algorithm, optimal paths, and the Bellman equation for multi-step horizons.

Official source

Related concepts (23)

Reinforcement learning

Reinforcement learning (RL) is an area of machine learning concerned with how intelligent agents ought to take actions in an environment in order to maximize the notion of cumulative reward. Reinforcement learning is one of three basic machine learning paradigms, alongside supervised learning and unsupervised learning. Reinforcement learning differs from supervised learning in not needing labelled input/output pairs to be presented, and in not needing sub-optimal actions to be explicitly corrected.

Q-learning

Q-learning is a model-free reinforcement learning algorithm to learn the value of an action in a particular state. It does not require a model of the environment (hence "model-free"), and it can handle problems with stochastic transitions and rewards without requiring adaptations. For any finite Markov decision process (FMDP), Q-learning finds an optimal policy in the sense of maximizing the expected value of the total reward over any and all successive steps, starting from the current state.

Bellman equation

A Bellman equation, named after Richard E. Bellman, is a necessary condition for optimality associated with the mathematical optimization method known as dynamic programming. It writes the "value" of a decision problem at a certain point in time in terms of the payoff from some initial choices and the "value" of the remaining decision problem that results from those initial choices. This breaks a dynamic optimization problem into a sequence of simpler subproblems, as Bellman's “principle of optimality" prescribes.

Temporal difference learning

Temporal difference (TD) learning refers to a class of model-free reinforcement learning methods which learn by bootstrapping from the current estimate of the value function. These methods sample from the environment, like Monte Carlo methods, and perform updates based on current estimates, like dynamic programming methods. While Monte Carlo methods only adjust their estimates once the final outcome is known, TD methods adjust predictions to match later, more accurate, predictions about the future before the final outcome is known.

Discounted cash flow

The discounted cash flow (DCF) analysis, in finance, is a method used to value a security, project, company, or asset, that incorporates the time value of money. Discounted cash flow analysis is widely used in investment finance, real estate development, corporate financial management, and patent valuation. Used in industry as early as the 1700s or 1800s, it was widely discussed in financial economics in the 1960s, and U.S. courts began employing the concept in the 1980s and 1990s.

Related lectures (32)

Markov Decision Processes: Dynamic Programming Techniques

Discusses Markov Decision Processes and dynamic programming techniques for solving optimal policies in various scenarios.

Reinforcement Learning: One-step Horizon (Bandit Problems)

Covers Bandit Problems in Reinforcement Learning, focusing on one-step horizon games and Q-values.

Policy Iteration and Linear Programming in MDPs

Discusses policy iteration and linear programming methods for solving Markov Decision Processes.

Advanced Machine Learning: Discrete Reinforcement Learning

Introduces the basics of Reinforcement Learning, covering discrete states, actions, policies, value functions, MDPs, and optimal policies.

Reinforcement Learning: Eligibility Traces

Explores Reinforcement Learning, focusing on updating previous action values along the trajectory using the SARSA algorithm.

Related concepts (23)

Reinforcement learning

Q-learning

Bellman equation

Temporal difference learning

Discounted cash flow