Mediaspace scheduled maintenance: Aug 25, 2026 07:00 - 12:00 AM. During this time, videos will be temporarily unavailable. Check status updates.

Lecture

Bellman Equation: Value Consistency and Optimal Actions

In course

CS-456: Deep reinforcement learning

This course provides an overview and introduces modern methods for reinforcement learning (RL.) The course starts with the fundamentals of RL, such as Q-learning, and delves into commonly used approaches, like PPO and DQN. The course will introduce student ...

Description

This lecture introduces the Bellman equation, which connects Q-values of state-action pairs with future rewards. It covers the importance of the discount factor, the concept of total expected discounted reward, and the value consistency of neighboring states. The instructor explains how the Bellman equation is used to determine optimal actions and the implications of different policies on the equation's formulation.

Instructors (2)

Wulfram Gerstner

Wulfram Gerstner is Director of the Laboratory of Computational Neuroscience LCN at the EPFL. His research in computational neuroscience concentrates on models of spiking neurons and spike-timing dependent plasticity, on the problem of neuronal coding in single neurons and populations, as well as on the link between biologically plausible learning rules and behavioral manifestations of learning. He teaches courses for Physicists, Computer Scientists, Mathematicians, and Life Scientists at the EPFL. After studies of Physics in Tübingen and at the Ludwig-Maximilians-University Munich (Master 1989), Wulfram Gerstner spent a year as a visiting researcher in Berkeley. He received his PhD in theoretical physics from the Technical University Munich in 1993 with a thesis on associative memory and dynamics in networks of spiking neurons. After short postdoctoral stays at Brandeis University and the Technical University of Munich, he joined the EPFL in 1996 as assistant professor. Promoted to Associate Professor with tenure in February 2001, he is since August 2006 a full professor with double appointment in the School of Computer and Communication Sciences and the School of Life Sciences. Wulfram Gerstner has been invited speaker at numerous international conferences and workshops. He has served on the editorial board of the Journal of Neuroscience, Network: Computation in Neural Systems', Journal of Computational Neuroscience', and `Science'.

Caglar Gulcehre

Official source

Related concepts (7)

Bellman equation

A Bellman equation, named after Richard E. Bellman, is a necessary condition for optimality associated with the mathematical optimization method known as dynamic programming. It writes the "value" of a decision problem at a certain point in time in terms of the payoff from some initial choices and the "value" of the remaining decision problem that results from those initial choices. This breaks a dynamic optimization problem into a sequence of simpler subproblems, as Bellman's “principle of optimality" prescribes.

Discounting

In finance, discounting is a mechanism in which a debtor obtains the right to delay payments to a creditor, for a defined period of time, in exchange for a charge or fee. Essentially, the party that owes money in the present purchases the right to delay the payment until some future date. This transaction is based on the fact that most people prefer current interest to delayed interest because of mortality effects, impatience effects, and salience effects.

Discounted cash flow

The discounted cash flow (DCF) analysis, in finance, is a method used to value a security, project, company, or asset, that incorporates the time value of money. Discounted cash flow analysis is widely used in investment finance, real estate development, corporate financial management, and patent valuation. Used in industry as early as the 1700s or 1800s, it was widely discussed in financial economics in the 1960s, and U.S. courts began employing the concept in the 1980s and 1990s.

Valuation using discounted cash flows

Valuation using discounted cash flows (DCF valuation) is a method of estimating the current value of a company based on projected future cash flows adjusted for the time value of money. The cash flows are made up of those within the “explicit” forecast period, together with a continuing or terminal value that represents the cash flow stream after the forecast period. In several contexts, DCF valuation is referred to as the "income approach".

Optimal stopping

In mathematics, the theory of optimal stopping or early stopping is concerned with the problem of choosing a time to take a particular action, in order to maximise an expected reward or minimise an expected cost. Optimal stopping problems can be found in areas of statistics, economics, and mathematical finance (related to the pricing of American options). A key example of an optimal stopping problem is the secretary problem. Optimal stopping problems can often be written in the form of a Bellman equation, and are therefore often solved using dynamic programming.

Related lectures (23)

Policy Iteration and Linear Programming in MDPs

Discusses policy iteration and linear programming methods for solving Markov Decision Processes.

Markov Decision Processes: Dynamic Programming Techniques

Discusses Markov Decision Processes and dynamic programming techniques for solving optimal policies in various scenarios.

Real Estate Economics MOOC: Soil and Real Estate Economics I

Covers inter-temporal calculations, landowner's arbitrage, and profitability evaluation in real estate economics.

Real Estate Investment Analysis

Explores real estate investment analysis, comparing rental costs with ownership and evaluating financial aspects for optimal purchase decisions.

Temporal difference learning