Mediaspace scheduled maintenance: Aug 25, 2026 07:00 - 12:00 AM. During this time, videos will be temporarily unavailable. Check status updates.

Lecture

Comparison n-step SARSA and eligibility traces

In course

CS-456: Deep reinforcement learning

This course provides an overview and introduces modern methods for reinforcement learning (RL.) The course starts with the fundamentals of RL, such as Q-learning, and delves into commonly used approaches, like PPO and DQN. The course will introduce student ...

Description

This lecture presents a quiz comparing the n-step SARSA algorithm with SARSA using eligibility traces. The quiz involves analyzing the Q values updates in both algorithms based on a graph traversal scenario with rewards and discount factors.

Instructors (2)

Wulfram Gerstner

Wulfram Gerstner is Director of the Laboratory of Computational Neuroscience LCN at the EPFL. His research in computational neuroscience concentrates on models of spiking neurons and spike-timing dependent plasticity, on the problem of neuronal coding in single neurons and populations, as well as on the link between biologically plausible learning rules and behavioral manifestations of learning. He teaches courses for Physicists, Computer Scientists, Mathematicians, and Life Scientists at the EPFL. After studies of Physics in Tübingen and at the Ludwig-Maximilians-University Munich (Master 1989), Wulfram Gerstner spent a year as a visiting researcher in Berkeley. He received his PhD in theoretical physics from the Technical University Munich in 1993 with a thesis on associative memory and dynamics in networks of spiking neurons. After short postdoctoral stays at Brandeis University and the Technical University of Munich, he joined the EPFL in 1996 as assistant professor. Promoted to Associate Professor with tenure in February 2001, he is since August 2006 a full professor with double appointment in the School of Computer and Communication Sciences and the School of Life Sciences. Wulfram Gerstner has been invited speaker at numerous international conferences and workshops. He has served on the editorial board of the Journal of Neuroscience, Network: Computation in Neural Systems', Journal of Computational Neuroscience', and `Science'.

Caglar Gulcehre

Official source

Related concepts (9)

Reinforcement learning

Reinforcement learning (RL) is an area of machine learning concerned with how intelligent agents ought to take actions in an environment in order to maximize the notion of cumulative reward. Reinforcement learning is one of three basic machine learning paradigms, alongside supervised learning and unsupervised learning. Reinforcement learning differs from supervised learning in not needing labelled input/output pairs to be presented, and in not needing sub-optimal actions to be explicitly corrected.

Q-learning

Q-learning is a model-free reinforcement learning algorithm to learn the value of an action in a particular state. It does not require a model of the environment (hence "model-free"), and it can handle problems with stochastic transitions and rewards without requiring adaptations. For any finite Markov decision process (FMDP), Q-learning finds an optimal policy in the sense of maximizing the expected value of the total reward over any and all successive steps, starting from the current state.

Discounting

In finance, discounting is a mechanism in which a debtor obtains the right to delay payments to a creditor, for a defined period of time, in exchange for a charge or fee. Essentially, the party that owes money in the present purchases the right to delay the payment until some future date. This transaction is based on the fact that most people prefer current interest to delayed interest because of mortality effects, impatience effects, and salience effects.

Discounted cash flow

The discounted cash flow (DCF) analysis, in finance, is a method used to value a security, project, company, or asset, that incorporates the time value of money. Discounted cash flow analysis is widely used in investment finance, real estate development, corporate financial management, and patent valuation. Used in industry as early as the 1700s or 1800s, it was widely discussed in financial economics in the 1960s, and U.S. courts began employing the concept in the 1980s and 1990s.

Temporal difference learning

Temporal difference (TD) learning refers to a class of model-free reinforcement learning methods which learn by bootstrapping from the current estimate of the value function. These methods sample from the environment, like Monte Carlo methods, and perform updates based on current estimates, like dynamic programming methods. While Monte Carlo methods only adjust their estimates once the final outcome is known, TD methods adjust predictions to match later, more accurate, predictions about the future before the final outcome is known.

Related lectures (29)

Temporal difference learning

Covers Reinforcement Learning theory, Q-Learning, and multi-step horizons.

Real Estate Economics MOOC: Soil and Real Estate Economics I

Covers inter-temporal calculations, landowner's arbitrage, and profitability evaluation in real estate economics.

Policy Gradient and Actor-Critic Methods: Eligibility Traces Explained

Discusses policy gradient and actor-critic methods, focusing on eligibility traces and their application in reinforcement learning tasks.

Markov Decision Processes: Dynamic Programming Techniques

Discusses Markov Decision Processes and dynamic programming techniques for solving optimal policies in various scenarios.

Reinforcement Learning: TD Learning and SARSA Variants

Discusses reinforcement learning, focusing on temporal difference learning and SARSA algorithm variations.

Related concepts (9)

Reinforcement learning

Q-learning

Discounting

Discounted cash flow

Temporal difference learning