Lecture

Eligibility Traces in Policy Gradient Algorithms

Graph Chat

Chat with Graph Search

Ask any question about EPFL courses, lectures, exercises, research, news, etc. or try the example questions below.

DISCLAIMER: The Graph Chatbot is not programmed to provide explicit or categorical answers to your questions. Rather, it transforms your questions into API requests that are distributed across the various IT services officially administered by EPFL. Its purpose is solely to collect and recommend relevant references to content that you can explore to help you answer your questions.

In course

CS-456: Deep reinforcement learning

This course provides an overview and introduces modern methods for reinforcement learning (RL.) The course starts with the fundamentals of RL, such as Q-learning, and delves into commonly used approaches, like PPO and DQN. The course will introduce student ...

Description

This lecture explores how eligibility traces emerge in policy gradient algorithms when optimizing return in a multi-step environment. The instructor sketches important mathematical steps, including the use of shadow variables and the update rule for parameters with eligibility trace. The concept of maximizing discounted return with policy gradient naturally leads to eligibility traces, making it easy to implement. The lecture also covers Actor-Critic with Eligibility Traces, emphasizing the estimation of state-value functions and policy parameterization. By updating eligibility traces while moving, the learning process becomes rapid and efficient.

Instructors (2)

Caglar Gulcehre

Teaching & PhD PhD Students Tim Ruben Davidson, Skander Moalla, Anja Surina, Xiuying Wei, Liangze Jiang, Mikhail Terekhov, Amin Mansouri, Justin Samuel Deschenaux, Yitao Xu Courses Training Large Language Models EE-628 This PhD-level course dives deep into the training of Large Language Models (LLMs), focusing on the complementary roles of datasets, pre-training and post training methodologies in shaping model performance and scalability.

Wulfram Gerstner

Wulfram Gerstner is Director of the Laboratory of Computational Neuroscience LCN at the EPFL. His research in computational neuroscience concentrates on models of spiking neurons and spike-timing dependent plasticity, on the problem of neuronal coding in single neurons and populations, as well as on the link between biologically plausible learning rules and behavioral manifestations of learning. He teaches courses for Physicists, Computer Scientists, Mathematicians, and Life Scientists at the EPFL. After studies of Physics in Tübingen and at the Ludwig-Maximilians-University Munich (Master 1989), Wulfram Gerstner spent a year as a visiting researcher in Berkeley. He received his PhD in theoretical physics from the Technical University Munich in 1993 with a thesis on associative memory and dynamics in networks of spiking neurons. After short postdoctoral stays at Brandeis University and the Technical University of Munich, he joined the EPFL in 1996 as assistant professor. Promoted to Associate Professor with tenure in February 2001, he is since August 2006 a full professor with double appointment in the School of Computer and Communication Sciences and the School of Life Sciences. Wulfram Gerstner has been invited speaker at numerous international conferences and workshops. He has served on the editorial board of the Journal of Neuroscience, Network: Computation in Neural Systems', Journal of Computational Neuroscience', and `Science'.

Official source

Related concepts (14)

Discounted cash flow

The discounted cash flow (DCF) analysis, in finance, is a method used to value a security, project, company, or asset, that incorporates the time value of money. Discounted cash flow analysis is widely used in investment finance, real estate development, corporate financial management, and patent valuation. Used in industry as early as the 1700s or 1800s, it was widely discussed in financial economics in the 1960s, and U.S. courts began employing the concept in the 1980s and 1990s.

Discounting

In finance, discounting is a mechanism in which a debtor obtains the right to delay payments to a creditor, for a defined period of time, in exchange for a charge or fee. Essentially, the party that owes money in the present purchases the right to delay the payment until some future date. This transaction is based on the fact that most people prefer current interest to delayed interest because of mortality effects, impatience effects, and salience effects.

Valuation using discounted cash flows

Valuation using discounted cash flows (DCF valuation) is a method of estimating the current value of a company based on projected future cash flows adjusted for the time value of money. The cash flows are made up of those within the “explicit” forecast period, together with a continuing or terminal value that represents the cash flow stream after the forecast period. In several contexts, DCF valuation is referred to as the "income approach".

Internal rate of return

Internal rate of return (IRR) is a method of quantifying the merits of a project or investment opportunity. The calculation is termed internal because it depends only on the cash flows of the investment being analyzed and excludes external factors, such as returns available elsewhere, the risk-free rate, inflation, the cost of capital, or financial risk. The method may be applied either ex-post or ex-ante. Applied ex-ante, the IRR is an estimate of a future annual rate of return.

Net present value

The net present value (NPV) or net present worth (NPW) applies to a series of cash flows occurring at different times. The present value of a cash flow depends on the interval of time between now and the cash flow. It also depends on the discount rate. NPV accounts for the time value of money. It provides a method for evaluating and comparing capital projects or financial products with cash flows spread over time, as in loans, investments, payouts from insurance contracts plus many other applications.

Related lectures (63)

Investment Decisions: Annuities and Discounting Costs

Covers investment decisions, focusing on annuities, discounting costs, and evaluating profitability through net present value and internal rates of return.

Principles of Finance: Stock Valuation and Investment Strategies

Explores stock valuation, risk-return relationship, and portfolio returns based on weights.

Principles of Finance: Risk and Return

Explores the principles of finance regarding risk and return in investment decisions.

Investment Choice: Analysis and Decisions

Explores investment analysis, including scrap values, discount rates, and internal rates of return.

Investment Valuation: Present and Future Value Concepts