Lecture

Reinforcement Learning: Eligibility Traces

Graph Chat

Chat with Graph Search

Ask any question about EPFL courses, lectures, exercises, research, news, etc. or try the example questions below.

DISCLAIMER: The Graph Chatbot is not programmed to provide explicit or categorical answers to your questions. Rather, it transforms your questions into API requests that are distributed across the various IT services officially administered by EPFL. Its purpose is solely to collect and recommend relevant references to content that you can explore to help you answer your questions.

Description

This lecture covers the theory of Reinforcement Learning, the Exploration/Exploitation dilemma, Temporal Difference Learning, and Eligibility Traces, focusing on updating previous action values along the trajectory. The SARSA algorithm is presented, along with its initialization and update rules. Additional reading material is recommended for further understanding.

Official source

Related concepts (10)

Reinforcement learning

Reinforcement learning (RL) is an area of machine learning concerned with how intelligent agents ought to take actions in an environment in order to maximize the notion of cumulative reward. Reinforcement learning is one of three basic machine learning paradigms, alongside supervised learning and unsupervised learning. Reinforcement learning differs from supervised learning in not needing labelled input/output pairs to be presented, and in not needing sub-optimal actions to be explicitly corrected.

Temporal difference learning

Temporal difference (TD) learning refers to a class of model-free reinforcement learning methods which learn by bootstrapping from the current estimate of the value function. These methods sample from the environment, like Monte Carlo methods, and perform updates based on current estimates, like dynamic programming methods. While Monte Carlo methods only adjust their estimates once the final outcome is known, TD methods adjust predictions to match later, more accurate, predictions about the future before the final outcome is known.

Q-learning

Q-learning is a model-free reinforcement learning algorithm to learn the value of an action in a particular state. It does not require a model of the environment (hence "model-free"), and it can handle problems with stochastic transitions and rewards without requiring adaptations. For any finite Markov decision process (FMDP), Q-learning finds an optimal policy in the sense of maximizing the expected value of the total reward over any and all successive steps, starting from the current state.

Group action

In mathematics, a group action on a space is a group homomorphism of a given group into the group of transformations of the space. Similarly, a group action on a mathematical structure is a group homomorphism of a group into the automorphism group of the structure. It is said that the group acts on the space or structure. If a group acts on a structure, it will usually also act on objects built from that structure. For example, the group of Euclidean isometries acts on Euclidean space and also on the figures drawn in it.

Continuous symmetry

In mathematics, continuous symmetry is an intuitive idea corresponding to the concept of viewing some symmetries as motions, as opposed to discrete symmetry, e.g. reflection symmetry, which is invariant under a kind of flip from one state to another. However, a discrete symmetry can always be reinterpreted as a subset of some higher-dimensional continuous symmetry, e.g. reflection of a 2 dimensional object in 3 dimensional space can be achieved by continuously rotating that object 180 degrees across a non-parallel plane.

Related lectures (6)

Neural Network Approximation and Learning Log in to Mediaspace to watch this video

Delves into neural network approximation, supervised learning, challenges in high-dimensional learning, and deep learning experimental revolution.

Temporal difference learning

Covers Reinforcement Learning theory, Q-Learning, and multi-step horizons.

Policy Gradient Methods in Reinforcement Learning

Covers policy gradient methods in reinforcement learning, focusing on optimization techniques and practical applications like the cartpole problem.

Galoisian Coating and Totally Discontinuous Actions

Covers the concept of Galoisian coating and actions that are totally discontinuous.

Reinforcement Learning Fundamentals

Delves into the fundamentals of reinforcement learning, discussing states, actions, rewards, policies, and neural network applications.

Related concepts (10)

Reinforcement learning

Temporal difference learning

Q-learning

Group action

Continuous symmetry