Lecture

Model-Free Prediction in Reinforcement Learning: Key Methods

Graph Chat

Chat with Graph Search

Ask any question about EPFL courses, lectures, exercises, research, news, etc. or try the example questions below.

DISCLAIMER: The Graph Chatbot is not programmed to provide explicit or categorical answers to your questions. Rather, it transforms your questions into API requests that are distributed across the various IT services officially administered by EPFL. Its purpose is solely to collect and recommend relevant references to content that you can explore to help you answer your questions.

In course

EE-568: Reinforcement learning

This course describes theory and methods for Reinforcement Learning (RL), which revolves around decision making under uncertainty. The course covers classic algorithms in RL as well as recent algorithms under the lens of contemporary optimization. ...

Description

This lecture introduces model-free prediction methods in reinforcement learning, focusing on estimating value functions without knowledge of transition dynamics. The instructor begins by contrasting model-based and model-free approaches, emphasizing the importance of learning from experience. The discussion covers two primary methods: Monte Carlo and Temporal Differences (TD). Monte Carlo methods estimate value functions by averaging returns from sampled trajectories, while TD methods update value estimates incrementally based on immediate rewards and subsequent estimates. The lecture also explores the stochastic gradient descent (SGD) algorithm as a means to optimize these estimates, highlighting the role of unbiased gradient estimators. The instructor explains the challenges of infinite horizon problems and how TD methods can address these issues. The session concludes with a summary of the differences between Monte Carlo and TD methods, including their biases and variances, and the implications for policy evaluation and improvement in reinforcement learning.

Instructor

Volkan Cevher

Volkan Cevher received the B.Sc. (valedictorian) in electrical engineering from Bilkent University in Ankara, Turkey, in 1999 and the Ph.D. in electrical and computer engineering from the Georgia Institute of Technology in Atlanta, GA in 2005. He was a Research Scientist with the University of Maryland, College Park from 2006-2007 and also with Rice University in Houston, TX, from 2008-2009. Currently, he is an Associate Professor at the Swiss Federal Institute of Technology Lausanne and a Faculty Fellow in the Electrical and Computer Engineering Department at Rice University. His research interests include machine learning, signal processing theory, optimization theory and methods, and information theory. Dr. Cevher is an ELLIS fellow and was the recipient of the Google Faculty Research award in 2018, the IEEE Signal Processing Society Best Paper Award in 2016, a Best Paper Award at CAMSAP in 2015, a Best Paper Award at SPARS in 2009, and an ERC CG in 2016 as well as an ERC StG in 2011.

Official source

Related concepts (32)

Control theory

Control theory is a field of control engineering and applied mathematics that deals with the control of dynamical systems in engineered processes and machines. The objective is to develop a model or algorithm governing the application of system inputs to drive the system to a desired state, while minimizing any delay, overshoot, or steady-state error and ensuring a level of control stability; often with the aim to achieve a degree of optimality. To do this, a controller with the requisite corrective behavior is required.

Event horizon

In astrophysics, an event horizon is a boundary beyond which events cannot affect an observer. Wolfgang Rindler coined the term in the 1950s. In 1784, John Michell proposed that gravity can be strong enough in the vicinity of massive compact objects that even light cannot escape. At that time, the Newtonian theory of gravitation and the so-called corpuscular theory of light were dominant. In these theories, if the escape velocity of the gravitational influence of a massive object exceeds the speed of light, then light originating inside or from it can escape temporarily but will return.

Perceptual control theory

Perceptual control theory (PCT) is a model of behavior based on the properties of negative feedback control loops. A control loop maintains a sensed variable at or near a reference value by means of the effects of its outputs upon that variable, as mediated by physical properties of the environment. In engineering control theory, reference values are set by a user outside the system. An example is a thermostat. In a living organism, reference values for controlled perceptual variables are endogenously maintained.

Online machine learning

In computer science, online machine learning is a method of machine learning in which data becomes available in a sequential order and is used to update the best predictor for future data at each step, as opposed to batch learning techniques which generate the best predictor by learning on the entire training data set at once. Online learning is a common technique used in areas of machine learning where it is computationally infeasible to train over the entire dataset, requiring the need of out-of-core algorithms.

Policy analysis

Policy analysis or public policy analysis is a technique used in the public administration sub-field of political science to enable civil servants, nonprofit organizations, and others to examine and evaluate the available options to implement the goals of laws and elected officials. People who regularly use policy analysis skills and techniques on the job, particularly those who use it as a major part of their job duties are generally known by the title Policy Analyst.

Related lectures (64)

Reinforcement Learning: Q-Learning

Covers Q-Learning in reinforcement learning, exploring action values, policies, and the societal impact of algorithms.

Policy Iteration and Linear Programming in MDPs

Discusses policy iteration and linear programming methods for solving Markov Decision Processes.

Reinforcement Learning: Policy Gradient and Actor-Critic Methods

Provides an overview of reinforcement learning, focusing on policy gradient and actor-critic methods for deep artificial neural networks.

Introduction to Reinforcement Learning: Key Concepts and Applications

Introduces reinforcement learning, covering its definitions, applications, and theoretical foundations, while outlining the course structure and objectives.

Policy Gradient Methods: Reinforcement Learning Techniques