Lecture

Relation of SARSA and Bellman equation

Graph Chat

Chat with Graph Search

Ask any question about EPFL courses, lectures, exercises, research, news, etc. or try the example questions below.

DISCLAIMER: The Graph Chatbot is not programmed to provide explicit or categorical answers to your questions. Rather, it transforms your questions into API requests that are distributed across the various IT services officially administered by EPFL. Its purpose is solely to collect and recommend relevant references to content that you can explore to help you answer your questions.

In course

CS-456: Deep reinforcement learning

This course provides an overview and introduces modern methods for reinforcement learning (RL.) The course starts with the fundamentals of RL, such as Q-learning, and delves into commonly used approaches, like PPO and DQN. The course will introduce student ...

Description

This lecture presents a sketch of a proof regarding the relation between fluctuating Q-values in SARSA and the Bellman equation. The instructor explains the assumptions, expectations, and updates in the SARSA algorithm, emphasizing the convergence to the Bellman equation through expectations of Q-values. The proof involves modifications to the update rule, expectations of rewards, and policies, highlighting the impact of a small learning rate on the approximation of policy constancy. By considering the policy as constant during statistical averaging, the expectation values of Q-hat SA are derived, showing a connection to the Bellman equation.

Instructors (2)

Wulfram Gerstner

Wulfram Gerstner is Director of the Laboratory of Computational Neuroscience LCN at the EPFL. His research in computational neuroscience concentrates on models of spiking neurons and spike-timing dependent plasticity, on the problem of neuronal coding in single neurons and populations, as well as on the link between biologically plausible learning rules and behavioral manifestations of learning. He teaches courses for Physicists, Computer Scientists, Mathematicians, and Life Scientists at the EPFL. After studies of Physics in Tübingen and at the Ludwig-Maximilians-University Munich (Master 1989), Wulfram Gerstner spent a year as a visiting researcher in Berkeley. He received his PhD in theoretical physics from the Technical University Munich in 1993 with a thesis on associative memory and dynamics in networks of spiking neurons. After short postdoctoral stays at Brandeis University and the Technical University of Munich, he joined the EPFL in 1996 as assistant professor. Promoted to Associate Professor with tenure in February 2001, he is since August 2006 a full professor with double appointment in the School of Computer and Communication Sciences and the School of Life Sciences. Wulfram Gerstner has been invited speaker at numerous international conferences and workshops. He has served on the editorial board of the Journal of Neuroscience, Network: Computation in Neural Systems', Journal of Computational Neuroscience', and `Science'.

Caglar Gulcehre

Teaching & PhD PhD Students Tim Ruben Davidson, Skander Moalla, Anja Surina, Xiuying Wei, Liangze Jiang, Mikhail Terekhov, Amin Mansouri, Justin Samuel Deschenaux, Yitao Xu Courses Training Large Language Models EE-628 This PhD-level course dives deep into the training of Large Language Models (LLMs), focusing on the complementary roles of datasets, pre-training and post training methodologies in shaping model performance and scalability.

Official source

Related concepts (11)

Policy

Policy is a deliberate system of guidelines to guide decisions and achieve rational outcomes. A policy is a statement of intent and is implemented as a procedure or protocol. Policies are generally adopted by a governance body within an organization. Policies can assist in both subjective and objective decision making. Policies used in subjective decision-making usually assist senior management with decisions that must be based on the relative merits of a number of factors, and as a result, are often hard to test objectively, e.

Policy analysis

Policy analysis or public policy analysis is a technique used in the public administration sub-field of political science to enable civil servants, nonprofit organizations, and others to examine and evaluate the available options to implement the goals of laws and elected officials. People who regularly use policy analysis skills and techniques on the job, particularly those who use it as a major part of their job duties are generally known by the title Policy Analyst.

Public policy

Public policy is an institutionalized proposal or a decided set of elements like laws, regulations, guidelines, and actions to solve or address relevant and real-world problems, guided by a conception and often implemented by programs. The implementation of public policy is known as public administration. Public policy can be considered to be the sum of a government's direct and indirect activities and has been conceptualized in a variety of ways. They are created and/or enacted on behalf of the public typically by a government.

Q-learning

Q-learning is a model-free reinforcement learning algorithm to learn the value of an action in a particular state. It does not require a model of the environment (hence "model-free"), and it can handle problems with stochastic transitions and rewards without requiring adaptations. For any finite Markov decision process (FMDP), Q-learning finds an optimal policy in the sense of maximizing the expected value of the total reward over any and all successive steps, starting from the current state.

Policy studies

Policy studies is a subdiscipline of political science that includes the analysis of the process of policymaking (the policy process) and the contents of policy (policy analysis). Policy analysis includes substantive area research (such as health or education policy), program evaluation and impact studies, and policy design. It "involves systematically studying the nature, causes, and effects of alternative public policies, with particular emphasis on determining the policies that will achieve given goals.

Related lectures (36)

Multilayer Neural Networks: Deep Learning

Covers the fundamentals of multilayer neural networks and deep learning.

Reinforcement Learning: TD Learning and SARSA Variants

Discusses reinforcement learning, focusing on temporal difference learning and SARSA algorithm variations.

Policy Iteration and Linear Programming in MDPs

Discusses policy iteration and linear programming methods for solving Markov Decision Processes.

Mini-Batches in On- and Off-Policy Deep Reinforcement Learning

Explains the significance of mini-batches in Deep Reinforcement Learning and the differences between on-policy and off-policy methods.

Adaptive Gradient Methods: Part 1