Lecture

Reinforcement Learning: Exploration, Credit Assignment, TRPO, PPO

Graph Chat

Chat with Graph Search

Ask any question about EPFL courses, lectures, exercises, research, news, etc. or try the example questions below.

DISCLAIMER: The Graph Chatbot is not programmed to provide explicit or categorical answers to your questions. Rather, it transforms your questions into API requests that are distributed across the various IT services officially administered by EPFL. Its purpose is solely to collect and recommend relevant references to content that you can explore to help you answer your questions.

Description

This lecture covers the main problems in Reinforcement Learning (RL), focusing on Exploration/Exploitation and Credit Assignment. It discusses the challenges of optimizing policies, emphasizing the need for balancing exploration and exploitation. The instructor explains the concepts of TRPO and PPO for monotonic improvement in RL, highlighting the importance of preventing overfitting to sampled trajectories. Additionally, the limitations of RL are explored, including the safety concerns of exploration in non-simulated settings and the difficulty of credit assignment in sparse-reward scenarios. The lecture concludes with a discussion on the trade-offs of using simulators in RL training and the implications for real-world applications.

This video is available exclusively on Mediaspace for a restricted audience. Please log in to MediaSpace to access it if you have the necessary permissions.

Watch on Mediaspace

Official source

Related concepts (17)

Reinforcement learning

Reinforcement learning (RL) is an area of machine learning concerned with how intelligent agents ought to take actions in an environment in order to maximize the notion of cumulative reward. Reinforcement learning is one of three basic machine learning paradigms, alongside supervised learning and unsupervised learning. Reinforcement learning differs from supervised learning in not needing labelled input/output pairs to be presented, and in not needing sub-optimal actions to be explicitly corrected.

Simulation

A simulation is the imitation of the operation of a real-world process or system over time. Simulations require the use of models; the model represents the key characteristics or behaviors of the selected system or process, whereas the simulation represents the evolution of the model over time. Often, computers are used to execute the simulation. Simulation is used in many contexts, such as simulation of technology for performance tuning or optimizing, safety engineering, testing, training, education, and video games.

Driving simulator

Driving simulators are used for entertainment as well as in training of driver's education courses taught in educational institutions and private businesses. They are also used for research purposes in the area of human factors and medical research, to monitor driver behavior, performance, and attention and in the car industry to design and evaluate new vehicles or new advanced driver assistance systems. Driving simulators are being increasingly used for training drivers. Versions exist for cars, trucks, buses, etc.

Flight simulator

A flight simulator is a device that artificially re-creates aircraft flight and the environment in which it flies, for pilot training, design, or other purposes. It includes replicating the equations that govern how aircraft fly, how they react to applications of flight controls, the effects of other aircraft systems, and how the aircraft reacts to external factors such as air density, turbulence, wind shear, cloud, precipitation, etc.

Motion simulator

A motion simulator or motion platform is a mechanism that creates the feelings of being in a real motion environment. In a simulator, the movement is synchronised with a visual display of the outside world (OTW) scene. Motion platforms can provide movement in all of the six degrees of freedom (DOF) that can be experienced by an object that is free to move, such as an aircraft or spacecraft:. These are the three rotational degrees of freedom (roll, pitch, yaw) and three translational or linear degrees of freedom (surge, heave, sway).

Related lectures (54)

Intelligent Maintenance Systems

Discusses faults in intelligent maintenance, Denoising Sparse Wavelet Network, Koopman operator theory, and Graph Neural Networks.

Programming the Thymio Robot MOOC: The Thymio robot as a tool for discovering digital science

Delves into alternative programming methods for the Thymio robot and the integration of a Raspberry Pi for enhanced capabilities.

Generalization in Deep Learning

Explores generalization in deep learning, covering model complexity, implicit bias, and the double descent phenomenon.

Disaster Risk Reduction: Preparedness & Technology MOOC: A Resilient Future: Science and Technology for Disaster Risk Reduction

Delves into disaster risk reduction through preparedness, emphasizing coordination and technology for resilience.

Learning with Deep Neural Networks