Lecture

Risk Minimization from Adaptively Collected Data

Graph Chat

Chat with Graph Search

Ask any question about EPFL courses, lectures, exercises, research, news, etc. or try the example questions below.

DISCLAIMER: The Graph Chatbot is not programmed to provide explicit or categorical answers to your questions. Rather, it transforms your questions into API requests that are distributed across the various IT services officially administered by EPFL. Its purpose is solely to collect and recommend relevant references to content that you can explore to help you answer your questions.

Description

This lecture covers risk minimization from adaptively collected data with guarantees for policy learning, focusing on adaptive experiments, sequential observations, and the importance of exploration strategies. The instructor discusses the key concepts, such as E-greedy exploration, martingale difference sequence, and the theoretical foundations behind policy learning.

Official source

Related concepts (31)

Reinforcement learning

Reinforcement learning (RL) is an area of machine learning concerned with how intelligent agents ought to take actions in an environment in order to maximize the notion of cumulative reward. Reinforcement learning is one of three basic machine learning paradigms, alongside supervised learning and unsupervised learning. Reinforcement learning differs from supervised learning in not needing labelled input/output pairs to be presented, and in not needing sub-optimal actions to be explicitly corrected.

Sequential analysis

In statistics, sequential analysis or sequential hypothesis testing is statistical analysis where the sample size is not fixed in advance. Instead data are evaluated as they are collected, and further sampling is stopped in accordance with a pre-defined stopping rule as soon as significant results are observed. Thus a conclusion may sometimes be reached at a much earlier stage than would be possible with more classical hypothesis testing or estimation, at consequently lower financial and/or human cost.

Q-learning

Q-learning is a model-free reinforcement learning algorithm to learn the value of an action in a particular state. It does not require a model of the environment (hence "model-free"), and it can handle problems with stochastic transitions and rewards without requiring adaptations. For any finite Markov decision process (FMDP), Q-learning finds an optimal policy in the sense of maximizing the expected value of the total reward over any and all successive steps, starting from the current state.

Temporal difference learning

Temporal difference (TD) learning refers to a class of model-free reinforcement learning methods which learn by bootstrapping from the current estimate of the value function. These methods sample from the environment, like Monte Carlo methods, and perform updates based on current estimates, like dynamic programming methods. While Monte Carlo methods only adjust their estimates once the final outcome is known, TD methods adjust predictions to match later, more accurate, predictions about the future before the final outcome is known.

CUSUM

In statistical quality control, the CUsUM (or cumulative sum control chart) is a sequential analysis technique developed by E. S. Page of the University of Cambridge. It is typically used for monitoring change detection. CUSUM was announced in Biometrika, in 1954, a few years after the publication of Wald's sequential probability ratio test (SPRT). E. S. Page referred to a "quality number" , by which he meant a parameter of the probability distribution; for example, the mean.

Related lectures (43)

Reinforcement Learning: TD Learning and SARSA Variants

Discusses reinforcement learning, focusing on temporal difference learning and SARSA algorithm variations.

Reinforcement Learning: Q-Learning

Covers Q-Learning in reinforcement learning, exploring action values, policies, and the societal impact of algorithms.

Deep Learning Agents: Reinforcement Learning

Explores Deep Learning Agents in Reinforcement Learning, emphasizing neural network approximations and challenges in training multiagent systems.

Reinforcement Learning: Reward-based Learning

Explores artificial neural networks, reward information in the brain, animal conditioning, deep reinforcement learning, and a quiz on rewards.

TD Learning: Temporal Difference Learning