Reward is not enough: Advances in reinforcement learning fromdemonstrations and preferences

This thesis studies the problem of reinforcement learning in Markov Decision Processes (MDPs) whose reward function is unknown and therefore cannot be used to learn an optimal policy. The problem is motivated by practical applications in robotics and in the training of large language models (LLMs), where it is not straightforward to design a reward function that induces the desired behavior. As an alternative, this thesis studies policy learning in MDPs from (i) demonstrations collected from an expert policy or (ii) preferences provided by an oracle capable of comparing pairs of actions. Such an oracle may, for example, be a user who clicks on the left or right response provided by ChatGPT or similar systems. For these problems, this thesis introduces new algorithms that improve, in certain aspects, upon the current state of the art. The first contribution initiates the study of learning from demonstrations in multi-agent systems. The thesis then proposes efficient algorithms for learning from demonstrations that do not include the expert policy's actions and do not require strong prior knowledge of the expert policy. This contribution is enabled by viewing learning from demonstrations through an alternative perspective that does not rely on classical supervised learning. We also demonstrate the impact of these techniques through applications in simulated robotics. The thesis concludes by proposing a novel algorithm for learning from preferences, for which we establish stronger guarantees than those available for the most commonly used algorithm for this purpose today: Direct Preference Optimization (DPO). Using this algorithm, we obtain improved results in the context of post-training LLMs.