Manipulation
All Publication Topics
Autonomous Driving
Autonomous Exploration
Benchmarks
Compression
Computer Vision
First-Person Video
Forecasting
Generative Models
Imitation Learning
Intrinsic Control
Machine Learning
Manipulation
Model-Based Control
Offroad Navigation
Paper Award
Reinforcement Learning
Reward Learning
Robotics
Social Navigation
Residual Reward Models for Preference-based Reinforcement Learning.
arXiv:2507.00611, (arXiv), 2025.
arXiv:2507.00611, (arXiv), 2025.
Preference-based Reinforcement Learning (PbRL) provides a way to learn high-performance policies in environments where the reward signal is hard to specify, avoiding heuristic and time-consuming reward design. However, PbRL can suffer from slow convergence speed since it requires training in a reward model. Prior work has proposed learning a reward model from demonstrations and fine-tuning it using preferences. However, when the model is a neural network, using different loss functions for pre-training and fine-tuning can pose challenges to reliable optimization. In this paper, we propose a method to effectively leverage prior knowledge with a Residual Reward Model (RRM). An RRM assumes that the true reward of the environment can be split into a sum of two parts: a prior reward and a learned reward. The prior reward is a term available before training, for example, a user’s ``best guess’’ reward function, or a reward function learned from inverse reinforcement learning (IRL), and the learned reward is trained with preferences. We introduce state-based and image-based versions of RRM and evaluate them on several tasks in the Meta-World environment suite. Experimental results show that our method substantially improves the performance of a common PbRL method. Our method achieves performance improvements for a variety of different types of prior rewards, including proxy rewards, a reward obtained from IRL, and even a negated version of the proxy reward. We also conduct experiments with a Franka Panda to show that our method leads to superior performance on a real robot. It significantly accelerates policy learning for different tasks, achieving success in fewer steps than the baseline. The videos are presented at https://sunlighted.github.io/RRM-web/.
@article{cao2025residualrewardmodelspreferencebased,
title={Residual Reward Models for Preference-based Reinforcement Learning},
author={Chenyang Cao and Miguel Rogel-García and Mohamed Nabail and Xueqian Wang and Nicholas Rhinehart},
year={2025},
eprint={2507.00611},
archivePrefix={arXiv},
primaryClass={cs.LG},
url={https://arxiv.org/abs/2507.00611},
}
Parrot: Data-driven behavioral priors for reinforcement learning.
International Conference on Learning Representations, (ICLR), 2020.
International Conference on Learning Representations, (ICLR), 2020.
Reinforcement learning provides a general framework for flexible decision making and control, but requires extensive data collection for each new task that an agent needs to learn. In other machine learning fields, such as natural language processing or computer vision, pre-training on large, previously collected datasets to bootstrap learning for new tasks has emerged as a powerful paradigm to reduce data requirements when learning a new task. In this paper, we ask the following question: how can we enable similarly useful pre-training for RL agents? We propose a method for pre-training behavioral priors that can capture complex input-output relationships observed in successful trials from a wide range of previously seen tasks, and we show how this learned prior can be used for rapidly learning new tasks without impeding the RL agent’s ability to try out novel behaviors. We demonstrate the effectiveness of our approach in challenging robotic manipulation domains involving image observations and sparse reward functions, where our method outperforms prior works by a substantial margin.
@article{singh2020parrot,
author = {Singh, Avi and Liu, Huihan and Zhou, Gaoyue and Yu, Albert and Rhinehart, Nicholas and Levine, Sergey},
journal = {arXiv preprint arXiv:2011.10024},
title = {Parrot: Data-driven behavioral priors for reinforcement learning},
year = {2020}
}