Intrinsic Control
All Publication Topics
Autonomous Driving
Autonomous Exploration
Benchmarks
Compression
Computer Vision
First-Person Video
Forecasting
Generative Models
Imitation Learning
Intrinsic Control
Machine Learning
Manipulation
Model-Based Control
Offroad Navigation
Paper Award
Reinforcement Learning
Reward Learning
Robotics
Social Navigation
Information is power: Intrinsic control via information capture.
Advances in Neural Information Processing Systems, (NeurIPS), 2021.
Advances in Neural Information Processing Systems, (NeurIPS), 2021.
Humans and animals explore their environment and acquire useful skills even in the absence of clear goals, exhibiting intrinsic motivation. The study of intrinsic motivation in artificial agents is concerned with the following question: what is a good general-purpose objective for an agent? We study this question in dynamic partially-observed environments, and argue that a compact and general learning objective is to minimize the entropy of the agent’s state visitation estimated using a latent state-space model. This objective induces an agent to both gather information about its environment, corresponding to reducing uncertainty, and to gain control over its environment, corresponding to reducing the unpredictability of future world states. We instantiate this approach as a deep reinforcement learning agent equipped with a deep variational Bayes filter. We find that our agent learns to discover, represent, and exercise control of dynamic objects in a variety of partially-observed environments sensed with visual observations without extrinsic reward.
@article{rhinehart2021information,
author = {Rhinehart, Nicholas and Wang, Jenny and Berseth, Glen and Co-Reyes, John and Hafner, Danijar and Finn, Chelsea and Levine, Sergey},
journal = {Advances in Neural Information Processing Systems},
pages = {10745--10758},
title = {Information is power: Intrinsic control via information capture},
volume = {34},
year = {2021}
}
Explore and control with adversarial surprise.
arXiv preprint arXiv:2107.07394, 2021.
arXiv preprint arXiv:2107.07394, 2021.
Unsupervised reinforcement learning (RL) studies how to leverage environment statistics to learn useful behaviors without the cost of reward engineering. However, a central challenge in unsupervised RL is to extract behaviors that meaningfully affect the world and cover the range of possible outcomes, without getting distracted by inherently unpredictable, uncontrollable, and stochastic elements in the environment. To this end, we propose an unsupervised RL method designed for high-dimensional, stochastic environments based on an adversarial game between two policies (which we call Explore and Control) controlling a single body and competing over the amount of observation entropy the agent experiences. The Explore agent seeks out states that maximally surprise the Control agent, which in turn aims to minimize surprise, and thereby manipulate the environment to return to familiar and predictable states. The competition between these two policies drives them to seek out increasingly surprising parts of the environment while learning to gain mastery over them. We show formally that the resulting algorithm maximizes coverage of the underlying state in block MDPs with stochastic observations, providing theoretical backing to our hypothesis that this procedure avoids uncontrollable and stochastic distractions. Our experiments further demonstrate that Adversarial Surprise leads to the emergence of complex and meaningful skills, and outperforms state-of-the-art unsupervised reinforcement learning methods in terms of both exploration and zero-shot transfer to downstream tasks.
@article{fickinger2021explore,
author = {Fickinger, Arnaud and Jaques, Natasha and Parajuli, Samyak and Chang, Michael and Rhinehart, Nicholas and Berseth, Glen and Russell, Stuart and Levine, Sergey},
journal = {arXiv preprint arXiv:2107.07394},
title = {Explore and control with adversarial surprise},
year = {2021}
}
SMiRL: Surprise Minimizing RL in Dynamic Environments.
International Conference on Representation Learning, (ICLR), 2019.
International Conference on Representation Learning, (ICLR), 2019.
Every living organism struggles against disruptive environmental forces to carve out and maintain an orderly niche. We propose that such a struggle to achieve and preserve order might offer a principle for the emergence of useful behaviors in artificial agents. We formalize this idea into an unsupervised reinforcement learning method called surprise minimizing reinforcement learning (SMiRL). SMiRL alternates between learning a density model to evaluate the surprise of a stimulus, and improving the policy to seek more predictable stimuli. The policy seeks out stable and repeatable situations that counteract the environment’s prevailing sources of entropy. This might include avoiding other hostile agents, or finding a stable, balanced pose for a bipedal robot in the face of disturbance forces. We demonstrate that our surprise minimizing agents can successfully play Tetris, Doom, control a humanoid to avoid falls, and navigate to escape enemies in a maze without any task-specific reward supervision. We further show that SMiRL can be used together with standard task rewards to accelerate reward-driven learning.
@inproceedings{berseth2019smirl,
author = {Berseth, Glen and Geng, Daniel and Devin, Coline and Rhinehart, Nicholas and Finn, Chelsea and Jayaraman, Dinesh and Levine, Sergey},
booktitle = {arXiv preprint arXiv:1912.05510},
title = {SMiRL: Surprise Minimizing RL in Dynamic Environments},
year = {2019}
}