Autonomous Exploration
All Publication Topics
Autonomous Driving
Autonomous Exploration
Benchmarks
Compression
Computer Vision
First-Person Video
Forecasting
Generative Models
Imitation Learning
Intrinsic Control
Machine Learning
Manipulation
Model-Based Control
Offroad Navigation
Paper Award
Reinforcement Learning
Reward Learning
Robotics
Social Navigation
Rapid exploration for open-world navigation with latent goal models.
Conference on Robot Learning, (CoRL), 2021.
Conference on Robot Learning, (CoRL), 2021.
We describe a robotic learning system for autonomous exploration and navigation in diverse, open-world environments. At the core of our method is a learned latent variable model of distances and actions, along with a non-parametric topological memory of images. We use an information bottleneck to regularize the learned policy, giving us (i) a compact visual representation of goals, (ii) improved generalization capabilities, and (iii) a mechanism for sampling feasible goals for exploration. Trained on a large offline dataset of prior experience, the model acquires a representation of visual goals that is robust to task-irrelevant distractors. We demonstrate our method on a mobile ground robot in open-world exploration scenarios. Given an image of a goal that is up to 80 meters away, our method leverages its representation to explore and discover the goal in under 20 minutes, even amidst previously-unseen obstacles and weather conditions. Please check out the project website for videos of our experiments and information about the real-world dataset used at https://sites.google.com/view/recon-robot.
@article{shah2021rapid,
author = {Shah, Dhruv and Eysenbach, Benjamin and Kahn, Gregory and Rhinehart, Nicholas and Levine, Sergey},
journal = {arXiv preprint arXiv:2104.05859},
title = {Rapid exploration for open-world navigation with latent goal models},
year = {2021}
}
Information is power: Intrinsic control via information capture.
Advances in Neural Information Processing Systems, (NeurIPS), 2021.
Advances in Neural Information Processing Systems, (NeurIPS), 2021.
Humans and animals explore their environment and acquire useful skills even in the absence of clear goals, exhibiting intrinsic motivation. The study of intrinsic motivation in artificial agents is concerned with the following question: what is a good general-purpose objective for an agent? We study this question in dynamic partially-observed environments, and argue that a compact and general learning objective is to minimize the entropy of the agent’s state visitation estimated using a latent state-space model. This objective induces an agent to both gather information about its environment, corresponding to reducing uncertainty, and to gain control over its environment, corresponding to reducing the unpredictability of future world states. We instantiate this approach as a deep reinforcement learning agent equipped with a deep variational Bayes filter. We find that our agent learns to discover, represent, and exercise control of dynamic objects in a variety of partially-observed environments sensed with visual observations without extrinsic reward.
@article{rhinehart2021information,
author = {Rhinehart, Nicholas and Wang, Jenny and Berseth, Glen and Co-Reyes, John and Hafner, Danijar and Finn, Chelsea and Levine, Sergey},
journal = {Advances in Neural Information Processing Systems},
pages = {10745--10758},
title = {Information is power: Intrinsic control via information capture},
volume = {34},
year = {2021}
}
Explore and control with adversarial surprise.
arXiv preprint arXiv:2107.07394, 2021.
arXiv preprint arXiv:2107.07394, 2021.
Unsupervised reinforcement learning (RL) studies how to leverage environment statistics to learn useful behaviors without the cost of reward engineering. However, a central challenge in unsupervised RL is to extract behaviors that meaningfully affect the world and cover the range of possible outcomes, without getting distracted by inherently unpredictable, uncontrollable, and stochastic elements in the environment. To this end, we propose an unsupervised RL method designed for high-dimensional, stochastic environments based on an adversarial game between two policies (which we call Explore and Control) controlling a single body and competing over the amount of observation entropy the agent experiences. The Explore agent seeks out states that maximally surprise the Control agent, which in turn aims to minimize surprise, and thereby manipulate the environment to return to familiar and predictable states. The competition between these two policies drives them to seek out increasingly surprising parts of the environment while learning to gain mastery over them. We show formally that the resulting algorithm maximizes coverage of the underlying state in block MDPs with stochastic observations, providing theoretical backing to our hypothesis that this procedure avoids uncontrollable and stochastic distractions. Our experiments further demonstrate that Adversarial Surprise leads to the emergence of complex and meaningful skills, and outperforms state-of-the-art unsupervised reinforcement learning methods in terms of both exploration and zero-shot transfer to downstream tasks.
@article{fickinger2021explore,
author = {Fickinger, Arnaud and Jaques, Natasha and Parajuli, Samyak and Chang, Michael and Rhinehart, Nicholas and Berseth, Glen and Russell, Stuart and Levine, Sergey},
journal = {arXiv preprint arXiv:2107.07394},
title = {Explore and control with adversarial surprise},
year = {2021}
}
Conservative safety critics for exploration.
International Conference on Represetation Learning, (ICLR), 2020.
International Conference on Represetation Learning, (ICLR), 2020.
Safe exploration presents a major challenge in reinforcement learning (RL): when active data collection requires deploying partially trained policies, we must ensure that these policies avoid catastrophically unsafe regions, while still enabling trial and error learning. In this paper, we target the problem of safe exploration in RL by learning a conservative safety estimate of environment states through a critic, and provably upper bound the likelihood of catastrophic failures at every training iteration. We theoretically characterize the tradeoff between safety and policy improvement, show that the safety constraints are likely to be satisfied with high probability during training, derive provable convergence guarantees for our approach, which is no worse asymptotically than standard RL, and demonstrate the efficacy of the proposed approach on a suite of challenging navigation, manipulation, and locomotion tasks. Empirically, we show that the proposed approach can achieve competitive task performance while incurring significantly lower catastrophic failure rates during training than prior methods. Videos are at this url https://sites.google.com/view/conservative-safety-critics/home
@article{bharadhwaj2020conservative,
author = {Bharadhwaj, Homanga and Kumar, Aviral and Rhinehart, Nicholas and Levine, Sergey and Shkurti, Florian and Garg, Animesh},
journal = {arXiv preprint arXiv:2010.14497},
title = {Conservative safety critics for exploration},
year = {2020}
}
SMiRL: Surprise Minimizing RL in Dynamic Environments.
International Conference on Representation Learning, (ICLR), 2019.
International Conference on Representation Learning, (ICLR), 2019.
Every living organism struggles against disruptive environmental forces to carve out and maintain an orderly niche. We propose that such a struggle to achieve and preserve order might offer a principle for the emergence of useful behaviors in artificial agents. We formalize this idea into an unsupervised reinforcement learning method called surprise minimizing reinforcement learning (SMiRL). SMiRL alternates between learning a density model to evaluate the surprise of a stimulus, and improving the policy to seek more predictable stimuli. The policy seeks out stable and repeatable situations that counteract the environment’s prevailing sources of entropy. This might include avoiding other hostile agents, or finding a stable, balanced pose for a bipedal robot in the face of disturbance forces. We demonstrate that our surprise minimizing agents can successfully play Tetris, Doom, control a humanoid to avoid falls, and navigate to escape enemies in a maze without any task-specific reward supervision. We further show that SMiRL can be used together with standard task rewards to accelerate reward-driven learning.
@inproceedings{berseth2019smirl,
author = {Berseth, Glen and Geng, Daniel and Devin, Coline and Rhinehart, Nicholas and Finn, Chelsea and Jayaraman, Dinesh and Levine, Sergey},
booktitle = {arXiv preprint arXiv:1912.05510},
title = {SMiRL: Surprise Minimizing RL in Dynamic Environments},
year = {2019}
}