Publications
All Publication Topics
Autonomous Driving
Autonomous Exploration
Benchmarks
Compression
Computer Vision
First-Person Video
Forecasting
Generative Models
Imitation Learning
Intrinsic Control
Machine Learning
Manipulation
Model-Based Control
Offroad Navigation
Paper Award
Reinforcement Learning
Reward Learning
Robotics
Social Navigation
UBP2: Uncertainty-Balanced Preference Planning for Efficient Preference-based Reinforcement Learning.
arXiv:2606.19328, (arXiv), 2026.
arXiv:2606.19328, (arXiv), 2026.
Preference-based RL provides an approach to learning reward models from pairwise comparisons of behaviors, bypassing the need for explicit reward design. However, existing methods typically rely on passive data collection and suffer from poor sample efficiency, especially during the early stages of learning. We introduce a model-based approach that actively directs exploration by jointly reasoning over uncertainties in the reward, dynamics, and value functions. Our method, Uncertainty-Balanced Preference Planning (UBP2), uses ensembles of reward, dynamics, and value function models to evaluate candidate trajectories according to a unified score that combines expected reward, terminal value, and epistemic uncertainty. Planning under this objective yields an explicit tradeoff between exploitation and information acquisition without requiring ad hoc exploration heuristics. Under standard regularity assumptions, we establish sublinear regret guarantees for both finite-horizon and infinite-horizon settings. Empirically, experiments on the Meta-World benchmark show UBP2 achieves substantially higher sample efficiency than model-free preference-based methods and non-optimistic model-based baselines.
@misc{nabail2026ubp2,
archiveprefix = {arXiv},
author = {Mohamed Nabail and Leo Cheng and Jingmin Wang and Nicholas Rhinehart},
eprint = {2606.19328},
primaryclass = {cs.LG},
title = {UBP2: Uncertainty-Balanced Preference Planning for Efficient Preference-based Reinforcement Learning},
url = {https://arxiv.org/abs/2606.19328},
year = {2026}
}
QPILOTS: Efficient Test-Time Q-Steering for Flow Policies.
arXiv:2606.14801, (arXiv), 2026.
arXiv:2606.14801, (arXiv), 2026.
Flow-matching and diffusion policies are expressive action generators, but optimizing them with temporal-difference reinforcement learning (RL) remains difficult. Effective policy extraction requires exploiting the critic’s action gradient, yet directly backpropagating this signal through a multi-step denoising process can be numerically unstable. Existing methods work around this either by discarding gradient information, distilling the policy into a simpler one-step actor, or repeatedly fine-tuning the denoising policy as the critic improves. We propose QPILOTS, a method that leaves the original policy unmodified and steers the denoising process at inference time. At each denoising step, instead of evaluating the critic on the noisy intermediate action where critic predictions are unreliable, we first project that intermediate state to an estimate of the final clean action and compute the critic gradient there. We introduce two variants: QPILOTS-U uses a fast single-point approximation, while QPILOTS-M draws differentiable posterior samples via a learned auxiliary network. On a standard offline-to-online RL benchmark, QPILOTS achieves the best aggregate performance, reaching an average success rate of 90% across 50 tasks. We also apply QPILOTS to steer a large, frozen, pretrained Vision-Language Action (VLA) foundation model, outperforming or matching prior inference-time approaches across six manipulation tasks in simulation.
@article{ruan2026qpilots,
title = {QPILOTS: Efficient Test-Time Q-Steering for Flow Policies},
author = {Ruan, Yifan and Cao, Chenyang and Burger, Andreas and Pesaranghader, Ali and Kamali, Kaveh and Kim, Jaehong and Vijaykumar, Nandita and Aspuru-Guzik, Alan and Gilitschenski, Igor and Rhinehart, Nicholas},
journal = {arXiv preprint arXiv:2606.14801},
year = {2026},
doi = {10.48550/arXiv.2606.14801},
url = {https://arxiv.org/abs/2606.14801}
}
OSCAR: Obstacle Survival Curves for Adaptive Robot Navigation.
arXiv:2606.00990, (arXiv), 2026.
arXiv:2606.00990, (arXiv), 2026.
A mobile robot following a graph of known routes can make costly navigation errors when a temporary obstacle blocks a critical edge: waiting too long behind a parked cart wastes time, but immediately rerouting around a person who would move in a few seconds is also inefficient. Standard reactive obstacle avoidance addresses local motion around obstacles, while fixed wait-or-reroute rules ignore how long different obstacle types tend to persist. We propose OSCAR: an adaptive survival-modeling framework for graph-based navigation with temporary blockages. Assuming obstacle class labels are available at encounter time, the robot learns class-conditioned residual clearance-time distributions from online experience, including right-censored observations when it reroutes before observing clearance. These survival models are integrated into a time-dependent graph planner that maintains obstacle memory and computes a patience threshold at each blocked edge: how long to wait before taking an alternate route. The method continuously updates its clearance estimates across episodes and uses them to balance waiting against rerouting. We evaluate the approach in simulation and on a real mobile robot in a university atrium with obstacles including people, chairs, bins, and tubes. In simulation, the learned policy’s time-to-goal converges to within 1% of an oracle with access to ground-truth clearance distributions after fewer than 20 observations per obstacle class, outperforming all heuristic baselines. Real-world deployment confirms that the policy improves online, adapting its patience thresholds from experience across 50 navigation episodes.
@misc{sahak2026oscar,
archiveprefix = {arXiv},
author = {Hshmat Sahak and Aoran Jiao and Nicholas Rhinehart and Tim Barfoot},
eprint = {2606.00990},
primaryclass = {cs.RO},
title = {OSCAR: Obstacle Survival Curves for Adaptive Robot Navigation},
url = {https://arxiv.org/abs/2606.00990},
year = {2026}
}
AutoWorld: Scaling Multi-Agent Traffic Simulation with Self-Supervised World Models.
arXiv:2603.28963, (arXiv), 2026.
arXiv:2603.28963, (arXiv), 2026.
Multi-agent traffic simulation is central to developing and testing autonomous driving systems. Recent data-driven simulators have achieved promising results, but rely heavily on supervised learning from labeled trajectories or semantic annotations, making it costly to scale their performance. Meanwhile, large amounts of unlabeled sensor data can be collected at scale but remain largely unused by existing traffic simulation frameworks. This raises a key question: How can a method harness unlabeled data to improve traffic simulation performance? In this work, we propose AutoWorld, a traffic simulation framework that employs a world model learned from unlabeled occupancy representations of LiDAR data. Given world model samples, AutoWorld constructs a coarse-to-fine predictive scene context as input to a multi-agent motion generation model. To promote sample diversity, AutoWorld uses a cascaded Determinantal Point Process framework to guide the sampling processes of both the world model and the motion model. Furthermore, we designed a motion-aware latent supervision objective that enhances AutoWorld’s representation of scene dynamics. Experiments on the WOSAC benchmark show that AutoWorld ranks first on the leaderboard according to the primary Realism Meta Metric (RMM). We further show that simulation performance consistently improves with the inclusion of unlabeled LiDAR data, and study the efficacy of each component with ablations. Our method paves the way for scaling traffic simulation realism without additional labeling. Our project page contains additional visualizations and released code.
@article{pourkeshavarz2026autoworld,
title = {AutoWorld: Scaling Multi-Agent Traffic Simulation with Self-Supervised World Models},
author = {Pourkeshavarz, Mozhgan and Liu, Tianran and Rhinehart, Nicholas},
journal = {arXiv preprint arXiv:2603.28963},
year = {2026},
doi = {10.48550/arXiv.2603.28963},
url = {https://arxiv.org/abs/2603.28963}
}
OccSim: Multi-kilometer Simulation with Long-horizon Occupancy World Models.
arXiv:2603.28887, (arXiv), 2026.
arXiv:2603.28887, (arXiv), 2026.
Data-driven autonomous driving simulation has long been constrained by its heavy reliance on pre-recorded driving logs or spatial priors, such as HD maps. This fundamental dependency severely limits scalability, restricting open-ended generation capabilities to the finite scale of existing collected datasets. To break this bottleneck, we present OccSim, the first occupancy world model-driven 3D simulator. OccSim obviates the requirement for continuous logs or HD maps; conditioned only on a single initial frame and a sequence of future ego-actions, it can stably generate over 3,000 continuous frames, enabling the continuous construction of large-scale 3D occupancy maps spanning over 4 kilometers for simulation. This represents an >80x improvement in stable generation length over previous state-of-the-art occupancy world models. OccSim is powered by two modules: W-DiT based static occupancy world model and the Layout Generator. W-DiT handles the ultra-long-horizon generation of static environments by explicitly introducing known rigid transformations in architecture design, while the Layout Generator populates the dynamic foreground with reactive agents based on the synthesized road topology. With these designs, OccSim can synthesize massive, diverse simulation streams. Extensive experiments demonstrate its downstream utility: data collected directly from OccSim can pre-train 4D semantic occupancy forecasting models to achieve up to 67% zero-shot performance on unseen data, outperforming previous asset-based simulator by 11%. When scaling the OccSim dataset to 5x the size, the zero-shot performance increases to about 74%, while the improvement over asset-based simulators expands to 22.1%.
@article{liu2026occsim,
title = {OccSim: Multi-kilometer Simulation with Long-horizon Occupancy World Models},
author = {Liu, Tianran and Zhao, Shengwen and Pourkeshavarz, Mozhgan and Li, Weican and Rhinehart, Nicholas},
journal = {arXiv preprint arXiv:2603.28887},
year = {2026},
doi = {10.48550/arXiv.2603.28887},
url = {https://arxiv.org/abs/2603.28887}
}
Ratatouille: Imitation Learning Ingredients for Real-world Social Robot Navigation.
arXiv:2509.17204, (arXiv), 2025.
arXiv:2509.17204, (arXiv), 2025.
Scaling Reinforcement Learning to in-the-wild social robot navigation is both data-intensive and unsafe, since policies must learn through direct interaction and inevitably encounter collisions. Offline Imitation learning (IL) avoids these risks by collecting expert demonstrations safely, training entirely offline, and deploying policies zero-shot. However, we find that naively applying Behaviour Cloning (BC) to social navigation is insufficient; achieving strong performance requires careful architectural and training choices. We present Ratatouille, a pipeline and model architecture that, without changing the data, reduces collisions per meter by 6 times and improves success rate by 3 times compared to naive BC. We validate our approach in both simulation and the real world, where we collected over 11 hours of data on a dense university campus. We further demonstrate qualitative results in a public food court. Our findings highlight that thoughtful IL design, rather than additional data, can substantially improve safety and reliability in real-world social navigation. Video: https://youtu.be/tOdLTXsaYLQ. Code will be released after acceptance.
@misc{han2025ratatouilleimitationlearningingredients,
title={Ratatouille: Imitation Learning Ingredients for Real-world Social Robot Navigation},
author={James R. Han and Mithun Vanniasinghe and Hshmat Sahak and Nicholas Rhinehart and Timothy D. Barfoot},
year={2025},
eprint={2509.17204},
archivePrefix={arXiv},
primaryClass={cs.RO},
url={https://arxiv.org/abs/2509.17204},
}
Residual Reward Models for Preference-based Reinforcement Learning.
arXiv:2507.00611, (arXiv), 2025.
arXiv:2507.00611, (arXiv), 2025.
Preference-based Reinforcement Learning (PbRL) provides a way to learn high-performance policies in environments where the reward signal is hard to specify, avoiding heuristic and time-consuming reward design. However, PbRL can suffer from slow convergence speed since it requires training in a reward model. Prior work has proposed learning a reward model from demonstrations and fine-tuning it using preferences. However, when the model is a neural network, using different loss functions for pre-training and fine-tuning can pose challenges to reliable optimization. In this paper, we propose a method to effectively leverage prior knowledge with a Residual Reward Model (RRM). An RRM assumes that the true reward of the environment can be split into a sum of two parts: a prior reward and a learned reward. The prior reward is a term available before training, for example, a user’s ``best guess’’ reward function, or a reward function learned from inverse reinforcement learning (IRL), and the learned reward is trained with preferences. We introduce state-based and image-based versions of RRM and evaluate them on several tasks in the Meta-World environment suite. Experimental results show that our method substantially improves the performance of a common PbRL method. Our method achieves performance improvements for a variety of different types of prior rewards, including proxy rewards, a reward obtained from IRL, and even a negated version of the proxy reward. We also conduct experiments with a Franka Panda to show that our method leads to superior performance on a real robot. It significantly accelerates policy learning for different tasks, achieving success in fewer steps than the baseline. The videos are presented at https://sunlighted.github.io/RRM-web/.
@article{cao2025residualrewardmodelspreferencebased,
title={Residual Reward Models for Preference-based Reinforcement Learning},
author={Chenyang Cao and Miguel Rogel-García and Mohamed Nabail and Xueqian Wang and Nicholas Rhinehart},
year={2025},
eprint={2507.00611},
archivePrefix={arXiv},
primaryClass={cs.LG},
url={https://arxiv.org/abs/2507.00611},
}
Towards foundational LiDAR world models with efficient latent flow matching.
Advances in Neural Information Processing Systems, (NeurIPS), 2025.
Advances in Neural Information Processing Systems, (NeurIPS), 2025.
LiDAR-based world models offer more structured and geometry-aware representations than their image-based counterparts. However, existing LiDAR world models are narrowly trained; each model excels only in the domain for which it was built. This raises a critical question: can we develop LiDAR world models that exhibit strong transferability across multiple domains? To answer this, we conduct the first systematic domain transfer study across three demanding scenarios: (i) outdoor to indoor generalization, (ii) sparse- to dense-beam adaptation, and (iii) non-semantic to semantic transfer. Given different amounts of fine-tuning data, our experiments show that a single pretrained model can achieve up to 11% absolute improvement (83% relative) over training from scratch and outperforms training from scratch in 30/36 of our comparisons. This transferability significantly reduces the reliance on manually annotated data for semantic occupancy forecasting: our method exceeds previous baselines with only 5% of the labeled training data of prior work. We also observed inefficiencies of current generative-model-based LiDAR world models, mainly through their under-compression of LiDAR data and inefficient training objectives. To address these issues, we propose a latent conditional flow matching (CFM)-based framework that achieves state-of-the-art reconstruction accuracy using only half the training data and a compression ratio 6 times higher than that of prior methods. Our model also achieves SOTA performance on semantic occupancy forecasting while being 1.98x-23x more computationally efficient (a 1.1x-3.9x FPS speedup) than previous methods.
@article{liu2025foundationallidarworldmodels,
title={Towards foundational LiDAR world models with efficient latent flow matching},
author={Tianran Liu and Shengwen Zhao and Nicholas Rhinehart},
year={2025},
eprint={2506.23434},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2506.23434},
}
DR-MPC: Deep Residual Model Predictive Control for Real-world Social Navigation.
IEEE Robotics and Automation Letters, (RA-L), 2025.
IEEE Robotics and Automation Letters, (RA-L), 2025.
How can a robot safely navigate around people with complex motion patterns? Deep Reinforcement Learning (DRL) in simulation holds some promise, but much prior work relies on simulators that fail to capture the nuances of real human motion. Thus, we propose Deep Residual Model Predictive Control (DR-MPC) to enable robots to quickly and safely perform DRL from real-world crowd navigation data. By blending MPC with model-free DRL, DR-MPC overcomes the DRL challenges of large data requirements and unsafe initial behavior. DR-MPC is initialized with MPC-based path tracking, and gradually learns to interact more effectively with humans. To further accelerate learning, a safety component estimates out-of-distribution states to guide the robot away from likely collisions. In simulation, we show that DR-MPC substantially outperforms prior work, including traditional DRL and residual DRL models. Hardware experiments show our approach successfully enables a robot to navigate a variety of crowded situations with few errors using less than 4 hours of training data.
@article{han2024dr,
title={DR-MPC: Deep Residual Model Predictive Control for Real-world Social Navigation},
author={Han, James R and Thomas, Hugues and Zhang, Jian and Rhinehart, Nicholas and Barfoot, Timothy D},
journal={IEEE Robotics and Automation Letters (RA-L)},
year={2025}
}
CARFF: Conditional Auto-encoded Radiance Field for 3D Scene Forecasting.
European Conference on Computer Vision, (ECCV), 2024.
European Conference on Computer Vision, (ECCV), 2024.
We propose CARFF, a method for predicting future 3D scenes given past observations. Our method maps 2D ego-centric images to a distribution over plausible 3D latent scene configurations and predicts the evolution of hypothesized scenes through time. Our latents condition a global Neural Radiance Field (NeRF) to represent a 3D scene model, enabling explainable predictions and straightforward downstream planning. This approach models the world as a POMDP and considers complex scenarios of uncertainty in environmental states and dynamics. Specifically, we employ a two-stage training of Pose-Conditional-VAE and NeRF to learn 3D representations, and auto-regressively predict latent scene representations utilizing a mixture density network. We demonstrate the utility of our method in scenarios using the CARLA driving simulator, where CARFF enables efficient trajectory and contingency planning in complex multi-agent autonomous driving scenarios involving occlusions.
@article{yang2024carff,
author = {Yang, Jiezhi and Desai, Khushi and Packer, Charles and Bhatia, Harshil and Rhinehart, Nicholas and McAllister, Rowan and Gonzalez, Joseph},
journal = {arXiv preprint arXiv:2401.18075},
title = {CARFF: Conditional Auto-encoded Radiance Field for 3D Scene Forecasting},
year = {2024}
}
Is anyone there? learning a planner contingent on perceptual uncertainty.
Conference on Robot Learning, (CoRL), 2023.
Conference on Robot Learning, (CoRL), 2023.
Robots in complex multi-agent environments should reason about the intentions of observed and currently unobserved agents. In this paper, we present a new learning-based method for prediction and planning in complex multi-agent environments where the states of the other agents are partially-observed. Our approach, Active Visual Planning (AVP), uses high-dimensional observations to learn a flow-based generative model of multi-agent joint trajectories, including unobserved agents that may be revealed in the near future, depending on the robot’s actions. Our predictive model is implemented using deep neural networks that map raw observations to future detection and pose trajectories and is learned entirely offline using a dataset of recorded observations (not ground-truth states). Once learned, our predictive model can be used for contingency planning over the potential existence, intentions, and positions of unobserved agents. We demonstrate the effectiveness of AVP on a set of autonomous driving environments inspired by real-world scenarios that require reasoning about the existence of other unobserved agents for safe and efficient driving. In these environments, AVP achieves optimal closed-loop performance, while methods that do not reason about potential unobserved agents exhibit either overconfident or underconfident behavior.
@inproceedings{packer2023anyone,
author = {Packer, Charles and Rhinehart, Nicholas and McAllister, Rowan Thomas and Wright, Matthew A and Wang, Xin and He, Jeff and Levine, Sergey and Gonzalez, Joseph E},
booktitle = {Conference on Robot Learning},
organization = {PMLR},
pages = {1607--1617},
title = {Is anyone there? learning a planner contingent on perceptual uncertainty},
year = {2023}
}
The Waymo Open Sim Agents Challenge.
Advances in Neural Information Processing Systems, (NeurIPS), 2023.
Advances in Neural Information Processing Systems, (NeurIPS), 2023.
Simulation with realistic, interactive agents represents a key task for autonomous vehicle software development. In this work, we introduce the Waymo Open Sim Agents Challenge (WOSAC). WOSAC is the first public challenge to tackle this task and propose corresponding metrics. The goal of the challenge is to stimulate the design of realistic simulators that can be used to evaluate and train a behavior model for autonomous driving. We outline our evaluation methodology, present results for a number of different baseline simulation agent methods, and analyze several submissions to the 2023 competition which ran from March 16, 2023 to May 23, 2023. The WOSAC evaluation server remains open for submissions and we discuss open problems for the task.
@article{montali2024waymo,
author = {Montali, Nico and Lambert, John and Mougin, Paul and Kuefler, Alex and Rhinehart, Nicholas and Li, Michelle and Gulino, Cole and Emrich, Tristan and Yang, Zoey and Whiteson, Shimon and others},
journal = {Advances in Neural Information Processing Systems},
title = {The waymo open sim agents challenge},
volume = {36},
year = {2024}
}
Hybrid imitative planning with geometric and predictive costs in off-road environments.
International Conference on Robotics and Automation, (ICRA), 2022.
International Conference on Robotics and Automation, (ICRA), 2022.
Geometric methods for solving open-world off-road navigation tasks, by learning occupancy and metric maps, provide good generalization but can be brittle in outdoor environments that violate their assumptions (e.g., tall grass). Learning-based methods can directly learn collision-free behavior from raw observations, but are difficult to integrate with standard geometry-based pipelines. This creates an unfortunate conflict – either use learning and lose out on well-understood geometric navigational components, or do not use it, in favor of extensively hand-tuned geometry-based cost maps. In this work, we reject this dichotomy by designing the learning and non-learning-based components in a way such that they can be effectively combined in a self-supervised manner. Both components contribute to a planning criterion: the learned component contributes predicted traversability as rewards, while the geometric component contributes obstacle cost information. We instantiate and comparatively evaluate our system in both in-distribution and out-of-distribution environments, showing that this approach inherits complementary gains from the learned and geometric components and significantly outperforms either of them.
@inproceedings{dashora2022hybrid,
author = {Dashora, Nitish and Shin, Daniel and Shah, Dhruv and Leopold, Henry and Fan, David and Agha-Mohammadi, Ali and Rhinehart, Nicholas and Levine, Sergey},
booktitle = {2022 International Conference on Robotics and Automation (ICRA)},
organization = {IEEE},
pages = {4452--4458},
title = {Hybrid imitative planning with geometric and predictive costs in off-road environments},
year = {2022}
}
Offline reinforcement learning for visual navigation.
Conference on Robot Learning, (CoRL), 2022.
Conference on Robot Learning, (CoRL), 2022.
Reinforcement learning can enable robots to navigate to distant goals while optimizing user-specified reward functions, including preferences for following lanes, staying on paved paths, or avoiding freshly mowed grass. However, online learning from trial-and-error for real-world robots is logistically challenging, and methods that instead can utilize existing datasets of robotic navigation data could be significantly more scalable and enable broader generalization. In this paper, we present ReViND, the first offline RL system for robotic navigation that can leverage previously collected data to optimize user-specified reward functions in the real-world. We evaluate our system for off-road navigation without any additional data collection or fine-tuning, and show that it can navigate to distant goals using only offline training from this dataset, and exhibit behaviors that qualitatively differ based on the user-specified reward function.
@article{shah2022offline,
author = {Shah, Dhruv and Bhorkar, Arjun and Leen, Hrish and Kostrikov, Ilya and Rhinehart, Nick and Levine, Sergey},
journal = {arXiv preprint arXiv:2212.08244},
title = {Offline reinforcement learning for visual navigation},
year = {2022}
}
S2Net: Stochastic Sequential Pointcloud Forecasting.
European Conference on Computer Vision, (ECCV), 2022.
European Conference on Computer Vision, (ECCV), 2022.
Predicting futures of surrounding agents is critical for autonomous systems such as self-driving cars. Instead of requiring accurate detection and tracking prior to trajectory prediction, an object agnostic Sequential Pointcloud Forecasting (SPF) task was proposed [28], which enables a forecast-then-detect pipeline effective for downstream detection and trajectory prediction. One limitation of prior work is that it forecasts only a deterministic sequence of future point clouds, despite the inherent uncertainty of dynamic scenes. In this work, we tackle the stochastic SPF problem by proposing a generative model with two main components: (1) a conditional variational recurrent neural network that models a temporally-dependent latent space; (2) a pyramid-LSTM that increases the fidelity of predictions with temporally-aligned skip connections. Through experiments on real-world autonomous driving datasets, our stochastic SPF model produces higher-fidelity predictions, reducing Chamfer distances by up to 56.6% compared to its deterministic counterpart. In addition, our model can estimate the uncertainty of predicted points, which can be helpful to downstream tasks.
@inproceedings{weng2022s2net,
author = {Weng, Xinshuo and Nan, Junyu and Lee, Kuan-Hui and McAllister, Rowan and Gaidon, Adrien and Rhinehart, Nicholas and Kitani, Kris},
booktitle = {European Conference on Computer Vision (ECCV)},
title = {S2Net: Stochastic Sequential Pointcloud Forecasting},
year = {2022}
}
Traffic prediction with reparameterized pushforward policy for autonomous vehicles.
U.S. Patent No. 11,189,171, 2021.
U.S. Patent No. 11,189,171, 2021.
Systems and methods for vehicle behavior prediction include an imaging device that captures images of a vehicle in traffic. A processing device including policy stored in a memory of the processing device in communication with the imaging device stochastically models future behavior of the vehicle based on the captured images. A policy simulator in communication with the processing device simulates the policy as a reparameterized pushforward policy of a base distribution. An evaluator receives the simulated policy from the policy simulator and performs cross-entropy optimization on the future behavior of the vehicle by analyzing the simulated policy and updating the policy according to cross-entropy error. An alert system retrieves the future behavior of the vehicle and recognizes hazardous trajectories of the future trajectories and generates an audible alert using a speaker.
@misc{vernaza2021traffic,
author = {Vernaza, Paul and Rhinehart, Nicholas},
month = {November~30},
note = {US Patent 11,189,171},
title = {Traffic prediction with reparameterized pushforward policy for autonomous vehicles},
year = {2021}
}
Contingencies from observations: Tractable contingency planning with learned behavior models.
IEEE International Conference on Robotics and Automation, (ICRA), 2021.
IEEE International Conference on Robotics and Automation, (ICRA), 2021.
Humans have a remarkable ability to accurately reason about future events, including the behaviors and states of mind of other agents. Consider driving a car through a busy intersection: it is necessary to reason about the physics of the vehicle, the intentions of other drivers, and their beliefs about your own intentions. For example, if you signal a turn, another driver might yield to you; or if you enter the passing lane, another driver might decelerate to give you room to merge in front. Competent drivers must plan how they can safely react to a variety of potential future behaviors of other agents before they make their next move. This requires contingency planning: explicitly planning a set of conditional actions that depend on the stochastic outcome of future events. In this work, we develop a general-purpose contingency planner that is learned end-to-end using high-dimensional scene observations and low-dimensional behavioral observations. We use a conditional autoregressive flow model for contingency planning. We show how this model can tractably learn contingencies from behavioral observations. We developed a closed-loop control benchmark of realistic multi-agent scenarios in a driving simulator (CARLA), on which we compare our method to various noncontingent methods that reason about multi-agent future behavior, and find that our contingency planning method achieves qualitatively and quantitatively superior performance.
@inproceedings{rhinehart2021contingencies,
author = {Rhinehart, Nicholas and He, Jeff and Packer, Charles and Wright, Matthew A and McAllister, Rowan and Gonzalez, Joseph E and Levine, Sergey},
booktitle = {2021 IEEE International Conference on Robotics and Automation (ICRA)},
organization = {IEEE},
pages = {13663--13669},
title = {Contingencies from observations: Tractable contingency planning with learned behavior models},
year = {2021}
}
Explore and control with adversarial surprise.
arXiv preprint arXiv:2107.07394, 2021.
arXiv preprint arXiv:2107.07394, 2021.
Unsupervised reinforcement learning (RL) studies how to leverage environment statistics to learn useful behaviors without the cost of reward engineering. However, a central challenge in unsupervised RL is to extract behaviors that meaningfully affect the world and cover the range of possible outcomes, without getting distracted by inherently unpredictable, uncontrollable, and stochastic elements in the environment. To this end, we propose an unsupervised RL method designed for high-dimensional, stochastic environments based on an adversarial game between two policies (which we call Explore and Control) controlling a single body and competing over the amount of observation entropy the agent experiences. The Explore agent seeks out states that maximally surprise the Control agent, which in turn aims to minimize surprise, and thereby manipulate the environment to return to familiar and predictable states. The competition between these two policies drives them to seek out increasingly surprising parts of the environment while learning to gain mastery over them. We show formally that the resulting algorithm maximizes coverage of the underlying state in block MDPs with stochastic observations, providing theoretical backing to our hypothesis that this procedure avoids uncontrollable and stochastic distractions. Our experiments further demonstrate that Adversarial Surprise leads to the emergence of complex and meaningful skills, and outperforms state-of-the-art unsupervised reinforcement learning methods in terms of both exploration and zero-shot transfer to downstream tasks.
@article{fickinger2021explore,
author = {Fickinger, Arnaud and Jaques, Natasha and Parajuli, Samyak and Chang, Michael and Rhinehart, Nicholas and Berseth, Glen and Russell, Stuart and Levine, Sergey},
journal = {arXiv preprint arXiv:2107.07394},
title = {Explore and control with adversarial surprise},
year = {2021}
}
Information is power: Intrinsic control via information capture.
Advances in Neural Information Processing Systems, (NeurIPS), 2021.
Advances in Neural Information Processing Systems, (NeurIPS), 2021.
Humans and animals explore their environment and acquire useful skills even in the absence of clear goals, exhibiting intrinsic motivation. The study of intrinsic motivation in artificial agents is concerned with the following question: what is a good general-purpose objective for an agent? We study this question in dynamic partially-observed environments, and argue that a compact and general learning objective is to minimize the entropy of the agent’s state visitation estimated using a latent state-space model. This objective induces an agent to both gather information about its environment, corresponding to reducing uncertainty, and to gain control over its environment, corresponding to reducing the unpredictability of future world states. We instantiate this approach as a deep reinforcement learning agent equipped with a deep variational Bayes filter. We find that our agent learns to discover, represent, and exercise control of dynamic objects in a variety of partially-observed environments sensed with visual observations without extrinsic reward.
@article{rhinehart2021information,
author = {Rhinehart, Nicholas and Wang, Jenny and Berseth, Glen and Co-Reyes, John and Hafner, Danijar and Finn, Chelsea and Levine, Sergey},
journal = {Advances in Neural Information Processing Systems},
pages = {10745--10758},
title = {Information is power: Intrinsic control via information capture},
volume = {34},
year = {2021}
}
Inverting the pose forecasting pipeline with SPF2: Sequential pointcloud forecasting for sequential pose forecasting.
Conference on robot learning, (CoRL), 2021.
Conference on robot learning, (CoRL), 2021.
Many autonomous systems forecast aspects of the future in order to aid decision-making. For example, self-driving vehicles and robotic manipulation systems often forecast future object poses by first detecting and tracking objects. However, this detect-then-forecast pipeline is expensive to scale, as pose forecasting algorithms typically require labeled sequences of object poses, which are costly to obtain in 3D space. Can we scale performance without requiring additional labels? We hypothesize yes, and propose inverting the detect-then-forecast pipeline. Instead of detecting, tracking and then forecasting the objects, we propose to first forecast 3D sensor data (e.g., point clouds with $100$k points) and then detect/track objects on the predicted point cloud sequences to obtain future poses, i.e., a forecast-then-detect pipeline. This inversion makes it less expensive to scale pose forecasting, as the sensor data forecasting task requires no labels. Part of this work’s focus is on the challenging first step –Sequential Pointcloud Forecasting (SPF), for which we also propose an effective approach, SPFNet. To compare our forecast-then-detect pipeline relative to the detect-then-forecast pipeline, we propose an evaluation procedure and two metrics. Through experiments on a robotic manipulation dataset and two driving datasets, we show that SPFNet is effective for the SPF task, our forecast-then-detect pipeline outperforms the detect-then-forecast approaches to which we compared, and that pose forecasting performance improves with the addition of unlabeled data.
@inproceedings{weng2021inverting,
author = {Weng, Xinshuo and Wang, Jianren and Levine, Sergey and Kitani, Kris and Rhinehart, Nicholas},
booktitle = {Conference on robot learning},
organization = {PMLR},
pages = {11--20},
title = {Inverting the pose forecasting pipeline with SPF2: Sequential pointcloud forecasting for sequential pose forecasting},
year = {2021}
}
Rapid exploration for open-world navigation with latent goal models.
Conference on Robot Learning, (CoRL), 2021.
Conference on Robot Learning, (CoRL), 2021.
We describe a robotic learning system for autonomous exploration and navigation in diverse, open-world environments. At the core of our method is a learned latent variable model of distances and actions, along with a non-parametric topological memory of images. We use an information bottleneck to regularize the learned policy, giving us (i) a compact visual representation of goals, (ii) improved generalization capabilities, and (iii) a mechanism for sampling feasible goals for exploration. Trained on a large offline dataset of prior experience, the model acquires a representation of visual goals that is robust to task-irrelevant distractors. We demonstrate our method on a mobile ground robot in open-world exploration scenarios. Given an image of a goal that is up to 80 meters away, our method leverages its representation to explore and discover the goal in under 20 minutes, even amidst previously-unseen obstacles and weather conditions. Please check out the project website for videos of our experiments and information about the real-world dataset used at https://sites.google.com/view/recon-robot.
@article{shah2021rapid,
author = {Shah, Dhruv and Eysenbach, Benjamin and Kahn, Gregory and Rhinehart, Nicholas and Levine, Sergey},
journal = {arXiv preprint arXiv:2104.05859},
title = {Rapid exploration for open-world navigation with latent goal models},
year = {2021}
}
Ving: Learning open-world navigation with visual goals.
IEEE International Conference on Robotics and Automation, (ICRA), 2021.
IEEE International Conference on Robotics and Automation, (ICRA), 2021.
We propose a learning-based navigation system for reaching visually indicated goals and demonstrate this system on a real mobile robot platform. Learning provides an appealing alternative to conventional methods for robotic navigation: instead of reasoning about environments in terms of geometry and maps, learning can enable a robot to learn about navigational affordances, understand what types of obstacles are traversable (e.g., tall grass) or not (e.g., walls), and generalize over patterns in the environment. However, unlike conventional planning algorithms, it is harder to change the goal for a learned policy during deployment. We propose a method for learning to navigate towards a goal image of the desired destination. By combining a learned policy with a topological graph constructed out of previously observed data, our system can determine how to reach this visually indicated goal even in the presence of variable appearance and lighting. Three key insights, waypoint proposal, graph pruning and negative mining, enable our method to learn to navigate in real-world environments using only offline data, a setting where prior methods struggle. We instantiate our method on a real outdoor ground robot and show that our system, which we call ViNG, outperforms previously-proposed methods for goal-conditioned reinforcement learning, including other methods that incorporate reinforcement learning and search. We also study how ViNG generalizes to unseen environments and evaluate its ability to adapt to such an environment with growing experience. Finally, we demonstrate ViNG on a number of real-world applications, such as last-mile delivery and warehouse inspection. We encourage the reader to visit the project website for videos of our experiments and demonstrations 1 .
@inproceedings{shah2021ving,
author = {Shah, Dhruv and Eysenbach, Benjamin and Kahn, Gregory and Rhinehart, Nicholas and Levine, Sergey},
booktitle = {2021 IEEE International Conference on Robotics and Automation (ICRA)},
organization = {IEEE},
pages = {13215--13222},
title = {Ving: Learning open-world navigation with visual goals},
year = {2021}
}
Generative adversarial inverse trajectory optimization for probabilistic vehicle forecasting.
U.S. Patent 10,739,773, 2020.
U.S. Patent 10,739,773, 2020.
Systems and methods for predicting vehicle behavior includes capturing images of a vehicle in traffic using an imaging device. Future behavior of the vehicle is stochastically modeled using a processing device including an energy-based model stored in a memory of the processing device. The energy-based model includes generating a distribution of possible future trajectories of the vehicle using a generator, sampling the distribution of possible future trajectories according to an energy value of each trajectory in the distribution of possible future trajectories an energy model to determine probable future trajectories, and optimizing parameters of each of the generator and the energy model using an optimizer. A user is audibly alerted with a speaker upon an alert system recognizing hazardous trajectories of the probable future trajectories.
@misc{vernaza2020generative,
author = {Vernaza, Paul and Choi, Wongun and Rhinehart, Nicholas},
month = {July~7},
note = {US Patent 10,705,531},
title = {Generative adversarial inverse trajectory optimization for probabilistic vehicle forecasting},
year = {2020}
}
Can autonomous vehicles identify, recover from, and adapt to distribution shifts?.
International Conference on Machine Learning, (ICML), 2020.
International Conference on Machine Learning, (ICML), 2020.
Out-of-training-distribution (OOD) scenarios are a common challenge of learning agents at deployment, typically leading to arbitrary deductions and poorly-informed decisions. In principle, detection of and adaptation to OOD scenes can mitigate their adverse effects. In this paper, we highlight the limitations of current approaches to novel driving scenes and propose an epistemic uncertainty-aware planning method, called \emph{robust imitative planning} (RIP). Our method can detect and recover from some distribution shifts, reducing the overconfident and catastrophic extrapolations in OOD scenes. If the model’s uncertainty is too great to suggest a safe course of action, the model can instead query the expert driver for feedback, enabling sample-efficient online adaptation, a variant of our method we term \emph{adaptive robust imitative planning} (AdaRIP). Our methods outperform current state-of-the-art approaches in the nuScenes \emph{prediction} challenge, but since no benchmark evaluating OOD detection and adaption currently exists to assess \emph{control}, we introduce an autonomous car novel-scene benchmark, \texttt{CARNOVEL}, to evaluate the robustness of driving agents to a suite of tasks with distribution shifts.
@inproceedings{filos2020can,
author = {Filos, Angelos and Tigkas, Panagiotis and McAllister, Rowan and Rhinehart, Nicholas and Levine, Sergey and Gal, Yarin},
booktitle = {International Conference on Machine Learning},
organization = {PMLR},
pages = {3145--3153},
title = {Can autonomous vehicles identify, recover from, and adapt to distribution shifts?},
year = {2020}
}
Conservative safety critics for exploration.
International Conference on Represetation Learning, (ICLR), 2020.
International Conference on Represetation Learning, (ICLR), 2020.
Safe exploration presents a major challenge in reinforcement learning (RL): when active data collection requires deploying partially trained policies, we must ensure that these policies avoid catastrophically unsafe regions, while still enabling trial and error learning. In this paper, we target the problem of safe exploration in RL by learning a conservative safety estimate of environment states through a critic, and provably upper bound the likelihood of catastrophic failures at every training iteration. We theoretically characterize the tradeoff between safety and policy improvement, show that the safety constraints are likely to be satisfied with high probability during training, derive provable convergence guarantees for our approach, which is no worse asymptotically than standard RL, and demonstrate the efficacy of the proposed approach on a suite of challenging navigation, manipulation, and locomotion tasks. Empirically, we show that the proposed approach can achieve competitive task performance while incurring significantly lower catastrophic failure rates during training than prior methods. Videos are at this url https://sites.google.com/view/conservative-safety-critics/home
@article{bharadhwaj2020conservative,
author = {Bharadhwaj, Homanga and Kumar, Aviral and Rhinehart, Nicholas and Levine, Sergey and Shkurti, Florian and Garg, Animesh},
journal = {arXiv preprint arXiv:2010.14497},
title = {Conservative safety critics for exploration},
year = {2020}
}
Deep Imitative Models for Flexible Inference, Planning, and Control.
International Conference on Learning Representations, (ICLR), 2020.
International Conference on Learning Representations, (ICLR), 2020.
Imitation Learning (IL) is an appealing approach to learn desirable autonomous behavior. However, directing IL to achieve arbitrary goals is difficult. In contrast, planning-based algorithms use dynamics models and reward functions to achieve goals. Yet, reward functions that evoke desirable behavior are often difficult to specify. In this paper, we propose Imitative Models to combine the benefits of IL and goal-directed planning. Imitative Models are probabilistic predictive models of desirable behavior able to plan interpretable expert-like trajectories to achieve specified goals. We derive families of flexible goal objectives, including constrained goal regions, unconstrained goal sets, and energy-based goals. We show that our method can use these objectives to successfully direct behavior. Our method substantially outperforms six IL approaches and a planning-based approach in a dynamic simulated autonomous driving task, and is efficiently learned from expert demonstrations without online data collection. We also show our approach is robust to poorly specified goals, such as goals on the wrong side of the road.
@inproceedings{rhinehart2020deep,
author = {Rhinehart, Nicholas and McAllister, Rowan and Levine, Sergey},
booktitle = {International Conference on Learning Representations (ICLR)},
title = {Deep Imitative Models for Flexible Inference, Planning, and Control},
year = {2020}
}
Parrot: Data-driven behavioral priors for reinforcement learning.
International Conference on Learning Representations, (ICLR), 2020.
International Conference on Learning Representations, (ICLR), 2020.
Reinforcement learning provides a general framework for flexible decision making and control, but requires extensive data collection for each new task that an agent needs to learn. In other machine learning fields, such as natural language processing or computer vision, pre-training on large, previously collected datasets to bootstrap learning for new tasks has emerged as a powerful paradigm to reduce data requirements when learning a new task. In this paper, we ask the following question: how can we enable similarly useful pre-training for RL agents? We propose a method for pre-training behavioral priors that can capture complex input-output relationships observed in successful trials from a wide range of previously seen tasks, and we show how this learned prior can be used for rapidly learning new tasks without impeding the RL agent’s ability to try out novel behaviors. We demonstrate the effectiveness of our approach in challenging robotic manipulation domains involving image observations and sparse reward functions, where our method outperforms prior works by a substantial margin.
@article{singh2020parrot,
author = {Singh, Avi and Liu, Huihan and Zhou, Gaoyue and Yu, Albert and Rhinehart, Nicholas and Levine, Sergey},
journal = {arXiv preprint arXiv:2011.10024},
title = {Parrot: Data-driven behavioral priors for reinforcement learning},
year = {2020}
}
Directed-Info GAIL: Learning Hierarchical Policies from Unsegmented Demonstrations using Directed Information.
International Conference on Learning Representations (ICLR), (ICLR), 2019.
International Conference on Learning Representations (ICLR), (ICLR), 2019.
The use of imitation learning to learn a single policy for a complex task that has multiple modes or hierarchical structure can be challenging. In fact, previous work has shown that when the modes are known, learning separate policies for each mode or sub-task can greatly improve the performance of imitation learning. In this work, we discover the interaction between sub-tasks from their resulting state-action trajectory sequences using a directed graphical model. We propose a new algorithm based on the generative adversarial imitation learning framework which automatically learns sub-task policies from unsegmented demonstrations. Our approach maximizes the directed information flow in the graphical model between sub-task latent variables and their generated trajectories. We also show how our approach connects with the existing Options framework, which is commonly used to learn hierarchical policies.
@inproceedings{sharma2019directed,
author = {Sharma, Arjun and Sharma, Mohit and Rhinehart, Nicholas and Kitani, Kris M},
booktitle = {International Conference on Learning Representations (ICLR)},
title = {Directed-Info GAIL: Learning Hierarchical Policies from Unsegmented Demonstrations using Directed Information},
year = {2019}
}
Generative Hybrid Representations for Activity Forecasting with No-Regret Learning.
Conference on Computer Vision and Pattern Recognition, (CVPR), 2019.
Conference on Computer Vision and Pattern Recognition, (CVPR), 2019.
Automatically reasoning about future human behaviors is a difficult problem but has significant practical applications to assistive systems. Part of this difficulty stems from learning systems’ inability to represent all kinds of behaviors. Some behaviors, such as motion, are best described with continuous representations, whereas others, such as picking up a cup, are best described with discrete representations. Furthermore, human behavior is generally not fixed: people can change their habits and routines. This suggests these systems must be able to learn and adapt continuously. In this work, we develop an efficient deep generative model to jointly forecast a person’s future discrete actions and continuous motions. On a large-scale egocentric dataset, EPIC-KITCHENS, we observe our method generates high-quality and diverse samples while exhibiting better generalization than related generative models. Finally, we propose a variant to continually learn our model from streaming data, observe its practical effectiveness, and theoretically justify its learning efficiency.
@article{guan2019generative,
author = {Guan, Jiaqi and Yuan, Ye and Kitani, Kris M and Rhinehart, Nicholas},
journal = {arXiv preprint arXiv:1904.06250},
title = {Generative Hybrid Representations for Activity Forecasting with No-Regret Learning},
year = {2019}
}
Jointly Forecasting and Controlling Behavior by Learning from High-Dimensional Data.
2019.
2019.
Achieving a precise predictive understanding of the future is difficult, yet widely studied in the natural sciences. Significant research activity has been dedicated to building testable models of cause and effect. From a certain view, the ability to forecast the universe is the “holy grail”; the ultimate goal of science. If we had it, we could anticipate, and therefore (at least implicitly) understand all observable phenomena. The human capability to forecast offers complementary motivation. Critical to our intelligence is our ability to plan behaviors by considering how our actions are likely to result in future payoff, especially in the presence of other collaborative and competitive agents. In this work, we seek to computationally model the future in the presence of agent behavior given rich observations of the environment. The brunt of our focus is to reason about what agents could do, instead of other sources of stochasticity. This focus on future agent behavior allows us to tightly couple and jointly perform forecasting and control. The field of Computer Vision (CV) is focused on designing algorithms to automatically understand images, videos, and other perceptual data. However, the field’s effort to-date focuses on non-interactive, present-focused tasks [79, 81, 158, 184]. Most CV contributions are algorithms to answer questions like “what is that”, and “what happened”, rather than “what could happen”, or “how could I achieve X”. Computer Vision has under-explored reasoning about the interactive and decision-based nature of the world. In contrast, Reinforcement Learning (RL) prioritizes modeling interactions and decisions by focusing on how to design algorithms to evoke behavior that maximizes a scalar reward signal. The resulting learning agents, in order to perform well, must have an understanding of how their current behaviors will affect their prospects of future reward. However, in the dominant paradigm of model-free RL [218], agents reason implicitly about the future. In contrast, model-based RL learns one-step dynamics to estimate “what could happen in the near future”. Yet model-based RL primarily focuses on control, rather than explicitly forecasting a single agent (let alone multiple agents). In this thesis, we consider the problem of designing algorithms to enable computational systems to (1) forecast future behavior of intelligent agents given rich observations of their environments, as well as to (2) use this reasoning for control. We believe these two problems should be tightly integrated and jointly considered, and use them to structure this thesis. We define forecasting to be the problem of estimating the set of possible outcomes of a system, whereas control is the problem of producing actions that generate a single outcome of a system. We often use Imitation Learning and Reinforcement Learning to formulate and situate our work. We contribute forecasting and control approaches to excel in diverse, realistic, single-agent, and multi-agent domains. The first part of the thesis focuses on progressively designing more capable forecasting models. We proceed through approaches to (1) forecast single actions of daily behavior by developing matrix factorization models [169], (2) forecast goal-driven action trajectories of daily behavior by developing Online Inverse Reinforcement Learning models [168, 170], (3) forecast motion trajectories of vehicles by developing a deep reversible generative models [171, 174]. The second part of the thesis focuses on progressively designing more capable models that tightly couple forecasting and control. We discuss (4) forecasting as auxiliary supervision for implicitly-planned control [228], (5) forecasting and explicitly planning with the same model [176], and (6) forecasting and planning future interactions of multiple agents [175].
@phdthesis{rhinehart2019jointly,
author = {Rhinehart, Nicholas},
school = {Carnegie Mellon University},
title = {Jointly Forecasting and Controlling Behavior by Learning from High-Dimensional Data},
year = {2019}
}
PRECOG: PREdiction Conditioned On Goals in Visual Multi-Agent Settings.
Proceedings of the IEEE International Conference on Computer Vision, (ICCV), 2019.
Best Paper Award @ ICML 2019 Workshop on AI for Autonomous Driving
Proceedings of the IEEE International Conference on Computer Vision, (ICCV), 2019.
Best Paper Award @ ICML 2019 Workshop on AI for Autonomous Driving
For autonomous vehicles (AVs) to behave appropriately on roads populated by human-driven vehicles, they must be able to reason about the uncertain intentions and decisions of other drivers from rich perceptual information. Towards these capabilities, we present a probabilistic forecasting model of future interactions between a variable number of agents. We perform both standard forecasting and the novel task of conditional forecasting, which reasons about how all agents will likely respond to the goal of a controlled agent (here, the AV). We train models on real and simulated data to forecast vehicle trajectories given past positions and LIDAR. Our evaluation shows that our model is substantially more accurate in multi-agent driving scenarios compared to existing state-of-the-art. Beyond its general ability to perform conditional forecasting queries, we show that our model’s predictions of all agents improve when conditioned on knowledge of the AV’s goal, further illustrating its capability to model agent interactions.
@inproceedings{rhinehart2019precog,
author = {Rhinehart, Nicholas and McAllister, Rowan and Kitani, Kris and Levine, Sergey},
booktitle = {Proceedings of the IEEE International Conference on Computer Vision},
title = {PRECOG: PREdiction Conditioned On Goals in Visual Multi-Agent Settings},
year = {2019}
}
SMiRL: Surprise Minimizing RL in Dynamic Environments.
International Conference on Representation Learning, (ICLR), 2019.
International Conference on Representation Learning, (ICLR), 2019.
Every living organism struggles against disruptive environmental forces to carve out and maintain an orderly niche. We propose that such a struggle to achieve and preserve order might offer a principle for the emergence of useful behaviors in artificial agents. We formalize this idea into an unsupervised reinforcement learning method called surprise minimizing reinforcement learning (SMiRL). SMiRL alternates between learning a density model to evaluate the surprise of a stimulus, and improving the policy to seek more predictable stimuli. The policy seeks out stable and repeatable situations that counteract the environment’s prevailing sources of entropy. This might include avoiding other hostile agents, or finding a stable, balanced pose for a bipedal robot in the face of disturbance forces. We demonstrate that our surprise minimizing agents can successfully play Tetris, Doom, control a humanoid to avoid falls, and navigate to escape enemies in a maze without any task-specific reward supervision. We further show that SMiRL can be used together with standard task rewards to accelerate reward-driven learning.
@inproceedings{berseth2019smirl,
author = {Berseth, Glen and Geng, Daniel and Devin, Coline and Rhinehart, Nicholas and Finn, Chelsea and Jayaraman, Dinesh and Levine, Sergey},
booktitle = {arXiv preprint arXiv:1912.05510},
title = {SMiRL: Surprise Minimizing RL in Dynamic Environments},
year = {2019}
}
First-Person Activity Forecasting from Video with Online Inverse Reinforcement Learning.
IEEE Transactions on Pattern Analysis and Machine Intelligence, (PAMI), 2018.
IEEE Transactions on Pattern Analysis and Machine Intelligence, (PAMI), 2018.
We address the problem of incrementally modeling and forecasting long-term goals of a first-person camera wearer: what the user will do, where they will go, and what goal they seek. In contrast to prior work in trajectory forecasting, our algorithm, Darko, goes further to reason about semantic states (will I pick up an object?), and future goal states that are far in terms of both space and time. Darko learns and forecasts from first-person visual observations of the user’s daily behaviors via an Online Inverse Reinforcement Learning (IRL) approach. Classical IRL discovers only the rewards in a batch setting, whereas Darko discovers the transitions, rewards, and goals of a user from streaming data. Among other results, we show Darko forecasts goals better than competing methods in both noisy and ideal settings, and our approach is theoretically and empirically no-regret.
@article{rhinehart2018first,
author = {Rhinehart, Nicholas and Kitani, Kris},
journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
publisher = {IEEE},
title = {First-Person Activity Forecasting from Video with Online Inverse Reinforcement Learning},
year = {2018}
}
Human-Interactive Subgoal Supervision for Efficient Inverse Reinforcement Learning.
Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, (AAMAS), 2018.
Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, (AAMAS), 2018.
Humans are able to understand and perform complex tasks by strategically structuring tasks into incremental steps or sub-goals. For a robot attempting to learn to perform a sequential task with critical subgoal states, these subgoal states can provide a natural opportunity for interaction with a human expert. This paper analyzes the benefit of incorporating a notion of subgoals into Inverse Reinforcement Learning (IRL) with a Human-In-The-Loop (HITL) framework. The learning process is interactive, with a human expert first providing input in the form of full demonstrations along with some subgoal states. These subgoal states defines a set of sub-tasks for the learning agent to complete in order to achieve the final goal. The learning agent queries for partial demonstrations corresponding to each sub-task as needed when the learning agent struggles with individual sub-task. The proposed Human Interactive IRL (HI-IRL) framework is evaluated on several discrete path-planning tasks. We demonstrate that subgoal-based interactive structuring of the learning task results in significantly more efficient learning, requiring only a fraction of the demonstration data needed for learning the underlying reward function with a baseline IRL model.
@inproceedings{pan2018human,
author = {Pan, Xinlei and Ohn-Bar, Eshed and Rhinehart, Nicholas and Xu, Yan and Shen, Yilin and Kitani, Kris M.},
booktitle = {Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems},
organization = {International Foundation for Autonomous Agents and Multiagent Systems},
pages = {1380--1387},
title = {Human-Interactive Subgoal Supervision for Efficient Inverse Reinforcement Learning},
year = {2018}
}
Learning Neural Parsers with Deterministic Differentiable Imitation Learning.
Conference on Robot Learning, (CoRL), 2018.
Conference on Robot Learning, (CoRL), 2018.
We explore the problem of learning to decompose spatial tasks into segments, as exemplified by the problem of a painting robot covering a large object. Inspired by the ability of classical decision tree algorithms to construct structured partitions of their input spaces, we formulate the problem of decomposing objects into segments as a parsing approach. We make the insight that the derivation of a parse-tree that decomposes the object into segments closely resembles a decision tree constructed by ID3, which can be done when the ground-truth available. We learn to imitate an expert parsing oracle, such that our neural parser can generalize to parse natural images without ground truth. We introduce a novel deterministic policy gradient update, DRAG (i.e., DeteRministically AGgrevate) in the form of a deterministic actor-critic variant of AggreVaTeD, to train our neural parser. From another perspective, our approach is a variant of the Deterministic Policy Gradient suitable for the imitation learning setting. The deterministic policy representation offered by training our neural parser with DRAG allows it to outperform state of the art imitation and reinforcement learning approaches.
@inproceedings{shankar2018learning,
author = {Shankar, Tanmay and Rhinehart, Nicholas and Muelling, Katharina and Kitani, Kris M.},
booktitle = {arXiv:1806.07822},
title = {Learning Neural Parsers with Deterministic Differentiable Imitation Learning},
year = {2018}
}
N2N learning: Network to Network Compression via Policy Gradient Reinforcement Learning.
International Conference on Learning Representations, (ICLR), 2018.
International Conference on Learning Representations, (ICLR), 2018.
While bigger and deeper neural network architectures continue to advance the state-of-the-art for many computer vision tasks, real-world adoption of these networks is impeded by hardware and speed constraints. Conventional model compression methods attempt to address this problem by modifying the architecture manually or using pre-defined heuristics. Since the space of all reduced architectures is very large, modifying the architecture of a deep neural network in this way is a difficult task. In this paper, we tackle this issue by introducing a principled method for learning reduced network architectures in a data-driven way using reinforcement learning. Our approach takes a larger
teacher' network as input and outputs a compressed student’ network derived from the teacher' network. In the first stage of our method, a recurrent policy network aggressively removes layers from the large teacher’ model. In the second stage, another recurrent policy network carefully reduces the size of each remaining layer. The resulting network is then evaluated to obtain a reward – a score based on the accuracy and compression of the network. Our approach uses this reward signal with policy gradients to train the policies to find a locally optimal student network. Our experiments show that we can achieve compression rates of more than 10x for models such as ResNet-34 while maintaining similar performance to the input teacher' network. We also present a valuable transfer learning result which shows that policies which are pre-trained on smaller teacher’ networks can be used to rapidly speed up training on larger `teacher’ networks.@inproceedings{ashok2018n2n,
author = {Ashok, Anubhav and Rhinehart, Nicholas and Beainy, Fares and Kitani, Kris M.},
booktitle = {International Conference on Learning Representations (ICLR)},
title = {N2N learning: Network to Network Compression via Policy Gradient Reinforcement Learning},
year = {2018}
}
R2P2: A Reparameterized Pushforward Policy for Diverse, Precise Generative Path Forecasting.
Proceedings of the European Conference on Computer Vision, (ECCV), 2018.
Proceedings of the European Conference on Computer Vision, (ECCV), 2018.
We propose a method to forecast a vehicle’s ego-motion as a distribution over spatiotemporal paths, conditioned on features (e.g., from LIDAR and images) embedded in an overhead map. The method learns a policy inducing a distribution over simulated trajectories that is both “diverse” (produces most of the likely paths) and “precise” (mostly produces likely paths). This balance is achieved through minimization of a symmetrized cross-entropy between the distribution and demonstration data. By viewing the simulated-outcome distribution as the pushforward of a simple distribution under a simulation operator, we obtain expressions for the cross-entropy metrics that can be efficiently evaluated and differentiated, enabling stochastic-gradient optimization. We propose concrete policy architectures for this model, discuss our evaluation metrics relative to previously-used degenerate metrics, and demonstrate the superiority of our method relative to state-of-the-art methods in both the Kitti dataset and a similar but novel and larger real-world dataset explicitly designed for the vehicle forecasting domain.
@inproceedings{rhinehart2018r2p2,
author = {Rhinehart, Nicholas and Kitani, Kris M. and Vernaza, Paul},
booktitle = {Proceedings of the European Conference on Computer Vision (ECCV)},
pages = {772--788},
title = {R2P2: A Reparameterized Pushforward Policy for Diverse, Precise Generative Path Forecasting},
year = {2018}
}
First-Person Activity Forecasting with Online Inverse Reinforcement Learning.
The IEEE International Conference on Computer Vision, (ICCV), 2017.
Best Paper Honorable Mention
The IEEE International Conference on Computer Vision, (ICCV), 2017.
Best Paper Honorable Mention
We address the problem of incrementally modeling and forecasting long-term goals of a first-person camera wearer: what the user will do, where they will go, and what goal they seek. In contrast to prior work in trajectory forecasting, our algorithm, DARKO, goes further to reason about semantic states (will I pick up an object?), and future goal states that are far in terms of both space and time. DARKO learns and forecasts from first-person visual observations of the user’s daily behaviors via an Online Inverse Reinforcement Learning (IRL) approach. Classical IRL discovers only the rewards in a batch setting, whereas DARKO discovers the states, transitions, rewards, and goals of a user from streaming data. Among other results, we show DARKO forecasts goals better than competing methods in both noisy and ideal settings, and our approach is theoretically and empirically no-regret.
@inproceedings{rhinehart2017first,
author = {Rhinehart, Nicholas and Kitani, Kris M.},
booktitle = {The IEEE International Conference on Computer Vision (ICCV)},
pages = {3716--3725},
title = {First-Person Activity Forecasting with Online Inverse Reinforcement Learning},
year = {2017}
}
Predictive-state decoders: Encoding the future into recurrent networks.
Advances in Neural Information Processing Systems, (NeurIPS), 2017.
Advances in Neural Information Processing Systems, (NeurIPS), 2017.
Recurrent neural networks (RNNs) are a vital modeling technique that rely on internal states learned indirectly by optimization of a supervised, unsupervised, or reinforcement training loss. RNNs are used to model dynamic processes that are characterized by underlying latent states whose form is often unknown, precluding its analytic representation inside an RNN. In the Predictive-State Representation (PSR) literature, latent state processes are modeled by an internal state representation that directly models the distribution of future observations, and most recent work in this area has relied on explicitly representing and targeting sufficient statistics of this probability distribution. We seek to combine the advantages of RNNs and PSRs by augmenting existing state-of-the-art recurrent neural networks with Predictive-State Decoders (PSDs), which add supervision to the network’s internal state representation to target predicting future observations. Predictive-State Decoders are simple to implement and easily incorporated into existing training pipelines via additional loss regularization. We demonstrate the effectiveness of PSDs with experimental results in three different domains: probabilistic filtering, Imitation Learning, and Reinforcement Learning. In each, our method improves statistical performance of state-of-the-art recurrent baselines and does so with fewer iterations and less data.
@inproceedings{venkatraman2017predictive,
author = {Venkatraman, Arun and Rhinehart, Nicholas and Sun, Wen and Pinto, Lerrel and Hebert, Martial and Boots, Byron and Kitani, Kris M. and Bagnell, J. A.},
booktitle = {Advances in Neural Information Processing Systems},
pages = {1172--1183},
title = {Predictive-state decoders: Encoding the future into recurrent networks},
year = {2017}
}
Learning Action Maps of Large Environments Via First-Person Vision.
The IEEE Conference on Computer Vision and Pattern Recognition, (CVPR), 2016.
The IEEE Conference on Computer Vision and Pattern Recognition, (CVPR), 2016.
When people observe and interact with physical spaces, they are able to associate functionality to regions in the environment. Our goal is to automate dense functional understanding of large spaces by leveraging sparse activity demonstrations recorded from an ego-centric viewpoint. The method we describe enables functionality estimation in large scenes where people have behaved, as well as novel scenes where no behaviors are observed. Our method learns and predicts “Action Maps”, which encode the ability for a user to perform activities at various locations. With the usage of an egocentric camera to observe human activities, our method scales with the size of the scene without the need for mounting multiple static surveillance cameras and is well-suited to the task of observing activities up-close. We demonstrate that by capturing appearance-based attributes of the environment and associating these attributes with activity demonstrations, our proposed mathematical framework allows for the prediction of Action Maps in new environments. Additionally, we offer a preliminary glance of the applicability of Action Maps by demonstrating a proof-of concept application in which they are used in concert with activity detections to perform localization.
@inproceedings{rhinehart2016learning,
author = {Rhinehart, Nicholas and Kitani, Kris M.},
booktitle = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
title = {Learning Action Maps of Large Environments Via First-Person Vision},
year = {2016}
}
Visual chunking: A list prediction framework for region-based object detection.
IEEE International Conference on Robotics and Automation, (ICRA), 2015.
IEEE International Conference on Robotics and Automation, (ICRA), 2015.
We consider detecting objects in an image by iteratively selecting from a set of arbitrarily shaped candidate regions. Our generic approach, which we term visual chunking, reasons about the locations of multiple object instances in an image while expressively describing object boundaries. We design an optimization criterion for measuring the performance of a list of such detections as a natural extension to a common per-instance metric. We present an efficient algorithm with provable performance for building a high-quality list of detections from any candidate set of region-based proposals. We also develop a simple class-specific algorithm to generate a candidate region instance in near-linear time in the number of low-level superpixels that outperforms other region generating methods. In order to make predictions on novel images at testing time without access to ground truth, we develop learning approaches to emulate these algorithms’ behaviors. We demonstrate that our new approach outperforms sophisticated baselines on benchmark datasets.
@inproceedings{rhinehart2015visual,
author = {Rhinehart, Nicholas and Zhou, Jiaji and Hebert, Martial and Bagnell, J Andrew},
booktitle = {2015 IEEE International Conference on Robotics and Automation (ICRA)},
organization = {IEEE},
pages = {5448--5454},
title = {Visual chunking: A list prediction framework for region-based object detection},
year = {2015}
}