Publications

All Publication Topics Autonomous Driving Autonomous Exploration Benchmarks Compression Computer Vision First-Person Video Forecasting Generative Models Imitation Learning Intrinsic Control Machine Learning Offroad Navigation Paper Award Planning Reinforcement Learning Reward Learning Robotics

DR-MPC: Deep Residual Model Predictive Control for Real-world Social Navigation

James R. Han, Hugues Thomas, Jian Zhang, Nicholas Rhinehart, Timothy D. Barfoot.
DR-MPC: Deep Residual Model Predictive Control for Real-world Social Navigation.
arXiv preprint arXiv:2410.10646, (arXiv), 2024.

Computer Vision Forecasting Machine Learning Robotics Reinforcement Learning

CARFF: Conditional Auto-encoded Radiance Field for 3D Scene Forecasting

Jiezhi Yang, Khushi Desai, Charles Packer, Harshil Bhatia, Nicholas Rhinehart, Rowan McAllister, Joseph Gonzalez.
CARFF: Conditional Auto-encoded Radiance Field for 3D Scene Forecasting.
European Conference on Computer Vision, (ECCV), 2024.

Autonomous Driving Computer Vision Forecasting Generative Models Machine Learning Robotics

The Waymo Open Sim Agents Challenge

Nico Montali, John Lambert, Paul Mougin, Alex Kuefler, Nicholas Rhinehart, Michelle Li, Cole Gulino, Tristan Emrich, Zoey Yang, Shimon Whiteson, Brandyn White, Dragomir Anguelov.
The Waymo Open Sim Agents Challenge.
Advances in Neural Information Processing Systems, (NeurIPS), 2023.

Autonomous Driving Benchmarks Computer Vision Forecasting Imitation Learning Machine Learning Robotics

Is anyone there? learning a planner contingent on perceptual uncertainty

Charles Packer, Nicholas Rhinehart, Rowan McAllister, Matthew a Wright, Xin Wang, Jeff He, Sergey Levine, Joseph E Gonzalez.
Is anyone there? learning a planner contingent on perceptual uncertainty.
Conference on Robot Learning, (CoRL), 2023.

Autonomous Driving Computer Vision Forecasting Generative Models Machine Learning Planning Robotics

S2Net: Stochastic Sequential Pointcloud Forecasting

Xinshuo Weng, Junyu Nan, Kuan-Hui Lee, Rowan McAllister, Adrien Gaidon, Nicholas Rhinehart, Kris Kitani.
S2Net: Stochastic Sequential Pointcloud Forecasting.
European Conference on Computer Vision, (ECCV), 2022.

Autonomous Driving Forecasting Generative Models Machine Learning

Offline reinforcement learning for visual navigation

Dhruv Shah, Arjun Bhorkar, Hrish Leen, Ilya Kostrikov, Nicholas Rhinehart, Sergey Levine.
Offline reinforcement learning for visual navigation.
Conference on Robot Learning, (CoRL), 2022.

Computer Vision Machine Learning Offroad Navigation Reinforcement Learning Robotics

Hybrid imitative planning with geometric and predictive costs in off-road environments

Nitish Dashora, Daniel Shin, Dhruv Shah, Henry Leopold, David Fan, Ali Agha-Mohammadi, Nicholas Rhinehart, Sergey Levine.
Hybrid imitative planning with geometric and predictive costs in off-road environments.
International Conference on Robotics and Automation, (ICRA), 2022.

Imitation Learning Machine Learning Planning Offroad Navigation Robotics

Paul Vernaza, Nicholas Rhinehart.
Traffic prediction with reparameterized pushforward policy for autonomous vehicles.
U.S. Patent No. 11,189,171, 2021.

Ving: Learning open-world navigation with visual goals

Dhruv Shah, Benjamin Eysenbach, Gregory Kahn, Nicholas Rhinehart, Sergey Levine.
Ving: Learning open-world navigation with visual goals.
IEEE International Conference on Robotics and Automation, (ICRA), 2021.

Autonomous Driving Computer Vision Machine Learning Offroad Navigation Planning Robotics

Rapid exploration for open-world navigation with latent goal models

Dhruv Shah, Benjamin Eysenbach, Gregory Kahn, Nicholas Rhinehart, Sergey Levine.
Rapid exploration for open-world navigation with latent goal models.
Conference on Robot Learning, (CoRL), 2021.

Autonomous Exploration Offroad Navigation Robotics Planning

Inverting the pose forecasting pipeline with SPF2: Sequential pointcloud forecasting for sequential pose forecasting

Xinshuo Weng, Jianren Wang, Sergey Levine, Kris Kitani, Nicholas Rhinehart.
Inverting the pose forecasting pipeline with SPF2: Sequential pointcloud forecasting for sequential pose forecasting.
Conference on robot learning, (CoRL), 2021.

Autonomous Driving Computer Vision Generative Models Machine Learning Robotics

Information is power: Intrinsic control via information capture

Nicholas Rhinehart, Jenny Wang, Glen Berseth, John Co-Reyes, Danijar Hafner, Chelsea Finn, Sergey Levine.
Information is power: Intrinsic control via information capture.
Advances in Neural Information Processing Systems, (NeurIPS), 2021.

Autonomous Exploration Intrinsic Control Generative Models Reinforcement Learning

Explore and control with adversarial surprise

Arnaud Fickinger, Natasha Jaques, Samyak Parajuli, Michael Chang, Nicholas Rhinehart, Glen Berseth, Stuart Russell, Sergey Levine.
Explore and control with adversarial surprise.
arXiv preprint arXiv:2107.07394, 2021.

Contingencies from observations: Tractable contingency planning with learned behavior models

Nicholas Rhinehart, Jeff He, Charles Packer, Matthew a Wright, Rowan McAllister, Joseph E Gonzalez, Sergey Levine.
Contingencies from observations: Tractable contingency planning with learned behavior models.
IEEE International Conference on Robotics and Automation, (ICRA), 2021.

Autonomous Driving Computer Vision Forecasting Imitation Learning Machine Learning Planning Robotics

Paul Vernaza, Wongun Choi, Nicholas Rhinehart.
Generative adversarial inverse trajectory optimization for probabilistic vehicle forecasting.
U.S. Patent 10,739,773, 2020.

Parrot: Data-driven behavioral priors for reinforcement learning

Avi Singh, Huihan Liu, Gaoyue Zhou, Albert Yu, Nicholas Rhinehart, Sergey Levine.
Parrot: Data-driven behavioral priors for reinforcement learning.
International Conference on Learning Representations, (ICLR), 2020.

Generative Models Imitation Learning Machine Learning Robotics

Deep Imitative Models for Flexible Inference, Planning, and Control

Nicholas Rhinehart, Rowan McAllister, Sergey Levine.
Deep Imitative Models for Flexible Inference, Planning, and Control.
International Conference on Learning Representations, (ICLR), 2020.

Autonomous Driving Computer Vision Forecasting Generative Models Imitation Learning Planning Reward Learning

Conservative safety critics for exploration

Homanga Bharadhwaj, Aviral Kumar, Nicholas Rhinehart, Sergey Levine, Florian Shkurti, Animesh Garg.
Conservative safety critics for exploration.
International Conference on Represetation Learning, (ICLR), 2020.

Can autonomous vehicles identify, recover from, and adapt to distribution shifts?

Angelos Filos, Panagiotis Tigkas, Rowan McAllister, Nicholas Rhinehart, Sergey Levine, Yarin Gal.
Can autonomous vehicles identify, recover from, and adapt to distribution shifts?.
International Conference on Machine Learning, (ICML), 2020.

Autonomous Driving Computer Vision Imitation Learning Planning Machine Learning

SMiRL: Surprise Minimizing RL in Dynamic Environments

Glen Berseth, Daniel Geng, Coline Devin, Nicholas Rhinehart, Chelsea Finn, Dinesh Jayaraman, Sergey Levine.
SMiRL: Surprise Minimizing RL in Dynamic Environments.
International Conference on Representation Learning, (ICLR), 2019.

Reinforcement Learning Intrinsic Control Machine Learning Autonomous Exploration

PRECOG: PREdiction Conditioned On Goals in Visual Multi-Agent Settings

Nicholas Rhinehart, Rowan McAllister, Kris Kitani, Sergey Levine.
PRECOG: PREdiction Conditioned On Goals in Visual Multi-Agent Settings.
Proceedings of the IEEE International Conference on Computer Vision, (ICCV), 2019.
Best Paper Award @ ICML 2019 Workshop on AI for Autonomous Driving

Paper Award Forecasting Generative Models Autonomous Driving

Jointly Forecasting and Controlling Behavior by Learning from High-Dimensional Data

Nicholas Rhinehart.
Jointly Forecasting and Controlling Behavior by Learning from High-Dimensional Data.
2019.

Autonomous Driving Forecasting Generative Models Imitation Learning Planning Reinforcement Learning Reward Learning

Generative Hybrid Representations for Activity Forecasting with No-Regret Learning

Jiaqi Guan, Ye Yuan, Kris M Kitani, Nicholas Rhinehart.
Generative Hybrid Representations for Activity Forecasting with No-Regret Learning.
Conference on Computer Vision and Pattern Recognition, (CVPR), 2019.

Computer Vision Imitation Learning First-Person Video Forecasting Generative Models Machine Learning

Directed-Info GAIL: Learning Hierarchical Policies from Unsegmented Demonstrations using Directed Information

Arjun Sharma, Mohit Sharma, Nicholas Rhinehart, Kris M Kitani.
Directed-Info GAIL: Learning Hierarchical Policies from Unsegmented Demonstrations using Directed Information.
International Conference on Learning Representations (ICLR), (ICLR), 2019.

Imitation Learning Robotics

R2P2: A Reparameterized Pushforward Policy for Diverse, Precise Generative Path Forecasting

Nicholas Rhinehart, Kris M. Kitani, Paul Vernaza.
R2P2: A Reparameterized Pushforward Policy for Diverse, Precise Generative Path Forecasting.
Proceedings of the European Conference on Computer Vision, (ECCV), 2018.

Autonomous Driving Imitation Learning Forecasting Generative Models Machine Learning Reward Learning

N2N learning: Network to Network Compression via Policy Gradient Reinforcement Learning

Anubhav Ashok, Nicholas Rhinehart, Fares Beainy, Kris M. Kitani.
N2N learning: Network to Network Compression via Policy Gradient Reinforcement Learning.
International Conference on Learning Representations, (ICLR), 2018.

Compression Machine Learning Reinforcement Learning

Learning Neural Parsers with Deterministic Differentiable Imitation Learning

Tanmay Shankar, Nicholas Rhinehart, Katharina Muelling, Kris M. Kitani.
Learning Neural Parsers with Deterministic Differentiable Imitation Learning.
Conference on Robot Learning, (CoRL), 2018.

Imitation Learning Machine Learning

Human-Interactive Subgoal Supervision for Efficient Inverse Reinforcement Learning

Xinlei Pan, Eshed Ohn-Bar, Nicholas Rhinehart, Yan Xu, Yilin Shen, Kris M. Kitani.
Human-Interactive Subgoal Supervision for Efficient Inverse Reinforcement Learning.
Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, (AAMAS), 2018.

Imitation Learning Machine Learning Reward Learning

First-Person Activity Forecasting from Video with Online Inverse Reinforcement Learning

Nicholas Rhinehart, Kris Kitani.
First-Person Activity Forecasting from Video with Online Inverse Reinforcement Learning.
IEEE Transactions on Pattern Analysis and Machine Intelligence, (PAMI), 2018.

Imitation Learning Forecasting Machine Learning Reward Learning

Predictive-state decoders: Encoding the future into recurrent networks

Arun Venkatraman, Nicholas Rhinehart, Wen Sun, Lerrel Pinto, Martial Hebert, Byron Boots, Kris M. Kitani, J. A. Bagnell.
Predictive-state decoders: Encoding the future into recurrent networks.
Advances in Neural Information Processing Systems, (NeurIPS), 2017.

Imitation Learning Forecasting Machine Learning Reinforcement Learning

First-Person Activity Forecasting with Online Inverse Reinforcement Learning

Nicholas Rhinehart, Kris M. Kitani.
First-Person Activity Forecasting with Online Inverse Reinforcement Learning.
The IEEE International Conference on Computer Vision, (ICCV), 2017.
Best Paper Honorable Mention

Paper Award Imitation Learning Reward Learning First-Person Video

Learning Action Maps of Large Environments Via First-Person Vision

Nicholas Rhinehart, Kris M. Kitani.
Learning Action Maps of Large Environments Via First-Person Vision.
The IEEE Conference on Computer Vision and Pattern Recognition, (CVPR), 2016.

Computer Vision First-Person Video Machine Learning Reward Learning

Visual chunking: A list prediction framework for region-based object detection

Nicholas Rhinehart, Jiaji Zhou, Martial Hebert, J Andrew Bagnell.
Visual chunking: A list prediction framework for region-based object detection.
IEEE International Conference on Robotics and Automation, (ICRA), 2015.

Computer Vision Imitation Learning Machine Learning