Computer Vision

OccSim: Multi-kilometer Simulation with Long-horizon Occupancy World Models.

arXiv:2603.28887, (arXiv), 2026.
Data-driven autonomous driving simulation has long been constrained by its heavy reliance on pre-recorded driving logs or spatial priors, such as HD maps. This fundamental dependency severely limits scalability, restricting open-ended generation capabilities to the finite scale of existing collected datasets. To break this bottleneck, we present OccSim, the first occupancy world model-driven 3D simulator. OccSim obviates the requirement for continuous logs or HD maps; conditioned only on a single initial frame and a sequence of future ego-actions, it can stably generate over 3,000 continuous frames, enabling the continuous construction of large-scale 3D occupancy maps spanning over 4 kilometers for simulation. This represents an >80x improvement in stable generation length over previous state-of-the-art occupancy world models. OccSim is powered by two modules: W-DiT based static occupancy world model and the Layout Generator. W-DiT handles the ultra-long-horizon generation of static environments by explicitly introducing known rigid transformations in architecture design, while the Layout Generator populates the dynamic foreground with reactive agents based on the synthesized road topology. With these designs, OccSim can synthesize massive, diverse simulation streams. Extensive experiments demonstrate its downstream utility: data collected directly from OccSim can pre-train 4D semantic occupancy forecasting models to achieve up to 67% zero-shot performance on unseen data, outperforming previous asset-based simulator by 11%. When scaling the OccSim dataset to 5x the size, the zero-shot performance increases to about 74%, while the improvement over asset-based simulators expands to 22.1%.
@article{liu2026occsim,
  title = {OccSim: Multi-kilometer Simulation with Long-horizon Occupancy World Models},
  author = {Liu, Tianran and Zhao, Shengwen and Pourkeshavarz, Mozhgan and Li, Weican and Rhinehart, Nicholas},
  journal = {arXiv preprint arXiv:2603.28887},
  year = {2026},
  doi = {10.48550/arXiv.2603.28887},
  url = {https://arxiv.org/abs/2603.28887}
}
AutoWorld: Scaling Multi-Agent Traffic Simulation with Self-Supervised World Models.

arXiv:2603.28963, (arXiv), 2026.
Multi-agent traffic simulation is central to developing and testing autonomous driving systems. Recent data-driven simulators have achieved promising results, but rely heavily on supervised learning from labeled trajectories or semantic annotations, making it costly to scale their performance. Meanwhile, large amounts of unlabeled sensor data can be collected at scale but remain largely unused by existing traffic simulation frameworks. This raises a key question: How can a method harness unlabeled data to improve traffic simulation performance? In this work, we propose AutoWorld, a traffic simulation framework that employs a world model learned from unlabeled occupancy representations of LiDAR data. Given world model samples, AutoWorld constructs a coarse-to-fine predictive scene context as input to a multi-agent motion generation model. To promote sample diversity, AutoWorld uses a cascaded Determinantal Point Process framework to guide the sampling processes of both the world model and the motion model. Furthermore, we designed a motion-aware latent supervision objective that enhances AutoWorld’s representation of scene dynamics. Experiments on the WOSAC benchmark show that AutoWorld ranks first on the leaderboard according to the primary Realism Meta Metric (RMM). We further show that simulation performance consistently improves with the inclusion of unlabeled LiDAR data, and study the efficacy of each component with ablations. Our method paves the way for scaling traffic simulation realism without additional labeling. Our project page contains additional visualizations and released code.
@article{pourkeshavarz2026autoworld,
  title = {AutoWorld: Scaling Multi-Agent Traffic Simulation with Self-Supervised World Models},
  author = {Pourkeshavarz, Mozhgan and Liu, Tianran and Rhinehart, Nicholas},
  journal = {arXiv preprint arXiv:2603.28963},
  year = {2026},
  doi = {10.48550/arXiv.2603.28963},
  url = {https://arxiv.org/abs/2603.28963}
}
Foundational LiDAR world model occupancy forecasting result animation
Towards foundational LiDAR world models with efficient latent flow matching.

Advances in Neural Information Processing Systems, (NeurIPS), 2025.
LiDAR-based world models offer more structured and geometry-aware representations than their image-based counterparts. However, existing LiDAR world models are narrowly trained; each model excels only in the domain for which it was built. This raises a critical question: can we develop LiDAR world models that exhibit strong transferability across multiple domains? To answer this, we conduct the first systematic domain transfer study across three demanding scenarios: (i) outdoor to indoor generalization, (ii) sparse- to dense-beam adaptation, and (iii) non-semantic to semantic transfer. Given different amounts of fine-tuning data, our experiments show that a single pretrained model can achieve up to 11% absolute improvement (83% relative) over training from scratch and outperforms training from scratch in 30/36 of our comparisons. This transferability significantly reduces the reliance on manually annotated data for semantic occupancy forecasting: our method exceeds previous baselines with only 5% of the labeled training data of prior work. We also observed inefficiencies of current generative-model-based LiDAR world models, mainly through their under-compression of LiDAR data and inefficient training objectives. To address these issues, we propose a latent conditional flow matching (CFM)-based framework that achieves state-of-the-art reconstruction accuracy using only half the training data and a compression ratio 6 times higher than that of prior methods. Our model also achieves SOTA performance on semantic occupancy forecasting while being 1.98x-23x more computationally efficient (a 1.1x-3.9x FPS speedup) than previous methods.
@article{liu2025foundationallidarworldmodels,
      title={Towards foundational LiDAR world models with efficient latent flow matching}, 
      author={Tianran Liu and Shengwen Zhao and Nicholas Rhinehart},
      year={2025},
      eprint={2506.23434},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2506.23434}, 
}
DR-MPC: Deep Residual Model Predictive Control for Real-world Social Navigation.

IEEE Robotics and Automation Letters, (RA-L), 2025.
How can a robot safely navigate around people with complex motion patterns? Deep Reinforcement Learning (DRL) in simulation holds some promise, but much prior work relies on simulators that fail to capture the nuances of real human motion. Thus, we propose Deep Residual Model Predictive Control (DR-MPC) to enable robots to quickly and safely perform DRL from real-world crowd navigation data. By blending MPC with model-free DRL, DR-MPC overcomes the DRL challenges of large data requirements and unsafe initial behavior. DR-MPC is initialized with MPC-based path tracking, and gradually learns to interact more effectively with humans. To further accelerate learning, a safety component estimates out-of-distribution states to guide the robot away from likely collisions. In simulation, we show that DR-MPC substantially outperforms prior work, including traditional DRL and residual DRL models. Hardware experiments show our approach successfully enables a robot to navigate a variety of crowded situations with few errors using less than 4 hours of training data.
@article{han2024dr,
  title={DR-MPC: Deep Residual Model Predictive Control for Real-world Social Navigation},
  author={Han, James R and Thomas, Hugues and Zhang, Jian and Rhinehart, Nicholas and Barfoot, Timothy D},
  journal={IEEE Robotics and Automation Letters (RA-L)},
  year={2025}
}

CARFF: Conditional Auto-encoded Radiance Field for 3D Scene Forecasting
CARFF: Conditional Auto-encoded Radiance Field for 3D Scene Forecasting.

European Conference on Computer Vision, (ECCV), 2024.
We propose CARFF, a method for predicting future 3D scenes given past observations. Our method maps 2D ego-centric images to a distribution over plausible 3D latent scene configurations and predicts the evolution of hypothesized scenes through time. Our latents condition a global Neural Radiance Field (NeRF) to represent a 3D scene model, enabling explainable predictions and straightforward downstream planning. This approach models the world as a POMDP and considers complex scenarios of uncertainty in environmental states and dynamics. Specifically, we employ a two-stage training of Pose-Conditional-VAE and NeRF to learn 3D representations, and auto-regressively predict latent scene representations utilizing a mixture density network. We demonstrate the utility of our method in scenarios using the CARLA driving simulator, where CARFF enables efficient trajectory and contingency planning in complex multi-agent autonomous driving scenarios involving occlusions.
@article{yang2024carff,
 author = {Yang, Jiezhi and Desai, Khushi and Packer, Charles and Bhatia, Harshil and Rhinehart, Nicholas and McAllister, Rowan and Gonzalez, Joseph},
 journal = {arXiv preprint arXiv:2401.18075},
 title = {CARFF: Conditional Auto-encoded Radiance Field for 3D Scene Forecasting},
 year = {2024}
}

The Waymo Open Sim Agents Challenge
The Waymo Open Sim Agents Challenge.

Advances in Neural Information Processing Systems, (NeurIPS), 2023.
Simulation with realistic, interactive agents represents a key task for autonomous vehicle software development. In this work, we introduce the Waymo Open Sim Agents Challenge (WOSAC). WOSAC is the first public challenge to tackle this task and propose corresponding metrics. The goal of the challenge is to stimulate the design of realistic simulators that can be used to evaluate and train a behavior model for autonomous driving. We outline our evaluation methodology, present results for a number of different baseline simulation agent methods, and analyze several submissions to the 2023 competition which ran from March 16, 2023 to May 23, 2023. The WOSAC evaluation server remains open for submissions and we discuss open problems for the task.
@article{montali2024waymo,
 author = {Montali, Nico and Lambert, John and Mougin, Paul and Kuefler, Alex and Rhinehart, Nicholas and Li, Michelle and Gulino, Cole and Emrich, Tristan and Yang, Zoey and Whiteson, Shimon and others},
 journal = {Advances in Neural Information Processing Systems},
 title = {The waymo open sim agents challenge},
 volume = {36},
 year = {2024}
}

Is anyone there? learning a planner contingent on perceptual uncertainty.

Conference on Robot Learning, (CoRL), 2023.
Robots in complex multi-agent environments should reason about the intentions of observed and currently unobserved agents. In this paper, we present a new learning-based method for prediction and planning in complex multi-agent environments where the states of the other agents are partially-observed. Our approach, Active Visual Planning (AVP), uses high-dimensional observations to learn a flow-based generative model of multi-agent joint trajectories, including unobserved agents that may be revealed in the near future, depending on the robot’s actions. Our predictive model is implemented using deep neural networks that map raw observations to future detection and pose trajectories and is learned entirely offline using a dataset of recorded observations (not ground-truth states). Once learned, our predictive model can be used for contingency planning over the potential existence, intentions, and positions of unobserved agents. We demonstrate the effectiveness of AVP on a set of autonomous driving environments inspired by real-world scenarios that require reasoning about the existence of other unobserved agents for safe and efficient driving. In these environments, AVP achieves optimal closed-loop performance, while methods that do not reason about potential unobserved agents exhibit either overconfident or underconfident behavior.
@inproceedings{packer2023anyone,
 author = {Packer, Charles and Rhinehart, Nicholas and McAllister, Rowan Thomas and Wright, Matthew A and Wang, Xin and He, Jeff and Levine, Sergey and Gonzalez, Joseph E},
 booktitle = {Conference on Robot Learning},
 organization = {PMLR},
 pages = {1607--1617},
 title = {Is anyone there? learning a planner contingent on perceptual uncertainty},
 year = {2023}
}

Offline reinforcement learning for visual navigation.

Conference on Robot Learning, (CoRL), 2022.
Reinforcement learning can enable robots to navigate to distant goals while optimizing user-specified reward functions, including preferences for following lanes, staying on paved paths, or avoiding freshly mowed grass. However, online learning from trial-and-error for real-world robots is logistically challenging, and methods that instead can utilize existing datasets of robotic navigation data could be significantly more scalable and enable broader generalization. In this paper, we present ReViND, the first offline RL system for robotic navigation that can leverage previously collected data to optimize user-specified reward functions in the real-world. We evaluate our system for off-road navigation without any additional data collection or fine-tuning, and show that it can navigate to distant goals using only offline training from this dataset, and exhibit behaviors that qualitatively differ based on the user-specified reward function.
@article{shah2022offline,
 author = {Shah, Dhruv and Bhorkar, Arjun and Leen, Hrish and Kostrikov, Ilya and Rhinehart, Nick and Levine, Sergey},
 journal = {arXiv preprint arXiv:2212.08244},
 title = {Offline reinforcement learning for visual navigation},
 year = {2022}
}

ViNG open-world navigation result animation
Ving: Learning open-world navigation with visual goals.

IEEE International Conference on Robotics and Automation, (ICRA), 2021.
We propose a learning-based navigation system for reaching visually indicated goals and demonstrate this system on a real mobile robot platform. Learning provides an appealing alternative to conventional methods for robotic navigation: instead of reasoning about environments in terms of geometry and maps, learning can enable a robot to learn about navigational affordances, understand what types of obstacles are traversable (e.g., tall grass) or not (e.g., walls), and generalize over patterns in the environment. However, unlike conventional planning algorithms, it is harder to change the goal for a learned policy during deployment. We propose a method for learning to navigate towards a goal image of the desired destination. By combining a learned policy with a topological graph constructed out of previously observed data, our system can determine how to reach this visually indicated goal even in the presence of variable appearance and lighting. Three key insights, waypoint proposal, graph pruning and negative mining, enable our method to learn to navigate in real-world environments using only offline data, a setting where prior methods struggle. We instantiate our method on a real outdoor ground robot and show that our system, which we call ViNG, outperforms previously-proposed methods for goal-conditioned reinforcement learning, including other methods that incorporate reinforcement learning and search. We also study how ViNG generalizes to unseen environments and evaluate its ability to adapt to such an environment with growing experience. Finally, we demonstrate ViNG on a number of real-world applications, such as last-mile delivery and warehouse inspection. We encourage the reader to visit the project website for videos of our experiments and demonstrations 1 .
@inproceedings{shah2021ving,
 author = {Shah, Dhruv and Eysenbach, Benjamin and Kahn, Gregory and Rhinehart, Nicholas and Levine, Sergey},
 booktitle = {2021 IEEE International Conference on Robotics and Automation (ICRA)},
 organization = {IEEE},
 pages = {13215--13222},
 title = {Ving: Learning open-world navigation with visual goals},
 year = {2021}
}

Sequential pointcloud forecasting result animation
Inverting the pose forecasting pipeline with SPF2: Sequential pointcloud forecasting for sequential pose forecasting.

Conference on robot learning, (CoRL), 2021.
Many autonomous systems forecast aspects of the future in order to aid decision-making. For example, self-driving vehicles and robotic manipulation systems often forecast future object poses by first detecting and tracking objects. However, this detect-then-forecast pipeline is expensive to scale, as pose forecasting algorithms typically require labeled sequences of object poses, which are costly to obtain in 3D space. Can we scale performance without requiring additional labels? We hypothesize yes, and propose inverting the detect-then-forecast pipeline. Instead of detecting, tracking and then forecasting the objects, we propose to first forecast 3D sensor data (e.g., point clouds with $100$k points) and then detect/track objects on the predicted point cloud sequences to obtain future poses, i.e., a forecast-then-detect pipeline. This inversion makes it less expensive to scale pose forecasting, as the sensor data forecasting task requires no labels. Part of this work’s focus is on the challenging first step –Sequential Pointcloud Forecasting (SPF), for which we also propose an effective approach, SPFNet. To compare our forecast-then-detect pipeline relative to the detect-then-forecast pipeline, we propose an evaluation procedure and two metrics. Through experiments on a robotic manipulation dataset and two driving datasets, we show that SPFNet is effective for the SPF task, our forecast-then-detect pipeline outperforms the detect-then-forecast approaches to which we compared, and that pose forecasting performance improves with the addition of unlabeled data.
@inproceedings{weng2021inverting,
 author = {Weng, Xinshuo and Wang, Jianren and Levine, Sergey and Kitani, Kris and Rhinehart, Nicholas},
 booktitle = {Conference on robot learning},
 organization = {PMLR},
 pages = {11--20},
 title = {Inverting the pose forecasting pipeline with SPF2: Sequential pointcloud forecasting for sequential pose forecasting},
 year = {2021}
}

Contingencies from observations: Tractable contingency planning with learned behavior models.

IEEE International Conference on Robotics and Automation, (ICRA), 2021.
Humans have a remarkable ability to accurately reason about future events, including the behaviors and states of mind of other agents. Consider driving a car through a busy intersection: it is necessary to reason about the physics of the vehicle, the intentions of other drivers, and their beliefs about your own intentions. For example, if you signal a turn, another driver might yield to you; or if you enter the passing lane, another driver might decelerate to give you room to merge in front. Competent drivers must plan how they can safely react to a variety of potential future behaviors of other agents before they make their next move. This requires contingency planning: explicitly planning a set of conditional actions that depend on the stochastic outcome of future events. In this work, we develop a general-purpose contingency planner that is learned end-to-end using high-dimensional scene observations and low-dimensional behavioral observations. We use a conditional autoregressive flow model for contingency planning. We show how this model can tractably learn contingencies from behavioral observations. We developed a closed-loop control benchmark of realistic multi-agent scenarios in a driving simulator (CARLA), on which we compare our method to various noncontingent methods that reason about multi-agent future behavior, and find that our contingency planning method achieves qualitatively and quantitatively superior performance.
@inproceedings{rhinehart2021contingencies,
 author = {Rhinehart, Nicholas and He, Jeff and Packer, Charles and Wright, Matthew A and McAllister, Rowan and Gonzalez, Joseph E and Levine, Sergey},
 booktitle = {2021 IEEE International Conference on Robotics and Automation (ICRA)},
 organization = {IEEE},
 pages = {13663--13669},
 title = {Contingencies from observations: Tractable contingency planning with learned behavior models},
 year = {2021}
}

Deep Imitative Models for Flexible Inference, Planning, and Control.

International Conference on Learning Representations, (ICLR), 2020.
Imitation Learning (IL) is an appealing approach to learn desirable autonomous behavior. However, directing IL to achieve arbitrary goals is difficult. In contrast, planning-based algorithms use dynamics models and reward functions to achieve goals. Yet, reward functions that evoke desirable behavior are often difficult to specify. In this paper, we propose Imitative Models to combine the benefits of IL and goal-directed planning. Imitative Models are probabilistic predictive models of desirable behavior able to plan interpretable expert-like trajectories to achieve specified goals. We derive families of flexible goal objectives, including constrained goal regions, unconstrained goal sets, and energy-based goals. We show that our method can use these objectives to successfully direct behavior. Our method substantially outperforms six IL approaches and a planning-based approach in a dynamic simulated autonomous driving task, and is efficiently learned from expert demonstrations without online data collection. We also show our approach is robust to poorly specified goals, such as goals on the wrong side of the road.
@inproceedings{rhinehart2020deep,
 author = {Rhinehart, Nicholas and McAllister, Rowan and Levine, Sergey},
 booktitle = {International Conference on Learning Representations (ICLR)},
 title = {Deep Imitative Models for Flexible Inference, Planning, and Control},
 year = {2020}
}

Robust imitative planning result animation
Can autonomous vehicles identify, recover from, and adapt to distribution shifts?.

International Conference on Machine Learning, (ICML), 2020.
Out-of-training-distribution (OOD) scenarios are a common challenge of learning agents at deployment, typically leading to arbitrary deductions and poorly-informed decisions. In principle, detection of and adaptation to OOD scenes can mitigate their adverse effects. In this paper, we highlight the limitations of current approaches to novel driving scenes and propose an epistemic uncertainty-aware planning method, called \emph{robust imitative planning} (RIP). Our method can detect and recover from some distribution shifts, reducing the overconfident and catastrophic extrapolations in OOD scenes. If the model’s uncertainty is too great to suggest a safe course of action, the model can instead query the expert driver for feedback, enabling sample-efficient online adaptation, a variant of our method we term \emph{adaptive robust imitative planning} (AdaRIP). Our methods outperform current state-of-the-art approaches in the nuScenes \emph{prediction} challenge, but since no benchmark evaluating OOD detection and adaption currently exists to assess \emph{control}, we introduce an autonomous car novel-scene benchmark, \texttt{CARNOVEL}, to evaluate the robustness of driving agents to a suite of tasks with distribution shifts.
@inproceedings{filos2020can,
 author = {Filos, Angelos and Tigkas, Panagiotis and McAllister, Rowan and Rhinehart, Nicholas and Levine, Sergey and Gal, Yarin},
 booktitle = {International Conference on Machine Learning},
 organization = {PMLR},
 pages = {3145--3153},
 title = {Can autonomous vehicles identify, recover from, and adapt to distribution shifts?},
 year = {2020}
}

Generative Hybrid Representations for Activity Forecasting with No-Regret Learning.

Conference on Computer Vision and Pattern Recognition, (CVPR), 2019.
Automatically reasoning about future human behaviors is a difficult problem but has significant practical applications to assistive systems. Part of this difficulty stems from learning systems’ inability to represent all kinds of behaviors. Some behaviors, such as motion, are best described with continuous representations, whereas others, such as picking up a cup, are best described with discrete representations. Furthermore, human behavior is generally not fixed: people can change their habits and routines. This suggests these systems must be able to learn and adapt continuously. In this work, we develop an efficient deep generative model to jointly forecast a person’s future discrete actions and continuous motions. On a large-scale egocentric dataset, EPIC-KITCHENS, we observe our method generates high-quality and diverse samples while exhibiting better generalization than related generative models. Finally, we propose a variant to continually learn our model from streaming data, observe its practical effectiveness, and theoretically justify its learning efficiency.
@article{guan2019generative,
 author = {Guan, Jiaqi and Yuan, Ye and Kitani, Kris M and Rhinehart, Nicholas},
 journal = {arXiv preprint arXiv:1904.06250},
 title = {Generative Hybrid Representations for Activity Forecasting with No-Regret Learning},
 year = {2019}
}

Learning Action Maps of Large Environments Via First-Person Vision.

The IEEE Conference on Computer Vision and Pattern Recognition, (CVPR), 2016.
When people observe and interact with physical spaces, they are able to associate functionality to regions in the environment. Our goal is to automate dense functional understanding of large spaces by leveraging sparse activity demonstrations recorded from an ego-centric viewpoint. The method we describe enables functionality estimation in large scenes where people have behaved, as well as novel scenes where no behaviors are observed. Our method learns and predicts “Action Maps”, which encode the ability for a user to perform activities at various locations. With the usage of an egocentric camera to observe human activities, our method scales with the size of the scene without the need for mounting multiple static surveillance cameras and is well-suited to the task of observing activities up-close. We demonstrate that by capturing appearance-based attributes of the environment and associating these attributes with activity demonstrations, our proposed mathematical framework allows for the prediction of Action Maps in new environments. Additionally, we offer a preliminary glance of the applicability of Action Maps by demonstrating a proof-of concept application in which they are used in concert with activity detections to perform localization.
@inproceedings{rhinehart2016learning,
 author = {Rhinehart, Nicholas and Kitani, Kris M.},
 booktitle = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
 title = {Learning Action Maps of Large Environments Via First-Person Vision},
 year = {2016}
}

Visual chunking: A list prediction framework for region-based object detection
Visual chunking: A list prediction framework for region-based object detection.

IEEE International Conference on Robotics and Automation, (ICRA), 2015.
We consider detecting objects in an image by iteratively selecting from a set of arbitrarily shaped candidate regions. Our generic approach, which we term visual chunking, reasons about the locations of multiple object instances in an image while expressively describing object boundaries. We design an optimization criterion for measuring the performance of a list of such detections as a natural extension to a common per-instance metric. We present an efficient algorithm with provable performance for building a high-quality list of detections from any candidate set of region-based proposals. We also develop a simple class-specific algorithm to generate a candidate region instance in near-linear time in the number of low-level superpixels that outperforms other region generating methods. In order to make predictions on novel images at testing time without access to ground truth, we develop learning approaches to emulate these algorithms’ behaviors. We demonstrate that our new approach outperforms sophisticated baselines on benchmark datasets.
@inproceedings{rhinehart2015visual,
 author = {Rhinehart, Nicholas and Zhou, Jiaji and Hebert, Martial and Bagnell, J Andrew},
 booktitle = {2015 IEEE International Conference on Robotics and Automation (ICRA)},
 organization = {IEEE},
 pages = {5448--5454},
 title = {Visual chunking: A list prediction framework for region-based object detection},
 year = {2015}
}