Imitation Learning
All Publication Topics
Autonomous Driving
Autonomous Exploration
Benchmarks
Compression
Computer Vision
First-Person Video
Forecasting
Generative Models
Imitation Learning
Intrinsic Control
Machine Learning
Manipulation
Model-Based Control
Offroad Navigation
Paper Award
Reinforcement Learning
Reward Learning
Robotics
Social Navigation
Ratatouille: Imitation Learning Ingredients for Real-world Social Robot Navigation.
arXiv:2509.17204, (arXiv), 2025.
arXiv:2509.17204, (arXiv), 2025.
Scaling Reinforcement Learning to in-the-wild social robot navigation is both data-intensive and unsafe, since policies must learn through direct interaction and inevitably encounter collisions. Offline Imitation learning (IL) avoids these risks by collecting expert demonstrations safely, training entirely offline, and deploying policies zero-shot. However, we find that naively applying Behaviour Cloning (BC) to social navigation is insufficient; achieving strong performance requires careful architectural and training choices. We present Ratatouille, a pipeline and model architecture that, without changing the data, reduces collisions per meter by 6 times and improves success rate by 3 times compared to naive BC. We validate our approach in both simulation and the real world, where we collected over 11 hours of data on a dense university campus. We further demonstrate qualitative results in a public food court. Our findings highlight that thoughtful IL design, rather than additional data, can substantially improve safety and reliability in real-world social navigation. Video: https://youtu.be/tOdLTXsaYLQ. Code will be released after acceptance.
@misc{han2025ratatouilleimitationlearningingredients,
title={Ratatouille: Imitation Learning Ingredients for Real-world Social Robot Navigation},
author={James R. Han and Mithun Vanniasinghe and Hshmat Sahak and Nicholas Rhinehart and Timothy D. Barfoot},
year={2025},
eprint={2509.17204},
archivePrefix={arXiv},
primaryClass={cs.RO},
url={https://arxiv.org/abs/2509.17204},
}
The Waymo Open Sim Agents Challenge.
Advances in Neural Information Processing Systems, (NeurIPS), 2023.
Advances in Neural Information Processing Systems, (NeurIPS), 2023.
Simulation with realistic, interactive agents represents a key task for autonomous vehicle software development. In this work, we introduce the Waymo Open Sim Agents Challenge (WOSAC). WOSAC is the first public challenge to tackle this task and propose corresponding metrics. The goal of the challenge is to stimulate the design of realistic simulators that can be used to evaluate and train a behavior model for autonomous driving. We outline our evaluation methodology, present results for a number of different baseline simulation agent methods, and analyze several submissions to the 2023 competition which ran from March 16, 2023 to May 23, 2023. The WOSAC evaluation server remains open for submissions and we discuss open problems for the task.
@article{montali2024waymo,
author = {Montali, Nico and Lambert, John and Mougin, Paul and Kuefler, Alex and Rhinehart, Nicholas and Li, Michelle and Gulino, Cole and Emrich, Tristan and Yang, Zoey and Whiteson, Shimon and others},
journal = {Advances in Neural Information Processing Systems},
title = {The waymo open sim agents challenge},
volume = {36},
year = {2024}
}
Hybrid imitative planning with geometric and predictive costs in off-road environments.
International Conference on Robotics and Automation, (ICRA), 2022.
International Conference on Robotics and Automation, (ICRA), 2022.
Geometric methods for solving open-world off-road navigation tasks, by learning occupancy and metric maps, provide good generalization but can be brittle in outdoor environments that violate their assumptions (e.g., tall grass). Learning-based methods can directly learn collision-free behavior from raw observations, but are difficult to integrate with standard geometry-based pipelines. This creates an unfortunate conflict – either use learning and lose out on well-understood geometric navigational components, or do not use it, in favor of extensively hand-tuned geometry-based cost maps. In this work, we reject this dichotomy by designing the learning and non-learning-based components in a way such that they can be effectively combined in a self-supervised manner. Both components contribute to a planning criterion: the learned component contributes predicted traversability as rewards, while the geometric component contributes obstacle cost information. We instantiate and comparatively evaluate our system in both in-distribution and out-of-distribution environments, showing that this approach inherits complementary gains from the learned and geometric components and significantly outperforms either of them.
@inproceedings{dashora2022hybrid,
author = {Dashora, Nitish and Shin, Daniel and Shah, Dhruv and Leopold, Henry and Fan, David and Agha-Mohammadi, Ali and Rhinehart, Nicholas and Levine, Sergey},
booktitle = {2022 International Conference on Robotics and Automation (ICRA)},
organization = {IEEE},
pages = {4452--4458},
title = {Hybrid imitative planning with geometric and predictive costs in off-road environments},
year = {2022}
}
Contingencies from observations: Tractable contingency planning with learned behavior models.
IEEE International Conference on Robotics and Automation, (ICRA), 2021.
IEEE International Conference on Robotics and Automation, (ICRA), 2021.
Humans have a remarkable ability to accurately reason about future events, including the behaviors and states of mind of other agents. Consider driving a car through a busy intersection: it is necessary to reason about the physics of the vehicle, the intentions of other drivers, and their beliefs about your own intentions. For example, if you signal a turn, another driver might yield to you; or if you enter the passing lane, another driver might decelerate to give you room to merge in front. Competent drivers must plan how they can safely react to a variety of potential future behaviors of other agents before they make their next move. This requires contingency planning: explicitly planning a set of conditional actions that depend on the stochastic outcome of future events. In this work, we develop a general-purpose contingency planner that is learned end-to-end using high-dimensional scene observations and low-dimensional behavioral observations. We use a conditional autoregressive flow model for contingency planning. We show how this model can tractably learn contingencies from behavioral observations. We developed a closed-loop control benchmark of realistic multi-agent scenarios in a driving simulator (CARLA), on which we compare our method to various noncontingent methods that reason about multi-agent future behavior, and find that our contingency planning method achieves qualitatively and quantitatively superior performance.
@inproceedings{rhinehart2021contingencies,
author = {Rhinehart, Nicholas and He, Jeff and Packer, Charles and Wright, Matthew A and McAllister, Rowan and Gonzalez, Joseph E and Levine, Sergey},
booktitle = {2021 IEEE International Conference on Robotics and Automation (ICRA)},
organization = {IEEE},
pages = {13663--13669},
title = {Contingencies from observations: Tractable contingency planning with learned behavior models},
year = {2021}
}
Parrot: Data-driven behavioral priors for reinforcement learning.
International Conference on Learning Representations, (ICLR), 2020.
International Conference on Learning Representations, (ICLR), 2020.
Reinforcement learning provides a general framework for flexible decision making and control, but requires extensive data collection for each new task that an agent needs to learn. In other machine learning fields, such as natural language processing or computer vision, pre-training on large, previously collected datasets to bootstrap learning for new tasks has emerged as a powerful paradigm to reduce data requirements when learning a new task. In this paper, we ask the following question: how can we enable similarly useful pre-training for RL agents? We propose a method for pre-training behavioral priors that can capture complex input-output relationships observed in successful trials from a wide range of previously seen tasks, and we show how this learned prior can be used for rapidly learning new tasks without impeding the RL agent’s ability to try out novel behaviors. We demonstrate the effectiveness of our approach in challenging robotic manipulation domains involving image observations and sparse reward functions, where our method outperforms prior works by a substantial margin.
@article{singh2020parrot,
author = {Singh, Avi and Liu, Huihan and Zhou, Gaoyue and Yu, Albert and Rhinehart, Nicholas and Levine, Sergey},
journal = {arXiv preprint arXiv:2011.10024},
title = {Parrot: Data-driven behavioral priors for reinforcement learning},
year = {2020}
}
Deep Imitative Models for Flexible Inference, Planning, and Control.
International Conference on Learning Representations, (ICLR), 2020.
International Conference on Learning Representations, (ICLR), 2020.
Imitation Learning (IL) is an appealing approach to learn desirable autonomous behavior. However, directing IL to achieve arbitrary goals is difficult. In contrast, planning-based algorithms use dynamics models and reward functions to achieve goals. Yet, reward functions that evoke desirable behavior are often difficult to specify. In this paper, we propose Imitative Models to combine the benefits of IL and goal-directed planning. Imitative Models are probabilistic predictive models of desirable behavior able to plan interpretable expert-like trajectories to achieve specified goals. We derive families of flexible goal objectives, including constrained goal regions, unconstrained goal sets, and energy-based goals. We show that our method can use these objectives to successfully direct behavior. Our method substantially outperforms six IL approaches and a planning-based approach in a dynamic simulated autonomous driving task, and is efficiently learned from expert demonstrations without online data collection. We also show our approach is robust to poorly specified goals, such as goals on the wrong side of the road.
@inproceedings{rhinehart2020deep,
author = {Rhinehart, Nicholas and McAllister, Rowan and Levine, Sergey},
booktitle = {International Conference on Learning Representations (ICLR)},
title = {Deep Imitative Models for Flexible Inference, Planning, and Control},
year = {2020}
}
Can autonomous vehicles identify, recover from, and adapt to distribution shifts?.
International Conference on Machine Learning, (ICML), 2020.
International Conference on Machine Learning, (ICML), 2020.
Out-of-training-distribution (OOD) scenarios are a common challenge of learning agents at deployment, typically leading to arbitrary deductions and poorly-informed decisions. In principle, detection of and adaptation to OOD scenes can mitigate their adverse effects. In this paper, we highlight the limitations of current approaches to novel driving scenes and propose an epistemic uncertainty-aware planning method, called \emph{robust imitative planning} (RIP). Our method can detect and recover from some distribution shifts, reducing the overconfident and catastrophic extrapolations in OOD scenes. If the model’s uncertainty is too great to suggest a safe course of action, the model can instead query the expert driver for feedback, enabling sample-efficient online adaptation, a variant of our method we term \emph{adaptive robust imitative planning} (AdaRIP). Our methods outperform current state-of-the-art approaches in the nuScenes \emph{prediction} challenge, but since no benchmark evaluating OOD detection and adaption currently exists to assess \emph{control}, we introduce an autonomous car novel-scene benchmark, \texttt{CARNOVEL}, to evaluate the robustness of driving agents to a suite of tasks with distribution shifts.
@inproceedings{filos2020can,
author = {Filos, Angelos and Tigkas, Panagiotis and McAllister, Rowan and Rhinehart, Nicholas and Levine, Sergey and Gal, Yarin},
booktitle = {International Conference on Machine Learning},
organization = {PMLR},
pages = {3145--3153},
title = {Can autonomous vehicles identify, recover from, and adapt to distribution shifts?},
year = {2020}
}
Jointly Forecasting and Controlling Behavior by Learning from High-Dimensional Data.
2019.
2019.
Achieving a precise predictive understanding of the future is difficult, yet widely studied in the natural sciences. Significant research activity has been dedicated to building testable models of cause and effect. From a certain view, the ability to forecast the universe is the “holy grail”; the ultimate goal of science. If we had it, we could anticipate, and therefore (at least implicitly) understand all observable phenomena. The human capability to forecast offers complementary motivation. Critical to our intelligence is our ability to plan behaviors by considering how our actions are likely to result in future payoff, especially in the presence of other collaborative and competitive agents. In this work, we seek to computationally model the future in the presence of agent behavior given rich observations of the environment. The brunt of our focus is to reason about what agents could do, instead of other sources of stochasticity. This focus on future agent behavior allows us to tightly couple and jointly perform forecasting and control. The field of Computer Vision (CV) is focused on designing algorithms to automatically understand images, videos, and other perceptual data. However, the field’s effort to-date focuses on non-interactive, present-focused tasks [79, 81, 158, 184]. Most CV contributions are algorithms to answer questions like “what is that”, and “what happened”, rather than “what could happen”, or “how could I achieve X”. Computer Vision has under-explored reasoning about the interactive and decision-based nature of the world. In contrast, Reinforcement Learning (RL) prioritizes modeling interactions and decisions by focusing on how to design algorithms to evoke behavior that maximizes a scalar reward signal. The resulting learning agents, in order to perform well, must have an understanding of how their current behaviors will affect their prospects of future reward. However, in the dominant paradigm of model-free RL [218], agents reason implicitly about the future. In contrast, model-based RL learns one-step dynamics to estimate “what could happen in the near future”. Yet model-based RL primarily focuses on control, rather than explicitly forecasting a single agent (let alone multiple agents). In this thesis, we consider the problem of designing algorithms to enable computational systems to (1) forecast future behavior of intelligent agents given rich observations of their environments, as well as to (2) use this reasoning for control. We believe these two problems should be tightly integrated and jointly considered, and use them to structure this thesis. We define forecasting to be the problem of estimating the set of possible outcomes of a system, whereas control is the problem of producing actions that generate a single outcome of a system. We often use Imitation Learning and Reinforcement Learning to formulate and situate our work. We contribute forecasting and control approaches to excel in diverse, realistic, single-agent, and multi-agent domains. The first part of the thesis focuses on progressively designing more capable forecasting models. We proceed through approaches to (1) forecast single actions of daily behavior by developing matrix factorization models [169], (2) forecast goal-driven action trajectories of daily behavior by developing Online Inverse Reinforcement Learning models [168, 170], (3) forecast motion trajectories of vehicles by developing a deep reversible generative models [171, 174]. The second part of the thesis focuses on progressively designing more capable models that tightly couple forecasting and control. We discuss (4) forecasting as auxiliary supervision for implicitly-planned control [228], (5) forecasting and explicitly planning with the same model [176], and (6) forecasting and planning future interactions of multiple agents [175].
@phdthesis{rhinehart2019jointly,
author = {Rhinehart, Nicholas},
school = {Carnegie Mellon University},
title = {Jointly Forecasting and Controlling Behavior by Learning from High-Dimensional Data},
year = {2019}
}
Generative Hybrid Representations for Activity Forecasting with No-Regret Learning.
Conference on Computer Vision and Pattern Recognition, (CVPR), 2019.
Conference on Computer Vision and Pattern Recognition, (CVPR), 2019.
Automatically reasoning about future human behaviors is a difficult problem but has significant practical applications to assistive systems. Part of this difficulty stems from learning systems’ inability to represent all kinds of behaviors. Some behaviors, such as motion, are best described with continuous representations, whereas others, such as picking up a cup, are best described with discrete representations. Furthermore, human behavior is generally not fixed: people can change their habits and routines. This suggests these systems must be able to learn and adapt continuously. In this work, we develop an efficient deep generative model to jointly forecast a person’s future discrete actions and continuous motions. On a large-scale egocentric dataset, EPIC-KITCHENS, we observe our method generates high-quality and diverse samples while exhibiting better generalization than related generative models. Finally, we propose a variant to continually learn our model from streaming data, observe its practical effectiveness, and theoretically justify its learning efficiency.
@article{guan2019generative,
author = {Guan, Jiaqi and Yuan, Ye and Kitani, Kris M and Rhinehart, Nicholas},
journal = {arXiv preprint arXiv:1904.06250},
title = {Generative Hybrid Representations for Activity Forecasting with No-Regret Learning},
year = {2019}
}
Directed-Info GAIL: Learning Hierarchical Policies from Unsegmented Demonstrations using Directed Information.
International Conference on Learning Representations (ICLR), (ICLR), 2019.
International Conference on Learning Representations (ICLR), (ICLR), 2019.
The use of imitation learning to learn a single policy for a complex task that has multiple modes or hierarchical structure can be challenging. In fact, previous work has shown that when the modes are known, learning separate policies for each mode or sub-task can greatly improve the performance of imitation learning. In this work, we discover the interaction between sub-tasks from their resulting state-action trajectory sequences using a directed graphical model. We propose a new algorithm based on the generative adversarial imitation learning framework which automatically learns sub-task policies from unsegmented demonstrations. Our approach maximizes the directed information flow in the graphical model between sub-task latent variables and their generated trajectories. We also show how our approach connects with the existing Options framework, which is commonly used to learn hierarchical policies.
@inproceedings{sharma2019directed,
author = {Sharma, Arjun and Sharma, Mohit and Rhinehart, Nicholas and Kitani, Kris M},
booktitle = {International Conference on Learning Representations (ICLR)},
title = {Directed-Info GAIL: Learning Hierarchical Policies from Unsegmented Demonstrations using Directed Information},
year = {2019}
}
R2P2: A Reparameterized Pushforward Policy for Diverse, Precise Generative Path Forecasting.
Proceedings of the European Conference on Computer Vision, (ECCV), 2018.
Proceedings of the European Conference on Computer Vision, (ECCV), 2018.
We propose a method to forecast a vehicle’s ego-motion as a distribution over spatiotemporal paths, conditioned on features (e.g., from LIDAR and images) embedded in an overhead map. The method learns a policy inducing a distribution over simulated trajectories that is both “diverse” (produces most of the likely paths) and “precise” (mostly produces likely paths). This balance is achieved through minimization of a symmetrized cross-entropy between the distribution and demonstration data. By viewing the simulated-outcome distribution as the pushforward of a simple distribution under a simulation operator, we obtain expressions for the cross-entropy metrics that can be efficiently evaluated and differentiated, enabling stochastic-gradient optimization. We propose concrete policy architectures for this model, discuss our evaluation metrics relative to previously-used degenerate metrics, and demonstrate the superiority of our method relative to state-of-the-art methods in both the Kitti dataset and a similar but novel and larger real-world dataset explicitly designed for the vehicle forecasting domain.
@inproceedings{rhinehart2018r2p2,
author = {Rhinehart, Nicholas and Kitani, Kris M. and Vernaza, Paul},
booktitle = {Proceedings of the European Conference on Computer Vision (ECCV)},
pages = {772--788},
title = {R2P2: A Reparameterized Pushforward Policy for Diverse, Precise Generative Path Forecasting},
year = {2018}
}
Learning Neural Parsers with Deterministic Differentiable Imitation Learning.
Conference on Robot Learning, (CoRL), 2018.
Conference on Robot Learning, (CoRL), 2018.
We explore the problem of learning to decompose spatial tasks into segments, as exemplified by the problem of a painting robot covering a large object. Inspired by the ability of classical decision tree algorithms to construct structured partitions of their input spaces, we formulate the problem of decomposing objects into segments as a parsing approach. We make the insight that the derivation of a parse-tree that decomposes the object into segments closely resembles a decision tree constructed by ID3, which can be done when the ground-truth available. We learn to imitate an expert parsing oracle, such that our neural parser can generalize to parse natural images without ground truth. We introduce a novel deterministic policy gradient update, DRAG (i.e., DeteRministically AGgrevate) in the form of a deterministic actor-critic variant of AggreVaTeD, to train our neural parser. From another perspective, our approach is a variant of the Deterministic Policy Gradient suitable for the imitation learning setting. The deterministic policy representation offered by training our neural parser with DRAG allows it to outperform state of the art imitation and reinforcement learning approaches.
@inproceedings{shankar2018learning,
author = {Shankar, Tanmay and Rhinehart, Nicholas and Muelling, Katharina and Kitani, Kris M.},
booktitle = {arXiv:1806.07822},
title = {Learning Neural Parsers with Deterministic Differentiable Imitation Learning},
year = {2018}
}
Human-Interactive Subgoal Supervision for Efficient Inverse Reinforcement Learning.
Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, (AAMAS), 2018.
Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, (AAMAS), 2018.
Humans are able to understand and perform complex tasks by strategically structuring tasks into incremental steps or sub-goals. For a robot attempting to learn to perform a sequential task with critical subgoal states, these subgoal states can provide a natural opportunity for interaction with a human expert. This paper analyzes the benefit of incorporating a notion of subgoals into Inverse Reinforcement Learning (IRL) with a Human-In-The-Loop (HITL) framework. The learning process is interactive, with a human expert first providing input in the form of full demonstrations along with some subgoal states. These subgoal states defines a set of sub-tasks for the learning agent to complete in order to achieve the final goal. The learning agent queries for partial demonstrations corresponding to each sub-task as needed when the learning agent struggles with individual sub-task. The proposed Human Interactive IRL (HI-IRL) framework is evaluated on several discrete path-planning tasks. We demonstrate that subgoal-based interactive structuring of the learning task results in significantly more efficient learning, requiring only a fraction of the demonstration data needed for learning the underlying reward function with a baseline IRL model.
@inproceedings{pan2018human,
author = {Pan, Xinlei and Ohn-Bar, Eshed and Rhinehart, Nicholas and Xu, Yan and Shen, Yilin and Kitani, Kris M.},
booktitle = {Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems},
organization = {International Foundation for Autonomous Agents and Multiagent Systems},
pages = {1380--1387},
title = {Human-Interactive Subgoal Supervision for Efficient Inverse Reinforcement Learning},
year = {2018}
}
First-Person Activity Forecasting from Video with Online Inverse Reinforcement Learning.
IEEE Transactions on Pattern Analysis and Machine Intelligence, (PAMI), 2018.
IEEE Transactions on Pattern Analysis and Machine Intelligence, (PAMI), 2018.
We address the problem of incrementally modeling and forecasting long-term goals of a first-person camera wearer: what the user will do, where they will go, and what goal they seek. In contrast to prior work in trajectory forecasting, our algorithm, Darko, goes further to reason about semantic states (will I pick up an object?), and future goal states that are far in terms of both space and time. Darko learns and forecasts from first-person visual observations of the user’s daily behaviors via an Online Inverse Reinforcement Learning (IRL) approach. Classical IRL discovers only the rewards in a batch setting, whereas Darko discovers the transitions, rewards, and goals of a user from streaming data. Among other results, we show Darko forecasts goals better than competing methods in both noisy and ideal settings, and our approach is theoretically and empirically no-regret.
@article{rhinehart2018first,
author = {Rhinehart, Nicholas and Kitani, Kris},
journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
publisher = {IEEE},
title = {First-Person Activity Forecasting from Video with Online Inverse Reinforcement Learning},
year = {2018}
}
Predictive-state decoders: Encoding the future into recurrent networks.
Advances in Neural Information Processing Systems, (NeurIPS), 2017.
Advances in Neural Information Processing Systems, (NeurIPS), 2017.
Recurrent neural networks (RNNs) are a vital modeling technique that rely on internal states learned indirectly by optimization of a supervised, unsupervised, or reinforcement training loss. RNNs are used to model dynamic processes that are characterized by underlying latent states whose form is often unknown, precluding its analytic representation inside an RNN. In the Predictive-State Representation (PSR) literature, latent state processes are modeled by an internal state representation that directly models the distribution of future observations, and most recent work in this area has relied on explicitly representing and targeting sufficient statistics of this probability distribution. We seek to combine the advantages of RNNs and PSRs by augmenting existing state-of-the-art recurrent neural networks with Predictive-State Decoders (PSDs), which add supervision to the network’s internal state representation to target predicting future observations. Predictive-State Decoders are simple to implement and easily incorporated into existing training pipelines via additional loss regularization. We demonstrate the effectiveness of PSDs with experimental results in three different domains: probabilistic filtering, Imitation Learning, and Reinforcement Learning. In each, our method improves statistical performance of state-of-the-art recurrent baselines and does so with fewer iterations and less data.
@inproceedings{venkatraman2017predictive,
author = {Venkatraman, Arun and Rhinehart, Nicholas and Sun, Wen and Pinto, Lerrel and Hebert, Martial and Boots, Byron and Kitani, Kris M. and Bagnell, J. A.},
booktitle = {Advances in Neural Information Processing Systems},
pages = {1172--1183},
title = {Predictive-state decoders: Encoding the future into recurrent networks},
year = {2017}
}
First-Person Activity Forecasting with Online Inverse Reinforcement Learning.
The IEEE International Conference on Computer Vision, (ICCV), 2017.
Best Paper Honorable Mention
The IEEE International Conference on Computer Vision, (ICCV), 2017.
Best Paper Honorable Mention
We address the problem of incrementally modeling and forecasting long-term goals of a first-person camera wearer: what the user will do, where they will go, and what goal they seek. In contrast to prior work in trajectory forecasting, our algorithm, DARKO, goes further to reason about semantic states (will I pick up an object?), and future goal states that are far in terms of both space and time. DARKO learns and forecasts from first-person visual observations of the user’s daily behaviors via an Online Inverse Reinforcement Learning (IRL) approach. Classical IRL discovers only the rewards in a batch setting, whereas DARKO discovers the states, transitions, rewards, and goals of a user from streaming data. Among other results, we show DARKO forecasts goals better than competing methods in both noisy and ideal settings, and our approach is theoretically and empirically no-regret.
@inproceedings{rhinehart2017first,
author = {Rhinehart, Nicholas and Kitani, Kris M.},
booktitle = {The IEEE International Conference on Computer Vision (ICCV)},
pages = {3716--3725},
title = {First-Person Activity Forecasting with Online Inverse Reinforcement Learning},
year = {2017}
}
Visual chunking: A list prediction framework for region-based object detection.
IEEE International Conference on Robotics and Automation, (ICRA), 2015.
IEEE International Conference on Robotics and Automation, (ICRA), 2015.
We consider detecting objects in an image by iteratively selecting from a set of arbitrarily shaped candidate regions. Our generic approach, which we term visual chunking, reasons about the locations of multiple object instances in an image while expressively describing object boundaries. We design an optimization criterion for measuring the performance of a list of such detections as a natural extension to a common per-instance metric. We present an efficient algorithm with provable performance for building a high-quality list of detections from any candidate set of region-based proposals. We also develop a simple class-specific algorithm to generate a candidate region instance in near-linear time in the number of low-level superpixels that outperforms other region generating methods. In order to make predictions on novel images at testing time without access to ground truth, we develop learning approaches to emulate these algorithms’ behaviors. We demonstrate that our new approach outperforms sophisticated baselines on benchmark datasets.
@inproceedings{rhinehart2015visual,
author = {Rhinehart, Nicholas and Zhou, Jiaji and Hebert, Martial and Bagnell, J Andrew},
booktitle = {2015 IEEE International Conference on Robotics and Automation (ICRA)},
organization = {IEEE},
pages = {5448--5454},
title = {Visual chunking: A list prediction framework for region-based object detection},
year = {2015}
}