Publications:
2025 |
Marougkas, I; Ramesh, D; Doerr, J; Granados, E; Sivaramakrishnan, A; Boularias, A; Bekris, K Integrating Model-based Control and RL for Sim2Real Transfer of Tight Insertion Policies Conference IEEE International Conference on Robotics and Automation (ICRA), 2025. @conference{marougkas2025integration, title = {Integrating Model-based Control and RL for Sim2Real Transfer of Tight Insertion Policies}, author = {I Marougkas and D Ramesh and J Doerr and E Granados and A Sivaramakrishnan and A Boularias and K Bekris}, year = {2025}, date = {2025-05-01}, booktitle = {IEEE International Conference on Robotics and Automation (ICRA)}, abstract = {Object insertion under tight tolerances (<1mm) is an important but challenging assembly task as even slight errors can result in undesirable contacts. Recent efforts have focused on using Reinforcement Learning (RL) and often depend on careful definition of dense reward functions. This work proposes an effective strategy for such tasks that integrates traditional model-based control with RL to achieve improved accuracy given training of the policy exclusively in simulation and zero- shot transfer to the real system. It employs a potential field- based controller to acquire a model-based policy for inserting a plug into a socket given full observability in simulation. This policy is then integrated with a residual RL one, which is trained in simulation given only a sparse, goal-reaching reward. A curriculum scheme over observation noise and action magnitude is used for training the residual RL policy. Both policy components use as input the SE(3) poses of both the plug and the socket and return the plug's SE(3) pose transform, which is executed by a robotic arm using a controller. The integrated policy is deployed on the real system without further training or fine-tuning, given a visual SE(3) object tracker. The proposed solution and alternatives are evaluated across a variety of objects and conditions in simulation and reality. The proposed approach outperforms recent RL methods in this domain and prior efforts for hybrid policies. Ablations highlight the impact of each component of the approach}, keywords = {}, pubstate = {published}, tppubtype = {conference} } Object insertion under tight tolerances (<1mm) is an important but challenging assembly task as even slight errors can result in undesirable contacts. Recent efforts have focused on using Reinforcement Learning (RL) and often depend on careful definition of dense reward functions. This work proposes an effective strategy for such tasks that integrates traditional model-based control with RL to achieve improved accuracy given training of the policy exclusively in simulation and zero- shot transfer to the real system. It employs a potential field- based controller to acquire a model-based policy for inserting a plug into a socket given full observability in simulation. This policy is then integrated with a residual RL one, which is trained in simulation given only a sparse, goal-reaching reward. A curriculum scheme over observation noise and action magnitude is used for training the residual RL policy. Both policy components use as input the SE(3) poses of both the plug and the socket and return the plug's SE(3) pose transform, which is executed by a robotic arm using a controller. The integrated policy is deployed on the real system without further training or fine-tuning, given a visual SE(3) object tracker. The proposed solution and alternatives are evaluated across a variety of objects and conditions in simulation and reality. The proposed approach outperforms recent RL methods in this domain and prior efforts for hybrid policies. Ablations highlight the impact of each component of the approach |
2023 |
Vieira, E; Gao, K; Nakhimovich, D; Bekris, K; Yu, J Persistent Homology Guided Monte-Carlo Tree Search for Effective Non-Prehensile Manipulation Inproceedings International Symposium on Experimental Robotics (ISER), 2023. @inproceedings{Vieira:2023ab, title = {Persistent Homology Guided Monte-Carlo Tree Search for Effective Non-Prehensile Manipulation}, author = {E Vieira and K Gao and D Nakhimovich and K Bekris and J Yu}, url = {http://arxiv.org/abs/2210.01283}, year = {2023}, date = {2023-12-01}, booktitle = {International Symposium on Experimental Robotics (ISER)}, abstract = {Performing object retrieval in real-world workspaces must tackle challenges including uncertainty and clutter. One option is to apply prehensile operations, which can be time consuming in highly-cluttered scenarios. On the other hand, non-prehensile actions, such as pushing simultaneously multiple objects, can help to quickly clear a cluttered workspace and retrieve a target object. Such actions, however, can also lead to increased uncertainty as it is difficult to estimate the outcome of pushing operations. The proposed framework in this work integrates topological tools and Monte-Carlo Tree Search (MCTS) to achieve effective and robust pushing for object retrieval. It employs persistent homology to automatically identify manageable clusters of blocking objects without the need for manually adjusting hyper-parameters. Then, MCTS uses this information to explore feasible actions to push groups of objects, aiming to minimize the number of operations needed to clear the path to the target. Real-world experiments using a Baxter robot, which involves some noise in actuation, show that the proposed framework achieves a higher success rate in solving retrieval tasks in dense clutter than alternatives. Moreover, it produces solutions with few pushing actions improving the overall execution time. More critically, it is robust enough that it allows one to plan the sequence of actions offline and then execute them reliably on a Baxter robot.}, keywords = {}, pubstate = {published}, tppubtype = {inproceedings} } Performing object retrieval in real-world workspaces must tackle challenges including uncertainty and clutter. One option is to apply prehensile operations, which can be time consuming in highly-cluttered scenarios. On the other hand, non-prehensile actions, such as pushing simultaneously multiple objects, can help to quickly clear a cluttered workspace and retrieve a target object. Such actions, however, can also lead to increased uncertainty as it is difficult to estimate the outcome of pushing operations. The proposed framework in this work integrates topological tools and Monte-Carlo Tree Search (MCTS) to achieve effective and robust pushing for object retrieval. It employs persistent homology to automatically identify manageable clusters of blocking objects without the need for manually adjusting hyper-parameters. Then, MCTS uses this information to explore feasible actions to push groups of objects, aiming to minimize the number of operations needed to clear the path to the target. Real-world experiments using a Baxter robot, which involves some noise in actuation, show that the proposed framework achieves a higher success rate in solving retrieval tasks in dense clutter than alternatives. Moreover, it produces solutions with few pushing actions improving the overall execution time. More critically, it is robust enough that it allows one to plan the sequence of actions offline and then execute them reliably on a Baxter robot. |
Li, S; Keipour, A; Jamieson, K; Hudson, N; Swan, C; Bekris, K Demonstrating Large-Scale Package Manipulation Via Learned Metrics of Pick Success Inproceedings Robotics: Science and Systems (RSS), Daegu, Korea, 2023. @inproceedings{Li:2023aa, title = {Demonstrating Large-Scale Package Manipulation Via Learned Metrics of Pick Success}, author = {S Li and A Keipour and K Jamieson and N Hudson and C Swan and K Bekris}, year = {2023}, date = {2023-07-01}, booktitle = {Robotics: Science and Systems (RSS)}, address = {Daegu, Korea}, abstract = {Automating warehouse operations can reduce logistics overhead costs, ultimately driving down the final price for consumers, increasing the speed of delivery, and enhancing the resiliency to workforce fluctuations. The past few years have seen increased interest in automating such repeated tasks but mostly in controlled settings. Tasks such as picking objects from unstructured, cluttered piles have only recently become robust enough for large-scale deployment with minimal human intervention. This paper demonstrates a large-scale package manipulation from unstructured piles in Amazon Robotics' Robot Induction (Robin) fleet, which utilizes a pick success predictor trained on real production data. Specifically, the system was trained on over 394K picks. It is used for singulating up to 5~million packages per day and has manipulated over 200~million packages during this paper's evaluation period. The developed learned pick quality measure ranks various pick alternatives in real-time and prioritizes the most promising ones for execution. The pick success predictor aims to estimate from prior experience the success probability of a desired pick by the deployed industrial robotic arms in cluttered scenes containing deformable and rigid objects with partially known properties. It is a shallow machine learning model, which allows us to evaluate which features are most important for the prediction. An online pick ranker leverages the learned success predictor to prioritize the most promising picks for the robotic arm, which are then assessed for collision avoidance. This learned ranking process is demonstrated to overcome the limitations and outperform the performance of manually engineered and heuristic alternatives. To the best of the authors' knowledge, this paper presents the first large-scale deployment of learned pick quality estimation methods in a real production system.}, keywords = {}, pubstate = {published}, tppubtype = {inproceedings} } Automating warehouse operations can reduce logistics overhead costs, ultimately driving down the final price for consumers, increasing the speed of delivery, and enhancing the resiliency to workforce fluctuations. The past few years have seen increased interest in automating such repeated tasks but mostly in controlled settings. Tasks such as picking objects from unstructured, cluttered piles have only recently become robust enough for large-scale deployment with minimal human intervention. This paper demonstrates a large-scale package manipulation from unstructured piles in Amazon Robotics' Robot Induction (Robin) fleet, which utilizes a pick success predictor trained on real production data. Specifically, the system was trained on over 394K picks. It is used for singulating up to 5~million packages per day and has manipulated over 200~million packages during this paper's evaluation period. The developed learned pick quality measure ranks various pick alternatives in real-time and prioritizes the most promising ones for execution. The pick success predictor aims to estimate from prior experience the success probability of a desired pick by the deployed industrial robotic arms in cluttered scenes containing deformable and rigid objects with partially known properties. It is a shallow machine learning model, which allows us to evaluate which features are most important for the prediction. An online pick ranker leverages the learned success predictor to prioritize the most promising picks for the robotic arm, which are then assessed for collision avoidance. This learned ranking process is demonstrated to overcome the limitations and outperform the performance of manually engineered and heuristic alternatives. To the best of the authors' knowledge, this paper presents the first large-scale deployment of learned pick quality estimation methods in a real production system. |
Nakhimovich, D; Miao, Y; Bekris, K Resolution Complete In-Place Object Retrieval Given Known Object Models Inproceedings IEEE International Conference on Robotics and Automatics (ICRA), London, UK, 2023. @inproceedings{Nakhimovich:2023aa, title = {Resolution Complete In-Place Object Retrieval Given Known Object Models}, author = {D Nakhimovich and Y Miao and K Bekris}, url = {https://arxiv.org/abs/2303.14562}, year = {2023}, date = {2023-01-01}, booktitle = {IEEE International Conference on Robotics and Automatics (ICRA)}, address = {London, UK}, abstract = {This work proposes a robot task planning framework for retrieving a target object in a confined workspace among multiple stacked objects that obstruct the target. The robot can use prehensile picking and in-workspace placing actions. The method assumes access to 3D models for the visible objects in the scene. The key contribution is in achieving desirable properties, i.e., to provide (a) safety, by avoiding collisions with sensed obstacles, objects, and occluded regions, and (b) resolution completeness (RC) - or probabilistic completeness (PC) depending on implementation - which indicates a solution will be eventually found (if it exists) as the resolution of algorithmic parameters increases. A heuristic variant of the basic RC algorithm is also proposed to solve the task more efficiently while retaining the desirable properties. Simulation results compare using random picking and placing operations against the basic RC algorithm that reasons about object dependency as well as its heuristic variant. The success rate is higher for the RC approaches given the same amount of time. The heuristic variant is able to solve the problem even more efficiently than the basic approach. The integration of the RC algorithm with perception, where an RGB-D sensor detects the objects as they are being moved, enables real robot demonstrations of safely retrieving target objects from a cluttered shelf.}, keywords = {}, pubstate = {published}, tppubtype = {inproceedings} } This work proposes a robot task planning framework for retrieving a target object in a confined workspace among multiple stacked objects that obstruct the target. The robot can use prehensile picking and in-workspace placing actions. The method assumes access to 3D models for the visible objects in the scene. The key contribution is in achieving desirable properties, i.e., to provide (a) safety, by avoiding collisions with sensed obstacles, objects, and occluded regions, and (b) resolution completeness (RC) - or probabilistic completeness (PC) depending on implementation - which indicates a solution will be eventually found (if it exists) as the resolution of algorithmic parameters increases. A heuristic variant of the basic RC algorithm is also proposed to solve the task more efficiently while retaining the desirable properties. Simulation results compare using random picking and placing operations against the basic RC algorithm that reasons about object dependency as well as its heuristic variant. The success rate is higher for the RC approaches given the same amount of time. The heuristic variant is able to solve the problem even more efficiently than the basic approach. The integration of the RC algorithm with perception, where an RGB-D sensor detects the objects as they are being moved, enables real robot demonstrations of safely retrieving target objects from a cluttered shelf. |
2022 |
Wen, B; Lian, W; Bekris, K; Schaal, S You Only Demonstrate Once: Category-Level Manipulation from Single Visual Demonstration Inproceedings Robotics: Science and Systems (RSS), 2022, (Nomination for Best Paper Award). @inproceedings{Wen:2022ab, title = {You Only Demonstrate Once: Category-Level Manipulation from Single Visual Demonstration}, author = {B Wen and W Lian and K Bekris and S Schaal}, url = {https://www.roboticsproceedings.org/rss18/p044.pdf}, year = {2022}, date = {2022-06-01}, booktitle = {Robotics: Science and Systems (RSS)}, abstract = {Promising results have been achieved recently in category-level manipulation that generalizes across object instances. Nevertheless, it often requires expensive real-world data collection and manual specification of semantic keypoints for each object category and task. Additionally, coarse keypoint predictions and ignoring intermediate action sequences hinder adoption in complex manipulation tasks beyond pick-and-place. This work proposes a novel, category-level manipulation framework that leverages an object-centric, category-level representation and model-free 6 DoF motion tracking. The canonical object representation is learned solely in simulation and then used to parse a category-level, task trajectory from a single demonstration video. The demonstration is reprojected to a target trajectory tailored to a novel object via the canonical representation. During execution, the manipulation horizon is decomposed into long range, collision-free motion and last-inch manipulation. For the latter part, a category-level behavior cloning (CatBC) method leverages motion tracking to perform closed-loop control. CatBC follows the target trajectory, projected from the demonstration and anchored to a dynamically selected category-level coordinate frame. The frame is automatically selected along the manipulation horizon by a local attention mechanism. This framework allows to teach different manipulation strategies by solely providing a single demonstration, without complicated manual programming. Extensive experiments demonstrate its efficacy in a range of challenging industrial tasks in high precision assembly, which involve learning complex, long-horizon policies. The process exhibits robustness against uncertainty due to dynamics as well as generalization across object instances and scene configurations.}, note = {Nomination for Best Paper Award}, keywords = {}, pubstate = {published}, tppubtype = {inproceedings} } Promising results have been achieved recently in category-level manipulation that generalizes across object instances. Nevertheless, it often requires expensive real-world data collection and manual specification of semantic keypoints for each object category and task. Additionally, coarse keypoint predictions and ignoring intermediate action sequences hinder adoption in complex manipulation tasks beyond pick-and-place. This work proposes a novel, category-level manipulation framework that leverages an object-centric, category-level representation and model-free 6 DoF motion tracking. The canonical object representation is learned solely in simulation and then used to parse a category-level, task trajectory from a single demonstration video. The demonstration is reprojected to a target trajectory tailored to a novel object via the canonical representation. During execution, the manipulation horizon is decomposed into long range, collision-free motion and last-inch manipulation. For the latter part, a category-level behavior cloning (CatBC) method leverages motion tracking to perform closed-loop control. CatBC follows the target trajectory, projected from the demonstration and anchored to a dynamically selected category-level coordinate frame. The frame is automatically selected along the manipulation horizon by a local attention mechanism. This framework allows to teach different manipulation strategies by solely providing a single demonstration, without complicated manual programming. Extensive experiments demonstrate its efficacy in a range of challenging industrial tasks in high precision assembly, which involve learning complex, long-horizon policies. The process exhibits robustness against uncertainty due to dynamics as well as generalization across object instances and scene configurations. |
Wang, R; Gao, K; Yu, J; Bekris, K Lazy Rearrangement Planning in Confined Spaces Inproceedings International Conference on Automated Planning and Scheduling (ICAPS), 2022. @inproceedings{Wang:2022ac, title = {Lazy Rearrangement Planning in Confined Spaces}, author = {R Wang and K Gao and J Yu and K Bekris}, url = {https://arxiv.org/abs/2203.10379}, year = {2022}, date = {2022-06-01}, booktitle = {International Conference on Automated Planning and Scheduling (ICAPS)}, abstract = {Object rearrangement is important for many applications but remains challenging, especially in confined spaces, such as shelves, where objects cannot be accessed from above and they block reachability to each other. Such constraints require many motion planning and collision checking calls, which are computationally expensive. In addition, the arrangement space grows exponentially with the number of objects. To address these issues, this work introduces a lazy evaluation framework with a local monotone solver and a global planner. Monotone instances are those that can be solved by moving each object at most once. A key insight is that reachability constraints at the grasps for objects' starts and goals can quickly reveal dependencies between objects without having to execute expensive motion planning queries. Given that, the local solver builds lazily a search tree that respects these reachability constraints without verifying that the arm paths are collision free. It only collision checks when a promising solution is found. If a monotone solution is not found, the non-monotone planner loads the lazy search tree and explores ways to move objects to intermediate locations from where monotone solutions to the goal can be found. Results show that the proposed framework can solve difficult instances in confined spaces with up to 16 objects, which state-of-the-art methods fail to solve. It also solves problems faster than alter- natives, when the alternatives find a solution. It also achieves high-quality solutions, i.e., only 1.8 additional actions on av- erage are needed for non-monotone instances.}, keywords = {}, pubstate = {published}, tppubtype = {inproceedings} } Object rearrangement is important for many applications but remains challenging, especially in confined spaces, such as shelves, where objects cannot be accessed from above and they block reachability to each other. Such constraints require many motion planning and collision checking calls, which are computationally expensive. In addition, the arrangement space grows exponentially with the number of objects. To address these issues, this work introduces a lazy evaluation framework with a local monotone solver and a global planner. Monotone instances are those that can be solved by moving each object at most once. A key insight is that reachability constraints at the grasps for objects' starts and goals can quickly reveal dependencies between objects without having to execute expensive motion planning queries. Given that, the local solver builds lazily a search tree that respects these reachability constraints without verifying that the arm paths are collision free. It only collision checks when a promising solution is found. If a monotone solution is not found, the non-monotone planner loads the lazy search tree and explores ways to move objects to intermediate locations from where monotone solutions to the goal can be found. Results show that the proposed framework can solve difficult instances in confined spaces with up to 16 objects, which state-of-the-art methods fail to solve. It also solves problems faster than alter- natives, when the alternatives find a solution. It also achieves high-quality solutions, i.e., only 1.8 additional actions on av- erage are needed for non-monotone instances. |
Lu, S; Wang, R; Miao, Y; Mitash, C; Bekris, K Online Object Model Reconstruction and Reuse for Lifelong Improvement of Robot Manipulation Inproceedings IEEE International Conference on Robotics and Automation (ICRA), 2022, (Nomination for Best Paper Award in Manipulation). @inproceedings{Lu:2022ab, title = {Online Object Model Reconstruction and Reuse for Lifelong Improvement of Robot Manipulation}, author = {S Lu and R Wang and Y Miao and C Mitash and K Bekris}, url = {https://arxiv.org/abs/2109.13910}, year = {2022}, date = {2022-05-01}, booktitle = {IEEE International Conference on Robotics and Automation (ICRA)}, abstract = {This work proposes a robotic pipeline for picking and constrained placement of objects without geometric shape priors. Compared to recent efforts developed for similar tasks, where every object was assumed to be novel, the proposed system recognizes previously manipulated objects and performs online model reconstruction and reuse. Over a lifelong manipulation process, the system keeps learning features of objects it has interacted with and updates their reconstructed models. Whenever an instance of a previously manipulated object reappears, the system aims to first recognize it and then register its previously reconstructed model given the current observation. This step greatly reduces object shape uncertainty allowing the system to even reason for parts of objects, which are currently not observable. This also results in better manipulation efficiency as it reduces the need for active perception of the target object during manipulation. To get a reusable reconstructed model, the proposed pipeline adopts: i) TSDF for object representation, and ii) a variant of the standard particle filter algorithm for pose estimation and tracking of the partial object model. Furthermore, an effective way to construct and maintain a dataset of manipulated objects is presented. A sequence of real-world manipulation experiments is performed. They show how future manipulation tasks become more effective and efficient by reusing reconstructed models of previously manipulated objects, which were generated during their prior manipulation, instead of treating objects as novel every time.}, note = {Nomination for Best Paper Award in Manipulation}, keywords = {}, pubstate = {published}, tppubtype = {inproceedings} } This work proposes a robotic pipeline for picking and constrained placement of objects without geometric shape priors. Compared to recent efforts developed for similar tasks, where every object was assumed to be novel, the proposed system recognizes previously manipulated objects and performs online model reconstruction and reuse. Over a lifelong manipulation process, the system keeps learning features of objects it has interacted with and updates their reconstructed models. Whenever an instance of a previously manipulated object reappears, the system aims to first recognize it and then register its previously reconstructed model given the current observation. This step greatly reduces object shape uncertainty allowing the system to even reason for parts of objects, which are currently not observable. This also results in better manipulation efficiency as it reduces the need for active perception of the target object during manipulation. To get a reusable reconstructed model, the proposed pipeline adopts: i) TSDF for object representation, and ii) a variant of the standard particle filter algorithm for pose estimation and tracking of the partial object model. Furthermore, an effective way to construct and maintain a dataset of manipulated objects is presented. A sequence of real-world manipulation experiments is performed. They show how future manipulation tasks become more effective and efficient by reusing reconstructed models of previously manipulated objects, which were generated during their prior manipulation, instead of treating objects as novel every time. |
Vieira, E; Nakhimovich, D; Gao, K; Wang, R; Yu, J; Bekris, K Persistent Homology for Effective Non-Prehensile Manipulation Inproceedings IEEE International Conference on Robotics and Automation (ICRA), 2022. @inproceedings{Vieira:2022ab, title = {Persistent Homology for Effective Non-Prehensile Manipulation}, author = {E Vieira and D Nakhimovich and K Gao and R Wang and J Yu and K Bekris}, url = {https://arxiv.org/abs/2202.02937}, year = {2022}, date = {2022-05-01}, booktitle = {IEEE International Conference on Robotics and Automation (ICRA)}, abstract = {This work explores the use of topological tools for achieving effective non-prehensile manipulation in cluttered, constrained workspaces. In particular, it proposes the use of persistent homology as a guiding principle in identifying the appropriate non-prehensile actions, such as pushing, to clean a cluttered space with a robotic arm so as to allow the retrieval of a target object. Persistent homology enables the automatic identification of connected components of blocking objects in the space without the need for manual input or tuning of parameters. The proposed algorithm uses this information to push groups of cylindrical objects together and aims to minimize the number of pushing actions needed to reach to the target. Simulated experiments in a physics engine using a model of the Baxter robot show that the proposed topology-driven solution is achieving significantly higher success rate in solving such constrained problems relatively to state-of-the-art alternatives from the literature. It manages to keep the number of pushing actions low, is computationally efficient and the resulting decisions and motion appear natural for effectively solving such tasks.}, keywords = {}, pubstate = {published}, tppubtype = {inproceedings} } This work explores the use of topological tools for achieving effective non-prehensile manipulation in cluttered, constrained workspaces. In particular, it proposes the use of persistent homology as a guiding principle in identifying the appropriate non-prehensile actions, such as pushing, to clean a cluttered space with a robotic arm so as to allow the retrieval of a target object. Persistent homology enables the automatic identification of connected components of blocking objects in the space without the need for manual input or tuning of parameters. The proposed algorithm uses this information to push groups of cylindrical objects together and aims to minimize the number of pushing actions needed to reach to the target. Simulated experiments in a physics engine using a model of the Baxter robot show that the proposed topology-driven solution is achieving significantly higher success rate in solving such constrained problems relatively to state-of-the-art alternatives from the literature. It manages to keep the number of pushing actions low, is computationally efficient and the resulting decisions and motion appear natural for effectively solving such tasks. |
Liang, J; Wen, B; Bekris, K; Boularias, A Learning Sensorimotor Primitives of Sequential Manipulation Tasks from Visual Demonstrations Inproceedings IEEE International Conference on Robotics and Automation (ICRA), 2022. @inproceedings{Liang:2022aa, title = {Learning Sensorimotor Primitives of Sequential Manipulation Tasks from Visual Demonstrations}, author = {J Liang and B Wen and K Bekris and A Boularias}, url = {https://arxiv.org/abs/2203.03797}, year = {2022}, date = {2022-05-01}, booktitle = {IEEE International Conference on Robotics and Automation (ICRA)}, abstract = {This work aims to learn how to perform complex robot manipulation tasks that are composed of several, consecutively executed low-level sub-tasks, given as input a few visual demonstrations of the tasks performed by a person. The sub-tasks consist of moving the robot's end-effector until it reaches a sub-goal region in the task space, performing an action, and triggering the next sub-task when a pre-condition is met. Most prior work in this domain has been concerned with learning only low-level tasks, such as hitting a ball or reaching an object and grasping it. This paper describes a new neural network-based framework for learning simultaneously low-level policies as well as high-level policies, such as deciding which object to pick next or where to place it relative to other objects in the scene. A key feature of the proposed approach is that the policies are learned directly from raw videos of task demonstrations, without any manual annotation or post-processing of the data. Empirical results on object manipulation tasks with a robotic arm show that the proposed network can efficiently learn from real visual demonstrations to perform the tasks, and outperforms popular imitation learning algorithms.}, keywords = {}, pubstate = {published}, tppubtype = {inproceedings} } This work aims to learn how to perform complex robot manipulation tasks that are composed of several, consecutively executed low-level sub-tasks, given as input a few visual demonstrations of the tasks performed by a person. The sub-tasks consist of moving the robot's end-effector until it reaches a sub-goal region in the task space, performing an action, and triggering the next sub-task when a pre-condition is met. Most prior work in this domain has been concerned with learning only low-level tasks, such as hitting a ball or reaching an object and grasping it. This paper describes a new neural network-based framework for learning simultaneously low-level policies as well as high-level policies, such as deciding which object to pick next or where to place it relative to other objects in the scene. A key feature of the proposed approach is that the policies are learned directly from raw videos of task demonstrations, without any manual annotation or post-processing of the data. Empirical results on object manipulation tasks with a robotic arm show that the proposed network can efficiently learn from real visual demonstrations to perform the tasks, and outperforms popular imitation learning algorithms. |
Gao, K; Lau, D; Huang, B; Bekris, K; Yu, J Fast High-Quality Tabletop Rearrangement in Bounded Workspace Inproceedings IEEE International Conference on Robotics and Automation (ICRA), 2022. @inproceedings{Gao:2022aa, title = {Fast High-Quality Tabletop Rearrangement in Bounded Workspace}, author = {K Gao and D Lau and B Huang and K Bekris and J Yu}, url = {https://arxiv.org/abs/2110.12325}, year = {2022}, date = {2022-05-01}, booktitle = {IEEE International Conference on Robotics and Automation (ICRA)}, abstract = {In this paper, we examine the problem of rearranging many objects on a tabletop in a cluttered setting using overhand grasps. Efficient solutions for the problem, which capture a common task that we solve on a daily basis, are essential in enabling truly intelligent robotic manipulation. In a given instance, objects may need to be placed at temporary positions (buffers) to complete the rearrangement, but allocating these buffer locations can be highly challenging in a cluttered environment. To tackle the challenge, a two-step baseline planner is first developed, which generates a primitive plan based on inherent combinatorial constraints induced by start and goal poses of the objects and then selects buffer locations assisted by the primitive plan. We then employ the "lazy" planner in a tree search framework which is further sped up by adapting a novel preprocessing routine. Simulation experiments show our methods can quickly generate high-quality solutions and are more robust in solving large-scale instances than existing state-of-the-art approaches.}, keywords = {}, pubstate = {published, manipulation}, tppubtype = {inproceedings} } In this paper, we examine the problem of rearranging many objects on a tabletop in a cluttered setting using overhand grasps. Efficient solutions for the problem, which capture a common task that we solve on a daily basis, are essential in enabling truly intelligent robotic manipulation. In a given instance, objects may need to be placed at temporary positions (buffers) to complete the rearrangement, but allocating these buffer locations can be highly challenging in a cluttered environment. To tackle the challenge, a two-step baseline planner is first developed, which generates a primitive plan based on inherent combinatorial constraints induced by start and goal poses of the objects and then selects buffer locations assisted by the primitive plan. We then employ the "lazy" planner in a tree search framework which is further sped up by adapting a novel preprocessing routine. Simulation experiments show our methods can quickly generate high-quality solutions and are more robust in solving large-scale instances than existing state-of-the-art approaches. |
Wang, R; Miao, Y; Bekris, K Efficient and High-Quality Prehensile Rearrangement in Cluttered and Confined Spaces Inproceedings IEEE International Conference on Robotics and Automation (ICRA), 2022. @inproceedings{Wang:2022ab, title = {Efficient and High-Quality Prehensile Rearrangement in Cluttered and Confined Spaces}, author = {R Wang and Y Miao and K Bekris}, url = {https://arxiv.org/abs/2110.02814}, year = {2022}, date = {2022-05-01}, booktitle = {IEEE International Conference on Robotics and Automation (ICRA)}, abstract = {Prehensile object rearrangement in cluttered and confined spaces has broad applications but is also challenging. For instance, rearranging products in a grocery shelf means that the robot cannot directly access all objects and has limited free space. This is harder than tabletop rearrangement where objects are easily accessible with top-down grasps, which simplifies robot-object interactions. This work focuses on problems where such interactions are critical for completing tasks. It proposes a new efficient and complete solver under general constraints for monotone instances, which can be solved by moving each object at most once. The monotone solver reasons about robot-object constraints and uses them to effectively prune the search space. The new monotone solver is integrated with a global planner to solve non-monotone instances with high-quality solutions fast. Furthermore, this work contributes an effective pre-processing tool to significantly speed up online motion planning queries for rearrangement in confined spaces. Experiments further demonstrate that the proposed monotone solver, equipped with the pre-processing tool, results in 57.3% faster computation and 3 times higher success rate than state-of-the-art methods. Similarly, the resulting global planner is computationally more efficient and has a higher success rate, while producing high-quality solutions for non-monotone instances (i.e., only 1.3 additional actions are needed on average).}, keywords = {}, pubstate = {published}, tppubtype = {inproceedings} } Prehensile object rearrangement in cluttered and confined spaces has broad applications but is also challenging. For instance, rearranging products in a grocery shelf means that the robot cannot directly access all objects and has limited free space. This is harder than tabletop rearrangement where objects are easily accessible with top-down grasps, which simplifies robot-object interactions. This work focuses on problems where such interactions are critical for completing tasks. It proposes a new efficient and complete solver under general constraints for monotone instances, which can be solved by moving each object at most once. The monotone solver reasons about robot-object constraints and uses them to effectively prune the search space. The new monotone solver is integrated with a global planner to solve non-monotone instances with high-quality solutions fast. Furthermore, this work contributes an effective pre-processing tool to significantly speed up online motion planning queries for rearrangement in confined spaces. Experiments further demonstrate that the proposed monotone solver, equipped with the pre-processing tool, results in 57.3% faster computation and 3 times higher success rate than state-of-the-art methods. Similarly, the resulting global planner is computationally more efficient and has a higher success rate, while producing high-quality solutions for non-monotone instances (i.e., only 1.3 additional actions are needed on average). |
Wen, B; Lian, W; Bekris, K; Schaal, S Catgrasp: Learning Category-Level Task-Relevant Grasping in Clutter from Simulation Inproceedings IEEE International Conference on Robotics and Automation (ICRA), 2022. @inproceedings{Wen:2022aa, title = {Catgrasp: Learning Category-Level Task-Relevant Grasping in Clutter from Simulation}, author = {B Wen and W Lian and K Bekris and S Schaal}, url = {https://arxiv.org/abs/2109.09163}, year = {2022}, date = {2022-05-01}, booktitle = {IEEE International Conference on Robotics and Automation (ICRA)}, abstract = {Task-relevant grasping is critical for industrial assembly, where downstream manipulation tasks constrain the set of valid grasps. Learning how to perform this task, however, is challenging, since task-relevant grasp labels are hard to define and annotate. There is also yet no consensus on proper representations for modeling or off-the-shelf tools for performing task-relevant grasps. This work proposes a framework to learn task-relevant grasping for industrial objects without the need of time-consuming real-world data collection or manual annotation. To achieve this, the entire framework is trained solely in simulation, including supervised training with synthetic label generation and self-supervised, hand-object interaction. In the context of this framework, this paper proposes a novel, object-centric canonical representation at the category level, which allows establishing dense correspondence across object instances and transferring task-relevant grasps to novel instances. Extensive experiments on task-relevant grasping of densely-cluttered industrial objects are conducted in both simulation and real-world setups, demonstrating the effectiveness of the proposed framework. Code and data is released at https://sites.google.com/view/catgrasp.}, keywords = {}, pubstate = {published}, tppubtype = {inproceedings} } Task-relevant grasping is critical for industrial assembly, where downstream manipulation tasks constrain the set of valid grasps. Learning how to perform this task, however, is challenging, since task-relevant grasp labels are hard to define and annotate. There is also yet no consensus on proper representations for modeling or off-the-shelf tools for performing task-relevant grasps. This work proposes a framework to learn task-relevant grasping for industrial objects without the need of time-consuming real-world data collection or manual annotation. To achieve this, the entire framework is trained solely in simulation, including supervised training with synthetic label generation and self-supervised, hand-object interaction. In the context of this framework, this paper proposes a novel, object-centric canonical representation at the category level, which allows establishing dense correspondence across object instances and transferring task-relevant grasps to novel instances. Extensive experiments on task-relevant grasping of densely-cluttered industrial objects are conducted in both simulation and real-world setups, demonstrating the effectiveness of the proposed framework. Code and data is released at https://sites.google.com/view/catgrasp. |
Morgan, A; Hang, K; Wen, B; Bekris, K; Dollar, A Complex In-Hand Manipulation Via Compliance-Enabled Finger Gaiting and Multi-Modal Planning Journal Article IEEE Robotics and Automation Letters (also at ICRA), 2022. @article{Morgan:2022aa, title = {Complex In-Hand Manipulation Via Compliance-Enabled Finger Gaiting and Multi-Modal Planning}, author = {A Morgan and K Hang and B Wen and K Bekris and A Dollar}, url = {https://arxiv.org/abs/2201.07928}, year = {2022}, date = {2022-05-01}, journal = {IEEE Robotics and Automation Letters (also at ICRA)}, abstract = {Constraining contacts to remain fixed on an object during manipulation limits the potential workspace size, as motion is subject to the hand's kinematic topology. Finger gaiting is one way to alleviate such restraints. It allows contacts to be freely broken and remade so as to operate on different manipulation manifolds. This capability, however, has traditionally been difficult or impossible to practically realize. A finger gaiting system must simultaneously plan for and control forces on the object while maintaining stability during contact switching. This work alleviates the traditional requirement by taking advantage of system compliance, allowing the hand to more easily switch contacts while maintaining a stable grasp. Our method achieves complete SO(3) finger gaiting control of grasped objects against gravity by developing a manipulation planner that operates via orthogonal safe modes of a compliant, underactuated hand absent of tactile sensors or joint encoders. During manipulation, a low-latency 6D pose object tracker provides feedback via vision, allowing the planner to update its plan online so as to adaptively recover from trajectory deviations. The efficacy of this method is showcased by manipulating both convex and non-convex objects on a real robot. Its robustness is evaluated via perturbation rejection and long trajectory goals. To the best of the authors' knowledge, this is the first work that has autonomously achieved full SO(3) control of objects within-hand via finger gaiting and without a support surface, elucidating a valuable step towards realizing true robot in-hand manipulation capabilities.}, keywords = {}, pubstate = {published}, tppubtype = {article} } Constraining contacts to remain fixed on an object during manipulation limits the potential workspace size, as motion is subject to the hand's kinematic topology. Finger gaiting is one way to alleviate such restraints. It allows contacts to be freely broken and remade so as to operate on different manipulation manifolds. This capability, however, has traditionally been difficult or impossible to practically realize. A finger gaiting system must simultaneously plan for and control forces on the object while maintaining stability during contact switching. This work alleviates the traditional requirement by taking advantage of system compliance, allowing the hand to more easily switch contacts while maintaining a stable grasp. Our method achieves complete SO(3) finger gaiting control of grasped objects against gravity by developing a manipulation planner that operates via orthogonal safe modes of a compliant, underactuated hand absent of tactile sensors or joint encoders. During manipulation, a low-latency 6D pose object tracker provides feedback via vision, allowing the planner to update its plan online so as to adaptively recover from trajectory deviations. The efficacy of this method is showcased by manipulating both convex and non-convex objects on a real robot. Its robustness is evaluated via perturbation rejection and long trajectory goals. To the best of the authors' knowledge, this is the first work that has autonomously achieved full SO(3) control of objects within-hand via finger gaiting and without a support surface, elucidating a valuable step towards realizing true robot in-hand manipulation capabilities. |
Miao, Y; Wang, R; Bekris, K Safe, Occlusion-Aware Manipulation for Online Object Reconstruction in Confined Space Inproceedings International Symposium on Robotics Research (ISRR), 2022. @inproceedings{Miao:2022aa, title = {Safe, Occlusion-Aware Manipulation for Online Object Reconstruction in Confined Space}, author = {Y Miao and R Wang and K Bekris}, url = {https://arxiv.org/abs/2205.11719}, year = {2022}, date = {2022-01-01}, booktitle = {International Symposium on Robotics Research (ISRR)}, abstract = {Recent work in robotic manipulation focuses on object retrieval in cluttered space under occlusion. Nevertheless, the majority of efforts lack an analysis of conditions for the completeness of the approaches or the methods apply only when objects can be removed from the workspace. This work formulates the general, occlusion-aware manipulation task, and focuses on safe object reconstruction in a confined space with in-place relocation. A framework that ensures safety with completeness guarantees is proposed. Furthermore, an algorithm, which is an instantiation of this framework for monotone instances, is developed and evaluated empirically by comparing against a random and a greedy baseline on randomly generated experiments in simulation. Even for cluttered scenes with realistic objects, the proposed algorithm significantly outperforms the baselines and maintains a high success rate across experimental conditions.}, keywords = {}, pubstate = {published}, tppubtype = {inproceedings} } Recent work in robotic manipulation focuses on object retrieval in cluttered space under occlusion. Nevertheless, the majority of efforts lack an analysis of conditions for the completeness of the approaches or the methods apply only when objects can be removed from the workspace. This work formulates the general, occlusion-aware manipulation task, and focuses on safe object reconstruction in a confined space with in-place relocation. A framework that ensures safety with completeness guarantees is proposed. Furthermore, an algorithm, which is an instantiation of this framework for monotone instances, is developed and evaluated empirically by comparing against a random and a greedy baseline on randomly generated experiments in simulation. Even for cluttered scenes with realistic objects, the proposed algorithm significantly outperforms the baselines and maintains a high success rate across experimental conditions. |
2021 |
Morgan, A; Wen, B; Junchi, L; Boularias, A; Dollar, A; Bekris, K Vision-Driven Compliant Manipulation for Reliable, High-Precision Assembly Tasks Conference Robotics: Science and Systems, 2021. @conference{Morgan:2021aa, title = {Vision-Driven Compliant Manipulation for Reliable, High-Precision Assembly Tasks}, author = {A Morgan and B Wen and L Junchi and A Boularias and A Dollar and K Bekris}, year = {2021}, date = {2021-07-01}, booktitle = {Robotics: Science and Systems}, abstract = {Highly constrained manipulation tasks continue to be challenging for autonomous robots as they require high levels of precision, typically less than 1mm, which is often incompatible with what can be achieved by traditional perception systems. This paper demonstrates that the combination of state-of-the-art object tracking with passively adaptive mechanical hardware can be leveraged to complete precision manipulation tasks with tight, industrially-relevant tolerances (0.25mm). The proposed control method closes the loop through vision by tracking the relative 6D pose of objects in the relevant workspace. It adjusts the control reference of both the compliant manipulator and the hand to complete object insertion tasks via within-hand manipulation. Contrary to previous efforts for insertion, our method does not require expensive force sensors, precision manipulators, or time-consuming, online learning, which is data hungry. Instead, this effort leverages mechanical compliance and utilizes an object-agnostic manipulation model of the hand learned offline, off-the-shelf motion planning, and an RGBD-based object tracker trained solely with synthetic data. These features allow the proposed system to easily generalize and transfer to new tasks and environments. This paper describes in detail the system components and showcases its efficacy with extensive experiments involving tight tolerance peg-in-hole insertion tasks of various geometries as well as open-world constrained placement tasks.}, keywords = {}, pubstate = {published}, tppubtype = {conference} } Highly constrained manipulation tasks continue to be challenging for autonomous robots as they require high levels of precision, typically less than 1mm, which is often incompatible with what can be achieved by traditional perception systems. This paper demonstrates that the combination of state-of-the-art object tracking with passively adaptive mechanical hardware can be leveraged to complete precision manipulation tasks with tight, industrially-relevant tolerances (0.25mm). The proposed control method closes the loop through vision by tracking the relative 6D pose of objects in the relevant workspace. It adjusts the control reference of both the compliant manipulator and the hand to complete object insertion tasks via within-hand manipulation. Contrary to previous efforts for insertion, our method does not require expensive force sensors, precision manipulators, or time-consuming, online learning, which is data hungry. Instead, this effort leverages mechanical compliance and utilizes an object-agnostic manipulation model of the hand learned offline, off-the-shelf motion planning, and an RGBD-based object tracker trained solely with synthetic data. These features allow the proposed system to easily generalize and transfer to new tasks and environments. This paper describes in detail the system components and showcases its efficacy with extensive experiments involving tight tolerance peg-in-hole insertion tasks of various geometries as well as open-world constrained placement tasks. |
Wang, R; Gao, K; Nakhimovich, D; Yu, J; Bekris, K Uniform Object Rearrangement: From Complete Monotone Primitives to Efficient Non-Monotone Informed Search Inproceedings International Conference on Robotics and Automation (ICRA) 2021, 2021. @inproceedings{Wang:2021ac, title = {Uniform Object Rearrangement: From Complete Monotone Primitives to Efficient Non-Monotone Informed Search}, author = {R Wang and K Gao and D Nakhimovich and J Yu and K Bekris}, url = {https://ieeexplore.ieee.org/document/9561716}, year = {2021}, date = {2021-05-01}, booktitle = {International Conference on Robotics and Automation (ICRA) 2021}, abstract = {Object rearrangement is a widely-applicable and challenging task for robots. Geometric constraints must be carefully examined to avoid collisions and combinatorial issues arise as the number of objects increases. This work studies the algorithmic structure of rearranging uniform objects, where robot-object collisions do not occur but object-object collisions have to be avoided. The objective is minimizing the number of object transfers under the assumption that the robot can manipulate one object at a time. An efficiently computable decomposition of the configuration space is used to create a "region graph", which classifies all continuous paths of equivalent collision possibilities. Based on this compact but rich representation, a complete dynamic programming primitive DFSDP performs a recursive depth first search to solve monotone problems quickly, i.e., those instances that do not require objects to be moved first to an intermediate buffer. DFSDP is extended to solve single-buffer, non-monotone instances, given a choice of an object and a buffer. This work utilizes these primitives as local planners in an informed search framework for more general, non-monotone instances. The search utilizes partial solutions from the primitives to identify the most promising choice of objects and buffers. Experiments demonstrate that the proposed solution returns near-optimal paths with higher success rate, even for challenging non-monotone instances, than other leading alternatives.}, keywords = {}, pubstate = {published}, tppubtype = {inproceedings} } Object rearrangement is a widely-applicable and challenging task for robots. Geometric constraints must be carefully examined to avoid collisions and combinatorial issues arise as the number of objects increases. This work studies the algorithmic structure of rearranging uniform objects, where robot-object collisions do not occur but object-object collisions have to be avoided. The objective is minimizing the number of object transfers under the assumption that the robot can manipulate one object at a time. An efficiently computable decomposition of the configuration space is used to create a "region graph", which classifies all continuous paths of equivalent collision possibilities. Based on this compact but rich representation, a complete dynamic programming primitive DFSDP performs a recursive depth first search to solve monotone problems quickly, i.e., those instances that do not require objects to be moved first to an intermediate buffer. DFSDP is extended to solve single-buffer, non-monotone instances, given a choice of an object and a buffer. This work utilizes these primitives as local planners in an informed search framework for more general, non-monotone instances. The search utilizes partial solutions from the primitives to identify the most promising choice of objects and buffers. Experiments demonstrate that the proposed solution returns near-optimal paths with higher success rate, even for challenging non-monotone instances, than other leading alternatives. |
Shome, R; Solovey, K; Yu, J; Bekris, K; Halperin, D Fast, High-Quality Two-Arm Rearrangement in Synchronous, Monotone Tabletop Setups Journal Article IEEE Transactions on Automation Science and Engineering, 2021. @article{Shome:2021aa, title = {Fast, High-Quality Two-Arm Rearrangement in Synchronous, Monotone Tabletop Setups}, author = {R Shome and K Solovey and J Yu and K Bekris and D Halperin}, url = {https://arxiv.org/abs/1810.12202}, year = {2021}, date = {2021-03-01}, journal = {IEEE Transactions on Automation Science and Engineering}, abstract = {Rearranging objects on a planar surface arises in a variety of robotic applications, such as product packaging. Using two arms can improve efficiency but introduces new computational challenges. This paper studies the problem structure of object rearrangement using two arms in synchronous, monotone tabletop setups and develops an optimal mixed integer model. It then describes an efficient and scalable algorithm, which first minimizes the cost of object transfers and then of moves between objects. This is motivated by the fact that, asymptotically, object transfers dominate the cost of solutions. Moreover, a lazy strategy minimizes the number of motion planning calls and results in significant speedups. Theoretical arguments support the benefits of using two arms and indicate that synchronous execution, in which the two arms perform together either transfers or moves, introduces only a small overhead. Experiments support these claims and show that the scalable method can quickly compute solutions close to the optimal for the considered setup.}, keywords = {}, pubstate = {published}, tppubtype = {article} } Rearranging objects on a planar surface arises in a variety of robotic applications, such as product packaging. Using two arms can improve efficiency but introduces new computational challenges. This paper studies the problem structure of object rearrangement using two arms in synchronous, monotone tabletop setups and develops an optimal mixed integer model. It then describes an efficient and scalable algorithm, which first minimizes the cost of object transfers and then of moves between objects. This is motivated by the fact that, asymptotically, object transfers dominate the cost of solutions. Moreover, a lazy strategy minimizes the number of motion planning calls and results in significant speedups. Theoretical arguments support the benefits of using two arms and indicate that synchronous execution, in which the two arms perform together either transfers or moves, introduces only a small overhead. Experiments support these claims and show that the scalable method can quickly compute solutions close to the optimal for the considered setup. |
Feng, S; Guo, T; Bekris, K; Yu, J Team Rubot's Experiences and Lessons from the Ariac Journal Article Robotics and Computer-Integrated Manufacturing, 70 , 2021. @article{Feng:2021aa, title = {Team Rubot's Experiences and Lessons from the Ariac}, author = {S Feng and T Guo and K Bekris and J Yu}, editor = {Erez Karpas}, url = {https://www.sciencedirect.com/science/article/abs/pii/S0736584521000120}, year = {2021}, date = {2021-01-01}, journal = {Robotics and Computer-Integrated Manufacturing}, volume = {70}, abstract = {We share experiences and lessons learned in participating the annual Agile Robotics for Industrial Automation Competition (ARIAC). ARIAC is a simulation-based competition focusing on pushing the agility of robotic systems for handling industrial pick-and-place challenges. Team RuBot started competing from 2019, placing 2nd place in ARIAC 2019 and 3rd place in ARIAC 2020. The article also discusses the difficulties we faced during the contest and our strategies for tackling them.}, keywords = {}, pubstate = {published}, tppubtype = {article} } We share experiences and lessons learned in participating the annual Agile Robotics for Industrial Automation Competition (ARIAC). ARIAC is a simulation-based competition focusing on pushing the agility of robotic systems for handling industrial pick-and-place challenges. Team RuBot started competing from 2019, placing 2nd place in ARIAC 2019 and 3rd place in ARIAC 2020. The article also discusses the difficulties we faced during the contest and our strategies for tackling them. |
2020 |
Mitash, C; Shome, R; Wen, B; Boularias, A; Bekris, K Task-Driven Perception and Manipulation for Constrained Placement of Unknown Objects Journal Article IEEE Robotics and Automation Letters (RA-L) (also appearing at IEEE/RSJ IROS 2020), 2020. @article{Mitash:2020ab, title = {Task-Driven Perception and Manipulation for Constrained Placement of Unknown Objects}, author = {C Mitash and R Shome and B Wen and A Boularias and K Bekris}, url = {https://arxiv.org/abs/2006.15503}, year = {2020}, date = {2020-10-01}, journal = {IEEE Robotics and Automation Letters (RA-L) (also appearing at IEEE/RSJ IROS 2020)}, abstract = {Recent progress in robotic manipulation has dealt with the case of no prior object models in the context of relatively simple tasks, such as bin-picking. Existing methods for more constrained problems, however, such as deliberate placement in a tight region, depend more critically on shape information to achieve safe execution. This work introduces a possibilistic object representation for solving constrained placement tasks without shape priors. A perception method is proposed to track and update the object representation during motion execution, which respects physical and geometric constraints. The method operates directly over sensor data, modeling the seen and unseen parts of the object given observations. It results in a dynamically updated conservative representation, which can be used to plan safe manipulation actions. This task-driven perception process is integrated with manipulation task planning architecture for a dual-arm manipulator to discover efficient solutions for the constrained placement task with minimal sensing. The planning process can make use of handoff operations when necessary for safe placement given the conservative representation. The pipeline is evaluated with data from over 240 real-world experiments involving constrained placement of various unknown objects using a dual-arm manipulator. While straightforward pick-sense-and-place architectures frequently fail to solve these problems, the proposed integrated pipeline achieves more than 95% success and faster execution times.}, keywords = {}, pubstate = {published}, tppubtype = {article} } Recent progress in robotic manipulation has dealt with the case of no prior object models in the context of relatively simple tasks, such as bin-picking. Existing methods for more constrained problems, however, such as deliberate placement in a tight region, depend more critically on shape information to achieve safe execution. This work introduces a possibilistic object representation for solving constrained placement tasks without shape priors. A perception method is proposed to track and update the object representation during motion execution, which respects physical and geometric constraints. The method operates directly over sensor data, modeling the seen and unseen parts of the object given observations. It results in a dynamically updated conservative representation, which can be used to plan safe manipulation actions. This task-driven perception process is integrated with manipulation task planning architecture for a dual-arm manipulator to discover efficient solutions for the constrained placement task with minimal sensing. The planning process can make use of handoff operations when necessary for safe placement given the conservative representation. The pipeline is evaluated with data from over 240 real-world experiments involving constrained placement of various unknown objects using a dual-arm manipulator. While straightforward pick-sense-and-place architectures frequently fail to solve these problems, the proposed integrated pipeline achieves more than 95% success and faster execution times. |
Mitash, C Scalable, Physics-Aware 6d Pose Estimation for Robot Manipulation PhD Thesis Rutgers University, 2020. @phdthesis{Mitash:2020aa, title = {Scalable, Physics-Aware 6d Pose Estimation for Robot Manipulation}, author = {C Mitash}, url = {https://rucore.libraries.rutgers.edu/rutgers-lib/64961/}, year = {2020}, date = {2020-09-01}, school = {Rutgers University}, abstract = {Robot manipulation often depend on some form of pose estimation to represent the state of the world and allow decision making both at the task-level and for motion or grasp planning. Recent progress in deep learning gives hope for a pose estimation solution that could generalize over textured and texture-less objects, objects with or without distinctive shape properties, and under different lighting conditions and clutter scenarios. Nevertheless, it gives rise to a new set of challenges such as the painful task of acquiring large-scale labeled training datasets and of dealing with their stochastic output over unforeseen scenarios that are not captured by the training. This restricts the scalability of such pose estimation solutions in robot manipulation tasks that often deal with a variety of objects and changing environments. The thesis first describes an automatic data generation and learning framework to address the scalability challenge. Learning is bootstrapped by generating labeled data via physics simulation and rendering. Then it self-improves over time by acquiring and labeling real-world images via a search-based pose estimation process. The thesis proposes algorithms to generate and validate object poses online based on the objects' geometry and based on the physical consistency of their scene-level interactions. These algorithms provide robustness even when there exists a domain gap between the synthetic training and the real test scenarios. Finally, the thesis proposes a manipulation planning framework that goes beyond model-based pose estimation. By utilizing a dynamic object representation, this integrated perception and manipulation framework can efficiently solve the task of picking unknown objects and placing them in a constrained space. The algorithms are evaluated over real-world robot manipulation experiments and over large-scale public datasets. The results indicate the usefulness of physical constraints in both the training and the online estimation phase. Moreover, the proposed framework, while only utilizing simulated data can obtain robust estimation in challenging scenarios such as densely-packed bins and clutter where other approaches suffer as a result of large occlusion and ambiguities due to similar looking texture-less surfaces.}, keywords = {}, pubstate = {published}, tppubtype = {phdthesis} } Robot manipulation often depend on some form of pose estimation to represent the state of the world and allow decision making both at the task-level and for motion or grasp planning. Recent progress in deep learning gives hope for a pose estimation solution that could generalize over textured and texture-less objects, objects with or without distinctive shape properties, and under different lighting conditions and clutter scenarios. Nevertheless, it gives rise to a new set of challenges such as the painful task of acquiring large-scale labeled training datasets and of dealing with their stochastic output over unforeseen scenarios that are not captured by the training. This restricts the scalability of such pose estimation solutions in robot manipulation tasks that often deal with a variety of objects and changing environments. The thesis first describes an automatic data generation and learning framework to address the scalability challenge. Learning is bootstrapped by generating labeled data via physics simulation and rendering. Then it self-improves over time by acquiring and labeling real-world images via a search-based pose estimation process. The thesis proposes algorithms to generate and validate object poses online based on the objects' geometry and based on the physical consistency of their scene-level interactions. These algorithms provide robustness even when there exists a domain gap between the synthetic training and the real test scenarios. Finally, the thesis proposes a manipulation planning framework that goes beyond model-based pose estimation. By utilizing a dynamic object representation, this integrated perception and manipulation framework can efficiently solve the task of picking unknown objects and placing them in a constrained space. The algorithms are evaluated over real-world robot manipulation experiments and over large-scale public datasets. The results indicate the usefulness of physical constraints in both the training and the online estimation phase. Moreover, the proposed framework, while only utilizing simulated data can obtain robust estimation in challenging scenarios such as densely-packed bins and clutter where other approaches suffer as a result of large occlusion and ambiguities due to similar looking texture-less surfaces. |
Shome, R; Bekris, K Synchronized Multi-Arm Rearrangement Guided by Mode Graphs with Capacity Constraints Conference Workshop on the Algorithmic Foundations of Robotics (WAFR), Oulu, Finland, 2020. @conference{Shome:2020ac, title = {Synchronized Multi-Arm Rearrangement Guided by Mode Graphs with Capacity Constraints}, author = {R Shome and K Bekris}, url = {https://arxiv.org/abs/2005.09127}, year = {2020}, date = {2020-06-01}, booktitle = {Workshop on the Algorithmic Foundations of Robotics (WAFR)}, address = {Oulu, Finland}, abstract = {Solving task planning problems involving multiple objects and multiple robotic arms poses scalability challenges. Such problems involve not only coordinating multiple high-DoF arms, but also searching through possible sequences of actions including object placements, and handoffs. The current work identifies a useful connection between multi-arm rearrangement and recent results in multi-body path planning on graphs with vertex capacity constraints. Solving a synchronized multi-arm rearrangement at a high-level involves reasoning over a modal graph, where nodes correspond to stable object placements and object transfer states by the arms. Edges of this graph correspond to pick, placement and handoff operations. The objects can be viewed as pebbles moving over this graph, which has capacity constraints. For instance, each arm can carry a single object but placement locations can accumulate many objects. Efficient integer linear programming-based solvers have been proposed for the corresponding pebble problem. The current work proposes a heuristic to guide the task planning process for synchronized multi-arm rearrangement. Results indicate good scalability to multiple arms and objects, and an algorithm that can find high-quality solutions fast and exhibiting desirable anytime behavior.}, keywords = {}, pubstate = {published}, tppubtype = {conference} } Solving task planning problems involving multiple objects and multiple robotic arms poses scalability challenges. Such problems involve not only coordinating multiple high-DoF arms, but also searching through possible sequences of actions including object placements, and handoffs. The current work identifies a useful connection between multi-arm rearrangement and recent results in multi-body path planning on graphs with vertex capacity constraints. Solving a synchronized multi-arm rearrangement at a high-level involves reasoning over a modal graph, where nodes correspond to stable object placements and object transfer states by the arms. Edges of this graph correspond to pick, placement and handoff operations. The objects can be viewed as pebbles moving over this graph, which has capacity constraints. For instance, each arm can carry a single object but placement locations can accumulate many objects. Efficient integer linear programming-based solvers have been proposed for the corresponding pebble problem. The current work proposes a heuristic to guide the task planning process for synchronized multi-arm rearrangement. Results indicate good scalability to multiple arms and objects, and an algorithm that can find high-quality solutions fast and exhibiting desirable anytime behavior. |
Sintov, A; Kimmel, A; Bekris, K; Boularias, A Motion Planning with Competency-Aware Transition Models for Underactuated Adaptive Hands Conference IEEE International Conference on Robotics and Automation (ICRA), Paris, France, 2020. @conference{Sintov:2020aa, title = {Motion Planning with Competency-Aware Transition Models for Underactuated Adaptive Hands}, author = {A Sintov and A Kimmel and K Bekris and A Boularias}, url = {https://ieeexplore.ieee.org/document/9196564}, year = {2020}, date = {2020-06-01}, booktitle = {IEEE International Conference on Robotics and Automation (ICRA)}, address = {Paris, France}, abstract = {Underactuated adaptive hands simplify grasping tasks but it is difficult to model their interactions with objects during in-hand manipulation. Learned data-driven models have been recently shown to be efficient in motion planning and control of such hands. Still, the accuracy of the models is limited even with the addition of more data. This becomes important for long horizon predictions, where errors are accumulated along the length of a path. Instead of throwing more data into learning the transition model, this work proposes to rather invest a portion of the training data in a critic model. The critic is trained to estimate the error of the transition model given a state and a sequence of future actions, along with information of past actions. The critic is used to reformulate the cost function of an asymptotically optimal motion planner. Given the critic, the planner directs planned paths to less erroneous regions in the state space. The approach is evaluated against standard motion planning on simulated and real hands. The results show that it outperforms an alternative where all the available data is used for training the transition model without a critic.}, keywords = {}, pubstate = {published}, tppubtype = {conference} } Underactuated adaptive hands simplify grasping tasks but it is difficult to model their interactions with objects during in-hand manipulation. Learned data-driven models have been recently shown to be efficient in motion planning and control of such hands. Still, the accuracy of the models is limited even with the addition of more data. This becomes important for long horizon predictions, where errors are accumulated along the length of a path. Instead of throwing more data into learning the transition model, this work proposes to rather invest a portion of the training data in a critic model. The critic is trained to estimate the error of the transition model given a state and a sequence of future actions, along with information of past actions. The critic is used to reformulate the cost function of an asymptotically optimal motion planner. Given the critic, the planner directs planned paths to less erroneous regions in the state space. The approach is evaluated against standard motion planning on simulated and real hands. The results show that it outperforms an alternative where all the available data is used for training the transition model without a critic. |
Alikhani, M; Khalid, B; Shome, R; Mitash, C; Bekris, K; Stone, M That and There: Judging the Intent of Pointing Actions with Robotic Arms Conference Thirty-Fourth AAAI Conference on Artificial Intelligence (AAAI-20), New York, NY, 2020. @conference{Alikhani:2020aa, title = {That and There: Judging the Intent of Pointing Actions with Robotic Arms}, author = {M Alikhani and B Khalid and R Shome and C Mitash and K Bekris and M Stone}, url = {https://cdn.aaai.org/ojs/6601/6601-13-9829-1-10-20200520.pdf}, year = {2020}, date = {2020-02-01}, booktitle = {Thirty-Fourth AAAI Conference on Artificial Intelligence (AAAI-20)}, journal = {Thirty-Fourth AAAI Conference on Artificial Intelligence (AAAI-20)}, address = {New York, NY}, abstract = {Collaborative robotics requires effective communication between a robot and a human partner. This work proposes a set of interpretive principles for how a robotic arm can use pointing actions to communicate task information to people by extending existing models from the related literature. These principles are evaluated through studies where English-speaking human subjects view animations of simulated robots instructing pick-and-place tasks. The evaluation distinguishes two classes of pointing actions that arise in pick-and-place tasks: referential pointing (identifying objects) and spatial pointing (identifying locations). The study indicates that human subjects show greater flexibility in interpreting the intent of referential pointing compared to spatial pointing, which needs to be more deliberate. The results also demonstrate the effects of variation in the environment and task context on the interpretation of pointing. The corpus and the experiments described in this work can impact models of context and coordination as well as the effect of common sense reasoning in human-robot interactions.}, keywords = {}, pubstate = {published}, tppubtype = {conference} } Collaborative robotics requires effective communication between a robot and a human partner. This work proposes a set of interpretive principles for how a robotic arm can use pointing actions to communicate task information to people by extending existing models from the related literature. These principles are evaluated through studies where English-speaking human subjects view animations of simulated robots instructing pick-and-place tasks. The evaluation distinguishes two classes of pointing actions that arise in pick-and-place tasks: referential pointing (identifying objects) and spatial pointing (identifying locations). The study indicates that human subjects show greater flexibility in interpreting the intent of referential pointing compared to spatial pointing, which needs to be more deliberate. The results also demonstrate the effects of variation in the environment and task context on the interpretation of pointing. The corpus and the experiments described in this work can impact models of context and coordination as well as the effect of common sense reasoning in human-robot interactions. |
Wang, R; Mitash, C; Lu, S; Boehm, D; Bekris, K Safe and Effective Picking Paths in Clutter Given Discrete Distributions of Object Poses Conference IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Las Vegas, NV, 2020. @conference{Wang:2020ab, title = {Safe and Effective Picking Paths in Clutter Given Discrete Distributions of Object Poses}, author = {R Wang and C Mitash and S Lu and D Boehm and K Bekris}, url = {https://arxiv.org/abs/2008.04465}, year = {2020}, date = {2020-01-01}, booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)}, address = {Las Vegas, NV}, abstract = {Picking an item in the presence of other objects can be challenging as it involves occlusions and partial views. Given object models, one approach is to perform object pose estimation and use the most likely candidate pose per object to pick the target without collisions. This approach, however, ignores the uncertainty of the perception process both regarding the target's and the surrounding objects' poses. This work proposes first a perception process for 6D pose estimation, which returns a discrete distribution of object poses in a scene. Then, an open-loop planning pipeline is proposed to return safe and effective solutions for moving a robotic arm to pick, which (a) minimizes the probability of collision with the obstructing objects; and (b) maximizes the probability of reaching the target item. The planning framework models the challenge as a stochastic variant of the Minimum Constraint Removal (MCR) problem. The effectiveness of the methodology is verified given both simulated and real data in different scenarios. The experiments demonstrate the importance of considering the uncertainty of the perception process in terms of safe execution. The results also show that the methodology is more effective than conservative MCR approaches, which avoid all possible object poses regardless of the reported uncertainty.}, keywords = {}, pubstate = {published}, tppubtype = {conference} } Picking an item in the presence of other objects can be challenging as it involves occlusions and partial views. Given object models, one approach is to perform object pose estimation and use the most likely candidate pose per object to pick the target without collisions. This approach, however, ignores the uncertainty of the perception process both regarding the target's and the surrounding objects' poses. This work proposes first a perception process for 6D pose estimation, which returns a discrete distribution of object poses in a scene. Then, an open-loop planning pipeline is proposed to return safe and effective solutions for moving a robotic arm to pick, which (a) minimizes the probability of collision with the obstructing objects; and (b) maximizes the probability of reaching the target item. The planning framework models the challenge as a stochastic variant of the Minimum Constraint Removal (MCR) problem. The effectiveness of the methodology is verified given both simulated and real data in different scenarios. The experiments demonstrate the importance of considering the uncertainty of the perception process in terms of safe execution. The results also show that the methodology is more effective than conservative MCR approaches, which avoid all possible object poses regardless of the reported uncertainty. |
Wen, B; Mitash, C; Soorian, S; Kimmel, A; Sintov, A; Bekris, K Robust, Occlusion-Aware Pose Estimation for Objects Grasped by Adaptive Hands Conference IEEE International Conference on Robotics and Automation (ICRA), Paris, France, 2020. @conference{Wen:2020aa, title = {Robust, Occlusion-Aware Pose Estimation for Objects Grasped by Adaptive Hands}, author = {B Wen and C Mitash and S Soorian and A Kimmel and A Sintov and K Bekris}, url = {https://arxiv.org/abs/2003.03518}, year = {2020}, date = {2020-01-01}, booktitle = {IEEE International Conference on Robotics and Automation (ICRA)}, address = {Paris, France}, abstract = {Many manipulation tasks, such as placement or within-hand manipulation, require the object's pose relative to a robot hand. The task is difficult when the hand significantly occludes the object. It is especially hard for adaptive hands, for which it is not easy to detect the finger's configuration. In addition, RGB-only approaches face issues with texture-less objects or when the hand and the object look similar. This paper presents a depth-based framework, which aims for robust pose estimation and short response times. The approach detects the adaptive hand's state via efficient parallel search given the highest overlap between the hand's model and the point cloud. The hand's point cloud is pruned and robust global registration is performed to generate object pose hypotheses, which are clustered. False hypotheses are pruned via physical reasoning. The remaining poses' quality is evaluated given agreement with observed data. Extensive evaluation on synthetic and real data demonstrates the accuracy and computational efficiency of the framework when applied on challenging, highly-occluded scenarios for different object types. An ablation study identifies how the framework's components help in performance. This work also provides a dataset for in-hand 6D object pose esti- mation. Code and dataset are available at: https://github. com/wenbowen123/icra20-hand-object-pose}, keywords = {}, pubstate = {published}, tppubtype = {conference} } Many manipulation tasks, such as placement or within-hand manipulation, require the object's pose relative to a robot hand. The task is difficult when the hand significantly occludes the object. It is especially hard for adaptive hands, for which it is not easy to detect the finger's configuration. In addition, RGB-only approaches face issues with texture-less objects or when the hand and the object look similar. This paper presents a depth-based framework, which aims for robust pose estimation and short response times. The approach detects the adaptive hand's state via efficient parallel search given the highest overlap between the hand's model and the point cloud. The hand's point cloud is pruned and robust global registration is performed to generate object pose hypotheses, which are clustered. False hypotheses are pruned via physical reasoning. The remaining poses' quality is evaluated given agreement with observed data. Extensive evaluation on synthetic and real data demonstrates the accuracy and computational efficiency of the framework when applied on challenging, highly-occluded scenarios for different object types. An ablation study identifies how the framework's components help in performance. This work also provides a dataset for in-hand 6D object pose esti- mation. Code and dataset are available at: https://github. com/wenbowen123/icra20-hand-object-pose |
Shome, R Rutgers University, 2020. @phdthesis{Shome:2020ad, title = {The Problem of Many: Efficient Multi-Arm, Multi-Object Task and and Motion Planning with Optimality Guarantees}, author = {R Shome}, url = {https://doi.org/doi:10.7282/t3-8fcf-xp94}, year = {2020}, date = {2020-01-01}, volume = {PhD}, address = {New Brunswick, NJ}, school = {Rutgers University}, abstract = {This thesis deals with task and motion planning challenges, specifically those involving manipulating multiple objects using multiple robot manipulators. The contributions range from a new foundational understanding of the problem and the conditions for achieving asymptotic optimality to devising application-oriented and efficient planning algorithms as well as experiments on real systems. A key focus corresponds to overcoming scalability challenges in motion planning and dealing with hybrid planning domains, i.e., those that combine continuous and discrete action spaces, to solve manipulation problems that involve multiple types of actions, such as picks, placements and handoffs. The thesis starts with a review of the theoretical foundations regarding the asymptotic optimality properties of sampling-based motion planners. The work outlines core ideas that motivated relevant algorithmic discoveries, as well as the various avenues of research that have followed since. It then presents a new foundational contribution regarding the theoretical conditions for guaranteeing asymptotic optimality in integrated task and motion planning problems. The work addresses the theoretical gap that existed in modeling interactions with the boundaries of the collision-free space, which invariably arise in task planning for manipulation. The second contribution pertains to the design of an efficient, heuristically guided, scalable and asymptotically optimal sampling-based algorithm specifically for solving high-dimensional multi-robot problems. The dRRT* algorithm extends the idea of a tensor roadmap decomposition of the underlying configuration space and uses efficient single-robot heuristics to solve challenging planning problems involving multiple manipulators in a coupled manner. The third area of impact relates to multi-arm task planning problems. Leveraging the efficient multi-arm planning paradigm provided by dRRT*, a multi-modal task planning approach has been developed to deal with pick-handoff-place problems involving up to $7$ robotic arms. A key benefit of integrated task planning enables every arm to preempt the motions that might be necessary for a sequence of actions. Similar task-planning challenges arise when instead of multiple arms, the number of objects increases, which leads to object rearrangement problems. The combinatorial explosion in this case arises from the choices available for assigning objects to arms, and sequencing such actions makes the problem more challenging. In this context, this thesis provides an efficient solution for dual-arm tabletop rearrangement by decomposing the problem into more efficiently solvable subproblems - weighted edge-matching and the traveling salesperson. The above two lines of work have been extended to address more general multi-arm rearrangement problems, dealing with instances involving up to 9 arms and 4 objects. The key insight is a specially constructed mode-graph with capacity constraints, where an efficiently solvable multi-agent path finding solution for the objects can be mapped to a solution to the task planning problem. The consideration of multiple agents in planning problems can extend to human and robotic agents as well. This thesis includes work in human-robot interaction which relate to legibility of manipulator motions, and different types of robotic pointing. It concludes by highlighting applications of the presented planning methods in important domains, such as solving robotic product packing, dual-arm constrained placement and the use of robots in exposure studies.}, keywords = {}, pubstate = {published}, tppubtype = {phdthesis} } This thesis deals with task and motion planning challenges, specifically those involving manipulating multiple objects using multiple robot manipulators. The contributions range from a new foundational understanding of the problem and the conditions for achieving asymptotic optimality to devising application-oriented and efficient planning algorithms as well as experiments on real systems. A key focus corresponds to overcoming scalability challenges in motion planning and dealing with hybrid planning domains, i.e., those that combine continuous and discrete action spaces, to solve manipulation problems that involve multiple types of actions, such as picks, placements and handoffs. The thesis starts with a review of the theoretical foundations regarding the asymptotic optimality properties of sampling-based motion planners. The work outlines core ideas that motivated relevant algorithmic discoveries, as well as the various avenues of research that have followed since. It then presents a new foundational contribution regarding the theoretical conditions for guaranteeing asymptotic optimality in integrated task and motion planning problems. The work addresses the theoretical gap that existed in modeling interactions with the boundaries of the collision-free space, which invariably arise in task planning for manipulation. The second contribution pertains to the design of an efficient, heuristically guided, scalable and asymptotically optimal sampling-based algorithm specifically for solving high-dimensional multi-robot problems. The dRRT* algorithm extends the idea of a tensor roadmap decomposition of the underlying configuration space and uses efficient single-robot heuristics to solve challenging planning problems involving multiple manipulators in a coupled manner. The third area of impact relates to multi-arm task planning problems. Leveraging the efficient multi-arm planning paradigm provided by dRRT*, a multi-modal task planning approach has been developed to deal with pick-handoff-place problems involving up to $7$ robotic arms. A key benefit of integrated task planning enables every arm to preempt the motions that might be necessary for a sequence of actions. Similar task-planning challenges arise when instead of multiple arms, the number of objects increases, which leads to object rearrangement problems. The combinatorial explosion in this case arises from the choices available for assigning objects to arms, and sequencing such actions makes the problem more challenging. In this context, this thesis provides an efficient solution for dual-arm tabletop rearrangement by decomposing the problem into more efficiently solvable subproblems - weighted edge-matching and the traveling salesperson. The above two lines of work have been extended to address more general multi-arm rearrangement problems, dealing with instances involving up to 9 arms and 4 objects. The key insight is a specially constructed mode-graph with capacity constraints, where an efficiently solvable multi-agent path finding solution for the objects can be mapped to a solution to the task planning problem. The consideration of multiple agents in planning problems can extend to human and robotic agents as well. This thesis includes work in human-robot interaction which relate to legibility of manipulator motions, and different types of robotic pointing. It concludes by highlighting applications of the presented planning methods in important domains, such as solving robotic product packing, dual-arm constrained placement and the use of robots in exposure studies. |
2019 |
Kimmel, A; Shome, R; Bekris, K Anytime Motion Planning for Prehensile Manipulation in Dense Clutter Journal Article Advanced Robotics, 2019. @article{Kimmel:2019ab, title = {Anytime Motion Planning for Prehensile Manipulation in Dense Clutter}, author = {A Kimmel and R Shome and K Bekris}, url = {https://www.rahulsho.me/papers/ar_gmp.pdf}, year = {2019}, date = {2019-11-01}, journal = {Advanced Robotics}, abstract = {Many methods have been developed for planning the motion of robotic arms for picking and placing, ranging from local optimization to global search techniques, which are effective for sparsely placed objects. Dense clutter, however, still adversely affects the success rate, computation times, and quality of solutions in many real-world setups. The proposed method achieves high success ratio in clutter with anytime performance by returning solutions quickly and improving their quality over time. The method first explores the lower dimensional end effector's task space efficiently by ignoring the arm, and build a discrete approximation of a navigation function. This is performed online, without prior knowledge of the scene. Then, an informed sampling-based planner for the entire arm uses Jacobian- based steering to reach promising end effector poses given the task space guidance. This process is also comprehensive and allows the exploration of alternative paths over time if the task space guidance is misleading. This paper evaluates the proposed method against alternatives in picking or placing tasks among varying amounts of clutter for a variety of robotic manipulators with different end-effectors. The results suggest that the method reliably provides higher quality solution paths quicker, with a higher success rate relative to alternatives.}, keywords = {}, pubstate = {published}, tppubtype = {article} } Many methods have been developed for planning the motion of robotic arms for picking and placing, ranging from local optimization to global search techniques, which are effective for sparsely placed objects. Dense clutter, however, still adversely affects the success rate, computation times, and quality of solutions in many real-world setups. The proposed method achieves high success ratio in clutter with anytime performance by returning solutions quickly and improving their quality over time. The method first explores the lower dimensional end effector's task space efficiently by ignoring the arm, and build a discrete approximation of a navigation function. This is performed online, without prior knowledge of the scene. Then, an informed sampling-based planner for the entire arm uses Jacobian- based steering to reach promising end effector poses given the task space guidance. This process is also comprehensive and allows the exploration of alternative paths over time if the task space guidance is misleading. This paper evaluates the proposed method against alternatives in picking or placing tasks among varying amounts of clutter for a variety of robotic manipulators with different end-effectors. The results suggest that the method reliably provides higher quality solution paths quicker, with a higher success rate relative to alternatives. |
Kimmel, A; Sintov, A; Tan, J; Wen, B; Boularias, A; Bekris, K Belief-Space Planning Using Learned Models with Application to Underactuated Hands Conference International Symposium on Robotics Research (ISRR), Hanoi, Vietnam, 2019. @conference{Kimmel:2019aa, title = {Belief-Space Planning Using Learned Models with Application to Underactuated Hands}, author = {A Kimmel and A Sintov and J Tan and B Wen and A Boularias and K Bekris}, url = {http://www.cs.rutgers.edu/~kb572/pubs/belief_space_learned_models_adaptive_hands.pdf}, year = {2019}, date = {2019-10-01}, booktitle = {International Symposium on Robotics Research (ISRR)}, address = {Hanoi, Vietnam}, abstract = {Acquiring a precise model is a challenging task for many important robotic tasks and systems - including in-hand manipulation using underactuated, adaptive hands. Learning stochastic, data-driven models is a promising alternative as they provide not only a way to propagate forward the system dynamics, but also express the uncertainty present in the collected data. Therefore, such models en- able planning in the space of state distributions, i.e., in the belief space. This paper proposes a planning framework that employs stochastic, learned models, which ex- press a distribution of states as a set of particles. The integration achieves anytime behavior in terms of returning paths of increasing quality under constraints for the probability of success to achieve a goal. The focus of this effort is on pushing the efficiency of the overall methodology despite the notorious computational hardness of belief-space planning. Experiments show that the proposed framework enables reaching a desired goal with higher success rate compared to alternatives in sim- ple benchmarks. This work also provides an application to the motivating domain of in-hand manipulation with underactuated, adaptive hands, both in the case of physically-simulated experiments as well as demonstrations with a real hand.}, keywords = {}, pubstate = {published}, tppubtype = {conference} } Acquiring a precise model is a challenging task for many important robotic tasks and systems - including in-hand manipulation using underactuated, adaptive hands. Learning stochastic, data-driven models is a promising alternative as they provide not only a way to propagate forward the system dynamics, but also express the uncertainty present in the collected data. Therefore, such models en- able planning in the space of state distributions, i.e., in the belief space. This paper proposes a planning framework that employs stochastic, learned models, which ex- press a distribution of states as a set of particles. The integration achieves anytime behavior in terms of returning paths of increasing quality under constraints for the probability of success to achieve a goal. The focus of this effort is on pushing the efficiency of the overall methodology despite the notorious computational hardness of belief-space planning. Experiments show that the proposed framework enables reaching a desired goal with higher success rate compared to alternatives in sim- ple benchmarks. This work also provides an application to the motivating domain of in-hand manipulation with underactuated, adaptive hands, both in the case of physically-simulated experiments as well as demonstrations with a real hand. |
Shome, R; Bekris, K Anytime Multi-Arm Task and Motion Planning for Pick-and-Place of Individual Objects Via Handoffs Conference IEEE International Conference on Multi-Robot and Multi-Agent Systems (MRS), New Brunswick, NJ, 2019. @conference{Shome:2019aa, title = {Anytime Multi-Arm Task and Motion Planning for Pick-and-Place of Individual Objects Via Handoffs}, author = {R Shome and K Bekris}, url = {https://arxiv.org/abs/1905.03179}, year = {2019}, date = {2019-06-01}, booktitle = {IEEE International Conference on Multi-Robot and Multi-Agent Systems (MRS)}, address = {New Brunswick, NJ}, abstract = {Automation applications are pushing the deployment of many high DoF manipulators in warehouse and manufacturing environments. This has motivated many efforts on optimizing manipulation tasks involving a single arm. Coordinating multiple arms for manipulation, however, introduces additional computational challenges arising from the increased DoFs, as well as the combinatorial increase in the available operations that many manipulators can perform, including handoffs between arms. The focus here is on the case of pick-and-place tasks, which require a sequence of handoffs to be executed, so as to achieve computational efficiency, asymptotic optimality and practical anytime performance. The paper leverages recent advances in multi-robot motion planning for high DoF systems to propose a novel multi-modal extension of the dRRT* algorithm. The key insight is that, instead of naively solving a sequence of motion planning problems, it is computationally advantageous to directly explore the composite space of the integrated multi-arm task and motion planning problem, given input sets of possible pick and handoff configurations. Asymptotic optimality guarantees are possible by sampling additional picks and handoffs over time. The evaluation shows that the approach finds initial solutions fast and improves their quality over time. It also succeeds in finding solutions to harder problem instances relative to alternatives and can scale effectively as the number of robots increases.}, keywords = {}, pubstate = {published}, tppubtype = {conference} } Automation applications are pushing the deployment of many high DoF manipulators in warehouse and manufacturing environments. This has motivated many efforts on optimizing manipulation tasks involving a single arm. Coordinating multiple arms for manipulation, however, introduces additional computational challenges arising from the increased DoFs, as well as the combinatorial increase in the available operations that many manipulators can perform, including handoffs between arms. The focus here is on the case of pick-and-place tasks, which require a sequence of handoffs to be executed, so as to achieve computational efficiency, asymptotic optimality and practical anytime performance. The paper leverages recent advances in multi-robot motion planning for high DoF systems to propose a novel multi-modal extension of the dRRT* algorithm. The key insight is that, instead of naively solving a sequence of motion planning problems, it is computationally advantageous to directly explore the composite space of the integrated multi-arm task and motion planning problem, given input sets of possible pick and handoff configurations. Asymptotic optimality guarantees are possible by sampling additional picks and handoffs over time. The evaluation shows that the approach finds initial solutions fast and improves their quality over time. It also succeeds in finding solutions to harder problem instances relative to alternatives and can scale effectively as the number of robots increases. |
Shome, R; Tang, W; Song, C; Mitash, C; Kourtev, C; Yu, J; Boularias, A; Bekris, K Towards Robust Product Packing with a Minimalistic End-Effector Conference IEEE International Conference on Robotics and Automation (ICRA), 2019, (Nomination for Best Paper Award in Automation). @conference{Shome:2019ab, title = {Towards Robust Product Packing with a Minimalistic End-Effector}, author = {R Shome and W Tang and C Song and C Mitash and C Kourtev and J Yu and A Boularias and K Bekris}, url = {http://rl.cs.rutgers.edu/publications/ICRA-2019-Packing.pdf}, year = {2019}, date = {2019-05-01}, booktitle = {IEEE International Conference on Robotics and Automation (ICRA)}, abstract = {Advances in sensor technologies, object detection algorithms, planning frameworks and hardware designs have motivated the deployment of robots in warehouse automation. A variety of such applications, like order fulfillment or packing tasks, require picking objects from unstructured piles and carefully arranging them in bins or containers. Desirable solutions need to be low-cost, easily deployable and controllable, making minimalistic hardware choices desirable. The challenge in designing an effective solution to this problem relates to appropriately integrating multiple components, so as to achieve a robust pipeline that minimizes failure conditions. The current work proposes a complete pipeline for solving such packing tasks, given access only to RGB-D data and a single robot arm with a minimalistic, vacuum-based end-effector. To achieve the desired level of robustness, three key manipulation primitives are identified, which take advantage of the environment and simple operations to successfully pack multiple cubic objects. The overall approach is demonstrated to be robust to execution and perception errors. The impact of each manipulation primitive is evaluated by considering different versions of the proposed pipeline that incrementally introduce reasoning about object poses and corrective manipulation actions.}, note = {Nomination for Best Paper Award in Automation}, keywords = {}, pubstate = {published}, tppubtype = {conference} } Advances in sensor technologies, object detection algorithms, planning frameworks and hardware designs have motivated the deployment of robots in warehouse automation. A variety of such applications, like order fulfillment or packing tasks, require picking objects from unstructured piles and carefully arranging them in bins or containers. Desirable solutions need to be low-cost, easily deployable and controllable, making minimalistic hardware choices desirable. The challenge in designing an effective solution to this problem relates to appropriately integrating multiple components, so as to achieve a robust pipeline that minimizes failure conditions. The current work proposes a complete pipeline for solving such packing tasks, given access only to RGB-D data and a single robot arm with a minimalistic, vacuum-based end-effector. To achieve the desired level of robustness, three key manipulation primitives are identified, which take advantage of the environment and simple operations to successfully pack multiple cubic objects. The overall approach is demonstrated to be robust to execution and perception errors. The impact of each manipulation primitive is evaluated by considering different versions of the proposed pipeline that incrementally introduce reasoning about object poses and corrective manipulation actions. |
Sintov, A; Morgan, A; Kimmel, A; Dollar, A; Bekris, K; Boularias, A Learning a State Transition Model of an Underactuated Adaptive Hand Journal Article IEEE Robotics and Automation Letters (RA-L) (also appearing at IEEE ICRA 2019), 2019. @article{Sintov:2019aa, title = {Learning a State Transition Model of an Underactuated Adaptive Hand}, author = {A Sintov and A Morgan and A Kimmel and A Dollar and K Bekris and A Boularias}, url = {http://www.cs.rutgers.edu/~kb572/pubs/Learning_a_State_Transition_Model.pdf}, year = {2019}, date = {2019-05-01}, journal = {IEEE Robotics and Automation Letters (RA-L) (also appearing at IEEE ICRA 2019)}, abstract = {Fully-actuated, multi-fingered robotic hands are often expensive and fragile. Low-cost, under-actuated hands are appealing but present challenges due to the lack of analytical models. This paper aims to learn a stochastic version of such models automatically from data with minimum user effort. The focus is on identifying the dominant, sensible features required to express hand state transitions given quasi-static motions, thereby enabling the learning of a probabilistic transition model from recorded trajectories. Experiments both with Gaussian Processes (GP) and Neural Network models are included for analysis and evaluation. The metric for local GP regression is obtained with a manifold learning approach, known as "Diffusion Maps", to uncover the lower-dimensional subspace in which the data lies and provide a geodesic metric. Results show that using Diffusion Maps with a feature space composed of the object position, actuator angles, and actuator loads, sufficiently expresses the hand-object system "configuration and can provide accurate enough predictions for a relatively long horizon. To the best of the authors knowledge, this is the first learned transition model for such underactuated hands that achieves this level of predictability. Notably, the same feature space implicitly embeds the size of the manipulated object and can generalize to new objects of varying sizes. Furthermore, the learned model can identify states that are on the verge of failure and which should be avoided during manipulation. The usefulness of the model is also demonstrated by integrating it with closed-loop control to successfully and safely complete manipulation tasks.}, keywords = {}, pubstate = {published}, tppubtype = {article} } Fully-actuated, multi-fingered robotic hands are often expensive and fragile. Low-cost, under-actuated hands are appealing but present challenges due to the lack of analytical models. This paper aims to learn a stochastic version of such models automatically from data with minimum user effort. The focus is on identifying the dominant, sensible features required to express hand state transitions given quasi-static motions, thereby enabling the learning of a probabilistic transition model from recorded trajectories. Experiments both with Gaussian Processes (GP) and Neural Network models are included for analysis and evaluation. The metric for local GP regression is obtained with a manifold learning approach, known as "Diffusion Maps", to uncover the lower-dimensional subspace in which the data lies and provide a geodesic metric. Results show that using Diffusion Maps with a feature space composed of the object position, actuator angles, and actuator loads, sufficiently expresses the hand-object system "configuration and can provide accurate enough predictions for a relatively long horizon. To the best of the authors knowledge, this is the first learned transition model for such underactuated hands that achieves this level of predictability. Notably, the same feature space implicitly embeds the size of the manipulated object and can generalize to new objects of varying sizes. Furthermore, the learned model can identify states that are on the verge of failure and which should be avoided during manipulation. The usefulness of the model is also demonstrated by integrating it with closed-loop control to successfully and safely complete manipulation tasks. |
2018 |
Shome, R; Solovey, K; Yu, J; Bekris, K; Halperin, D Fast and High-Quality Dual-Arm Rearrangement in Synchronous, Monotone Tabletop Setups Conference Workshop on the Algorithmic Foundations of Robotics (WAFR), Mérida, México, 2018. @conference{Shome:2018aa, title = {Fast and High-Quality Dual-Arm Rearrangement in Synchronous, Monotone Tabletop Setups}, author = {R Shome and K Solovey and J Yu and K Bekris and D Halperin}, url = {http://www.cs.rutgers.edu/~kb572/pubs/Fast_High_Quality_Dual_Arm_Rearrangement.pdf}, year = {2018}, date = {2018-12-01}, booktitle = {Workshop on the Algorithmic Foundations of Robotics (WAFR)}, address = {Mérida, México}, abstract = {Rearranging objects on a planar surface arises in a variety of applications, such as packaging. Using two arms can improve efficiency but introduces new combinatorial challenges. This paper studies the structure of dual-arm rearrangement for synchronous, monotone tabletop setups and develops an optimal MILP model. It then describes an efficient and scalable algorithm, which first minimizes the cost of object transfers and then of transitions between objects. This is motivated by the fact that asymptotically object transfers dominate the cost of solutions. Moreover, a lazy strategy minimizes the number of motion planning calls and results in significant speedups. Theoretical arguments support the benefits of using two arms and indicate that synchronous operation introduces only a small cost increase. Experiments support these points and show that the scalable method can quickly compute solutions close to optimal for the considered setup.}, keywords = {}, pubstate = {published}, tppubtype = {conference} } Rearranging objects on a planar surface arises in a variety of applications, such as packaging. Using two arms can improve efficiency but introduces new combinatorial challenges. This paper studies the structure of dual-arm rearrangement for synchronous, monotone tabletop setups and develops an optimal MILP model. It then describes an efficient and scalable algorithm, which first minimizes the cost of object transfers and then of transitions between objects. This is motivated by the fact that asymptotically object transfers dominate the cost of solutions. Moreover, a lazy strategy minimizes the number of motion planning calls and results in significant speedups. Theoretical arguments support the benefits of using two arms and indicate that synchronous operation introduces only a small cost increase. Experiments support these points and show that the scalable method can quickly compute solutions close to optimal for the considered setup. |
Calli, B; Kimmel, A; Hang, K; Bekris, K; Dollar, A Path Planning for Within-Hand Manipulation Over Learned Representations of Safe States Conference International Symposium on Experimental Robotics (ISER), Buenos Aires, Argentina, 2018. @conference{Calli:2018aa, title = {Path Planning for Within-Hand Manipulation Over Learned Representations of Safe States}, author = {B Calli and A Kimmel and K Hang and K Bekris and A Dollar}, url = {http://www.cs.rutgers.edu/~kb572/pubs/within_hand_planning_over_learning.pdf}, year = {2018}, date = {2018-11-01}, booktitle = {International Symposium on Experimental Robotics (ISER)}, address = {Buenos Aires, Argentina}, abstract = {This work proposes a framework for tracking a desired path of an object held by an adaptive hand via within-hand manipulation. Such underactuated hands are able to passively achieve stable contacts with objects. Combined with vision-based control and data-driven state estimation process, they can solve tasks without accurate hand-object models or multi-modal sensory feedback. In particular, a data-driven regression process is used here to estimate the probability of dropping the object for given manipulation states. Then, an optimization-based planner aims to track the desired path while avoiding states that are above a threshold probability of dropping the object. The optimized cost function, based on the principle of Dynamic-Time Warping (DTW), seeks to minimize the area between the desired and the followed path. By adapting the threshold for the probability of dropping the object, the framework can handle objects of different weights without retraining. Experiments involving writing letters with a marker, as well as tracing randomized paths, were conducted on the Yale Model T-42 hand. Results indicate that the framework successfully avoids undesirable states, while minimizing the proposed cost function, thereby producing object paths for within-hand manipulation that closely match the target ones.}, keywords = {}, pubstate = {published}, tppubtype = {conference} } This work proposes a framework for tracking a desired path of an object held by an adaptive hand via within-hand manipulation. Such underactuated hands are able to passively achieve stable contacts with objects. Combined with vision-based control and data-driven state estimation process, they can solve tasks without accurate hand-object models or multi-modal sensory feedback. In particular, a data-driven regression process is used here to estimate the probability of dropping the object for given manipulation states. Then, an optimization-based planner aims to track the desired path while avoiding states that are above a threshold probability of dropping the object. The optimized cost function, based on the principle of Dynamic-Time Warping (DTW), seeks to minimize the area between the desired and the followed path. By adapting the threshold for the probability of dropping the object, the framework can handle objects of different weights without retraining. Experiments involving writing letters with a marker, as well as tracing randomized paths, were conducted on the Yale Model T-42 hand. Results indicate that the framework successfully avoids undesirable states, while minimizing the proposed cost function, thereby producing object paths for within-hand manipulation that closely match the target ones. |
Han, S; Stiffler, N; Bekris, K; Yu, J Efficient, High-Quality Stack Rearrangement Journal Article IEEE Robotics and Automation Letters (RA-L) [Also accepted to appear at the 2018 IEEE International Conference on Robotics and Automation (ICRA)], 3 , pp. 1608–1615, 2018. @article{187, title = {Efficient, High-Quality Stack Rearrangement}, author = {S Han and N Stiffler and K Bekris and J Yu}, url = {https://www.cs.rutgers.edu/~kb572/pubs/stack_rearrangement.pdf}, year = {2018}, date = {2018-07-01}, journal = {IEEE Robotics and Automation Letters (RA-L) [Also accepted to appear at the 2018 IEEE International Conference on Robotics and Automation (ICRA)]}, volume = {3}, pages = {1608--1615}, abstract = {This work studies rearrangement problems involving the sorting of robots or objects in stack-like containers, which can be accessed only from one side. Two scenarios are considered: one where every robot or object needs to reach a particular stack, and a setting in which each robot has a distinct position within a stack. In both cases, the goal is to minimize the number of stack removals that need to be performed. Stack rearrangement is shown to be intimately connected to pebble motion problems, a useful abstraction in multi-robot path planning. Through this connection, feasibility of stack rearrangement can be readily addressed. The paper continues to establish lower and upper bounds on optimality, which differ only by a logarithmic factor, in terms of stack removals. An algorithmic solution is then developed that produces suboptimal paths much quicker than a pebble motion solver. Furthermore, informed search-based methods are proposed for finding high-quality solutions. The efficiency and desirable scalability of the methods is demonstrated in simulation.}, keywords = {}, pubstate = {published}, tppubtype = {article} } This work studies rearrangement problems involving the sorting of robots or objects in stack-like containers, which can be accessed only from one side. Two scenarios are considered: one where every robot or object needs to reach a particular stack, and a setting in which each robot has a distinct position within a stack. In both cases, the goal is to minimize the number of stack removals that need to be performed. Stack rearrangement is shown to be intimately connected to pebble motion problems, a useful abstraction in multi-robot path planning. Through this connection, feasibility of stack rearrangement can be readily addressed. The paper continues to establish lower and upper bounds on optimality, which differ only by a logarithmic factor, in terms of stack removals. An algorithmic solution is then developed that produces suboptimal paths much quicker than a pebble motion solver. Furthermore, informed search-based methods are proposed for finding high-quality solutions. The efficiency and desirable scalability of the methods is demonstrated in simulation. |
Han, S; Stiffler, N; Krontiris, A; Bekris, K; Yu, J Complexity Results and Fast Methods for Optimal Tabletop Rearrangement with Overhand Grasps Journal Article International Journal of Robotics Research (IJRR), 2018. @article{Shuai:2018aa, title = {Complexity Results and Fast Methods for Optimal Tabletop Rearrangement with Overhand Grasps}, author = {S Han and N Stiffler and A Krontiris and K Bekris and J Yu}, url = {https://www.cs.rutgers.edu/~kb572/pubs/optimal_tabletop_rearrangement.pdf}, year = {2018}, date = {2018-07-01}, journal = {International Journal of Robotics Research (IJRR)}, abstract = {This paper studies the underlying combinatorial structure of a class of object rearrangement problems, which appear frequently in applications. The problems involve multiple, similar-geometry objects placed on a flat, horizontal surface, where a robot can approach them from above and perform pick-and-place operations to rearrange them. The paper considers both the case where the start and goal object poses overlap, and where they do not. For overlapping poses, the primary objective is to minimize the number of pick-and-place actions and then to minimize the distance traveled by the end-effector. For the non-overlapping case, the objective is solely to minimize the travel distance of the end-effector. While such problems do not involve all the complexities of general rearrangement, they remain computationally hard in both cases. This is shown through reductions from well-understood, hard combinatorial challenges to these rearrangement problems. The reductions are also shown to hold in the reverse direction, which enables the convenient application on rearrangement of well studied algorithms. These algorithms can be very efficient in practice despite the hardness results. The paper builds on these reduction results to propose an algorithmic pipeline for dealing with the rearrangement problems. Experimental evaluation, including hardware-based trials, shows that the proposed pipeline computes high-quality paths with regards to the optimization objectives. Furthermore, it exhibits highly desirable scalability as the number of objects increases in both the overlapping and non-overlapping setup.}, keywords = {}, pubstate = {published}, tppubtype = {article} } This paper studies the underlying combinatorial structure of a class of object rearrangement problems, which appear frequently in applications. The problems involve multiple, similar-geometry objects placed on a flat, horizontal surface, where a robot can approach them from above and perform pick-and-place operations to rearrange them. The paper considers both the case where the start and goal object poses overlap, and where they do not. For overlapping poses, the primary objective is to minimize the number of pick-and-place actions and then to minimize the distance traveled by the end-effector. For the non-overlapping case, the objective is solely to minimize the travel distance of the end-effector. While such problems do not involve all the complexities of general rearrangement, they remain computationally hard in both cases. This is shown through reductions from well-understood, hard combinatorial challenges to these rearrangement problems. The reductions are also shown to hold in the reverse direction, which enables the convenient application on rearrangement of well studied algorithms. These algorithms can be very efficient in practice despite the hardness results. The paper builds on these reduction results to propose an algorithmic pipeline for dealing with the rearrangement problems. Experimental evaluation, including hardware-based trials, shows that the proposed pipeline computes high-quality paths with regards to the optimization objectives. Furthermore, it exhibits highly desirable scalability as the number of objects increases in both the overlapping and non-overlapping setup. |
2017 |
Dobson, A; Solovey, K; Shome, R; Halperin, D; Bekris, K Scalable Asymptotically-Optimal Multi-Robot Motion Planning Conference 1st IEEE International Symposium on Multi-Robot and Multi-Agent Systems (MRS), [Best Paper Award] [Best Paper Award], Los Angeles, CA, USA, 2017. @conference{Dobson:2017aa, title = {Scalable Asymptotically-Optimal Multi-Robot Motion Planning}, author = {A Dobson and K Solovey and R Shome and D Halperin and K Bekris}, url = {https://arxiv.org/abs/1706.09932}, year = {2017}, date = {2017-12-01}, booktitle = {1st IEEE International Symposium on Multi-Robot and Multi-Agent Systems (MRS)}, publisher = {[Best Paper Award]}, address = {Los Angeles, CA, USA}, organization = {[Best Paper Award]}, abstract = {Discovering high-quality paths for multi-robot problems can be achieved, in principle, through asymptotically-optimal data structures in the composite space of all robots, such as a sampling-based roadmap or a tree. The hardness of motion planning, however, which depends exponentially on the number of robots, renders the explicit construction of such structures impractical. This work proposes a scalable, sampling-based planner for coupled multi-robot problems that provides desirable path-quality guarantees. The proposed dRRT* is an informed, asymptotically-optimal extension of a prior method dRRT, which introduced the idea of building roadmaps for each robot and implicitly searching the tensor product of these structures in the composite space. The paper describes the conditions for convergence to optimal paths in multi-robot problems. Moreover, simulated experiments indicate dRRT* converges to high-quality paths and scales to higher numbers of robots where various alternatives fail. It can also be used on high-dimensional challenges, such as planning for robot manipulators.}, keywords = {}, pubstate = {published}, tppubtype = {conference} } Discovering high-quality paths for multi-robot problems can be achieved, in principle, through asymptotically-optimal data structures in the composite space of all robots, such as a sampling-based roadmap or a tree. The hardness of motion planning, however, which depends exponentially on the number of robots, renders the explicit construction of such structures impractical. This work proposes a scalable, sampling-based planner for coupled multi-robot problems that provides desirable path-quality guarantees. The proposed dRRT* is an informed, asymptotically-optimal extension of a prior method dRRT, which introduced the idea of building roadmaps for each robot and implicitly searching the tensor product of these structures in the composite space. The paper describes the conditions for convergence to optimal paths in multi-robot problems. Moreover, simulated experiments indicate dRRT* converges to high-quality paths and scales to higher numbers of robots where various alternatives fail. It can also be used on high-dimensional challenges, such as planning for robot manipulators. |
Shome, R; Bekris, K IEEE International Conference on Humanoid Robots, Birmingham, UK, 2017. @conference{Shome:2017aa, title = {Improving the Scalability of Asymptotically Optimal Motion Planning for Humanoid Dual-Arm Manipulators}, author = {R Shome and K Bekris}, url = {https://www.cs.rutgers.edu/~kb572/pubs/asymp_optimal_dual_arm.pdf}, year = {2017}, date = {2017-11-01}, booktitle = {IEEE International Conference on Humanoid Robots}, address = {Birmingham, UK}, abstract = {Due to high-dimensionality, many motion planners for dual-arm systems follow a decoupled approach but do not provide guarantees. Asymptotically optimal sampling-based planners provide guarantees, but in practice face computational scalability challenges. This work improves the computational scalability of the latter methods in this domain. It builds on top of recent advances in multi-robot motion planning, which provide guarantees without having to explicitly construct a roadmap in the composite space of all robots. The proposed framework builds roadmaps for components of a humanoid robottextquoterights kinematic chain. Then, the tensor product of these component roadmaps is searched implicitly online in a way that asymptotic optimality is provided. Appropriate heuristics from the component roadmaps are utilized for discovering the solution in the composite space effectively. Evaluation on various dual-arm problems show that the method returns paths of increasing quality, has significantly reduced space requirements and improved convergence rate relative to the standard asymptotically optimal approaches.}, keywords = {}, pubstate = {published}, tppubtype = {conference} } Due to high-dimensionality, many motion planners for dual-arm systems follow a decoupled approach but do not provide guarantees. Asymptotically optimal sampling-based planners provide guarantees, but in practice face computational scalability challenges. This work improves the computational scalability of the latter methods in this domain. It builds on top of recent advances in multi-robot motion planning, which provide guarantees without having to explicitly construct a roadmap in the composite space of all robots. The proposed framework builds roadmaps for components of a humanoid robottextquoterights kinematic chain. Then, the tensor product of these component roadmaps is searched implicitly online in a way that asymptotic optimality is provided. Appropriate heuristics from the component roadmaps are utilized for discovering the solution in the composite space effectively. Evaluation on various dual-arm problems show that the method returns paths of increasing quality, has significantly reduced space requirements and improved convergence rate relative to the standard asymptotically optimal approaches. |
Krontiris, A; Bekris, K Tradeoffs in the Computation of Minimum Constraint Removal Paths for Manipulation Planning Journal Article Advanced Robotics Journal, 31 , pp. 1313–1324, 2017. @article{Krontiris:2017aa, title = {Tradeoffs in the Computation of Minimum Constraint Removal Paths for Manipulation Planning}, author = {A Krontiris and K Bekris}, url = {https://www.cs.rutgers.edu/~kb572/pubs/min_constraint_removal.pdf}, year = {2017}, date = {2017-09-01}, journal = {Advanced Robotics Journal}, volume = {31}, pages = {1313--1324}, abstract = {The typical objective in path planning is to find the shortest feasible path. Many times, however, such paths may not be available given constraints, such as movable obstacles. This frequently happens in manipulation planning, where it may be desirable to identify the minimum set of movable obstacles to be cleared to manipulate a target object. This is a similar objective to that of the Minimum Constraint Removal problem, which, however, does not exhibit dynamic programming properties, i.e., subsets of optimum solutions are not necessarily optimal. Thus, searching for MCR paths is computationally expensive. Motivated by this challenge and related work, this paper investigates approximations for computing MCR paths in the context of manipulation planning. The proposed framework searches for MCR paths up to a certain length of solution in terms of end-effector distance. This length can be defined as a multiple of the shortest path length in the space when movable objects are ignored. Given experimental evaluation on simulated manipulation planning challenges, the bounded-length approximation provides a desirable tradeoff between minimizing constraints, computational cost and path length.}, keywords = {}, pubstate = {published}, tppubtype = {article} } The typical objective in path planning is to find the shortest feasible path. Many times, however, such paths may not be available given constraints, such as movable obstacles. This frequently happens in manipulation planning, where it may be desirable to identify the minimum set of movable obstacles to be cleared to manipulate a target object. This is a similar objective to that of the Minimum Constraint Removal problem, which, however, does not exhibit dynamic programming properties, i.e., subsets of optimum solutions are not necessarily optimal. Thus, searching for MCR paths is computationally expensive. Motivated by this challenge and related work, this paper investigates approximations for computing MCR paths in the context of manipulation planning. The proposed framework searches for MCR paths up to a certain length of solution in terms of end-effector distance. This length can be defined as a multiple of the shortest path length in the space when movable objects are ignored. Given experimental evaluation on simulated manipulation planning challenges, the bounded-length approximation provides a desirable tradeoff between minimizing constraints, computational cost and path length. |
Han, S; Stiffler, N; Krontiris, A; Bekris, K; Yu, J High-Quality Tabletop Rearrangement with Overhand Grasps: Hardness Results and Fast Methods Conference Robotics: Science and Systems (RSS), [Best Student Paper Award Finalist] [Best Student Paper Award Finalist], Cambridge, MA, 2017. @conference{172, title = {High-Quality Tabletop Rearrangement with Overhand Grasps: Hardness Results and Fast Methods}, author = {S Han and N Stiffler and A Krontiris and K Bekris and J Yu}, url = {https://www.roboticsproceedings.org/rss13/p51.pdf}, year = {2017}, date = {2017-07-01}, booktitle = {Robotics: Science and Systems (RSS)}, publisher = {[Best Student Paper Award Finalist]}, address = {Cambridge, MA}, organization = {[Best Student Paper Award Finalist]}, abstract = {This paper studies the underlying combinatorial structure of a class of object rearrangement problems that appear frequently in applications. This class considers multiple, similar-geometry objects placed on a flat, horizontal surface, where a robot can approach them with overhand grasps and perform pick-and-place operations to rearrange them. The paper considers both the case where the start and goal object poses overlap, and where they do not. For overlapping poses, the primary objective is to minimize the number of pick-and-place actions and then to minimize the distance traveled by the end-effector. For the non-overlapping case, the objective is solely to minimize the end-effector distance. While this class of problems does not involve all the complexities of general rearrangement, it remains a computationally hard challenge for both cases. This is shown by reducing well understood combinatorial challenges that are hard to these rearrangement problems. The benefit of this reduction is that there are well studied algorithms for solving the combinatorial challenges. These algorithms can be very efficient in practice despite the hardness results. The paper builds on top of these reduction results to propose an algorithmic pipeline for dealing with the rearrangement problem. Experimental evaluation shows that the proposed pipeline achieves high-quality paths in terms of the optimization objective(s) and exhibits highly desirable scalability as the number of objects increases in both overlapping and non-overlapping setups.}, keywords = {}, pubstate = {published}, tppubtype = {conference} } This paper studies the underlying combinatorial structure of a class of object rearrangement problems that appear frequently in applications. This class considers multiple, similar-geometry objects placed on a flat, horizontal surface, where a robot can approach them with overhand grasps and perform pick-and-place operations to rearrange them. The paper considers both the case where the start and goal object poses overlap, and where they do not. For overlapping poses, the primary objective is to minimize the number of pick-and-place actions and then to minimize the distance traveled by the end-effector. For the non-overlapping case, the objective is solely to minimize the end-effector distance. While this class of problems does not involve all the complexities of general rearrangement, it remains a computationally hard challenge for both cases. This is shown by reducing well understood combinatorial challenges that are hard to these rearrangement problems. The benefit of this reduction is that there are well studied algorithms for solving the combinatorial challenges. These algorithms can be very efficient in practice despite the hardness results. The paper builds on top of these reduction results to propose an algorithmic pipeline for dealing with the rearrangement problem. Experimental evaluation shows that the proposed pipeline achieves high-quality paths in terms of the optimization objective(s) and exhibits highly desirable scalability as the number of objects increases in both overlapping and non-overlapping setups. |
2016 |
Littlefield, Z; Zhu, S; Kourtev, C; Psarakis, Z; Shome, R; Kimmel, A; Dobson, A; Souza, F; Bekris, K 12th IEEE International Conference on Automation Science and Engineering (IEEE CASE), Fort Worth, TX, 2016. @conference{Littlefield:2016aa, title = {Evaluating End-Effector Modalities for Warehouse Picking: A Vacuum Gripper Vs a 3-Finger Underactuated Hand}, author = {Z Littlefield and S Zhu and C Kourtev and Z Psarakis and R Shome and A Kimmel and A Dobson and F Souza and K Bekris}, url = {https://www.cs.rutgers.edu/~kb572/pubs/apc_grasping_evaluation.pdf}, year = {2016}, date = {2016-08-01}, booktitle = {12th IEEE International Conference on Automation Science and Engineering (IEEE CASE)}, address = {Fort Worth, TX}, abstract = {This paper evaluates two end-effector modalities in the context of warehouse picking tasks, where a robot has to grasp objects inside shelves. The two end-effectors correspond to (i) a recently developed, underactuated three-finger hand and (ii) a custom built, vacuum-based gripper. The two systems significantly differ on how they need to be placed relative to an object so that a successful grasp occurs. The first tool provides increased flexibility, while the vacuum alternative is simpler and has smaller form. The objective is to highlight how the end-effector choice can significantly influence the success rate of robotic picking as well as the speed of the overall solution. For the evaluation, the same grasping planning process is followed with both end-effectors given knowledge of an objectstextquoteright pose. Multiple objects with different geometries and characteristics are placed in various poses for testing purposes. The resulting trajectories are executed on a real system to evaluate the effectiveness of the corresponding end-effector modalities in practice. The results indicate that, under different conditions, different types of end-effectors can be beneficial, which motivates the development of hybrid solutions.}, keywords = {}, pubstate = {published}, tppubtype = {conference} } This paper evaluates two end-effector modalities in the context of warehouse picking tasks, where a robot has to grasp objects inside shelves. The two end-effectors correspond to (i) a recently developed, underactuated three-finger hand and (ii) a custom built, vacuum-based gripper. The two systems significantly differ on how they need to be placed relative to an object so that a successful grasp occurs. The first tool provides increased flexibility, while the vacuum alternative is simpler and has smaller form. The objective is to highlight how the end-effector choice can significantly influence the success rate of robotic picking as well as the speed of the overall solution. For the evaluation, the same grasping planning process is followed with both end-effectors given knowledge of an objectstextquoteright pose. Multiple objects with different geometries and characteristics are placed in various poses for testing purposes. The resulting trajectories are executed on a real system to evaluate the effectiveness of the corresponding end-effector modalities in practice. The results indicate that, under different conditions, different types of end-effectors can be beneficial, which motivates the development of hybrid solutions. |
Kimmel, A; Bekris, K Scheduling Pick-and-Place Tasks for Dual-Arm Manipulators Using Incremental Search on Coordination Diagrams Journal Article 2016. @article{Kimmel:2016aa, title = {Scheduling Pick-and-Place Tasks for Dual-Arm Manipulators Using Incremental Search on Coordination Diagrams}, author = {A Kimmel and K Bekris}, url = {http://www.cs.rutgers.edu/~kb572/pubs/kimmel_schedule.pdf}, year = {2016}, date = {2016-06-01}, booktitle = {ICAPS workshop on planning and robotics (PlanRob)}, address = {London, UK}, abstract = {A premise of dual-arm robots is increased efficiency relative to single-arm counterparts in manipulation challenges. Nevertheless, moving two high-dimensional arms simultaneously in the same space is challenging and care must be taken so that collisions are avoided. Given trajectories for two arms to pick two objects, velocity tuning over a coordination diagram can resolve collisions. When multiple objects need to be moved, a scheduling challenge also arises. It involves finding the order with which objects should be manipulated. This paper considers two ways to approach this combination of scheduling and coordination challenges: (i) a ``batch'' approach, where an ordering of objects is selected first; for the given ordering, velocity tuning is performed over a matrix of coordination diagrams that considers all pairs of pick-and-place trajectories; and (ii) an incremental approach, where the ordering of objects is discovered on the fly given the subset of coordination diagrams that arise depending on which object one of the arms is currently manipulating. Simulated experiments for a Baxter robot show that both methods return significantly more efficient trajectories relative to the naive ``round-robin'' schedule, where only one arm moves at a time. Furthermore, the incremental approach is computationally faster, it implicitly provides a good schedule for picking objects and can be used effectively when objects appear dynamically.}, keywords = {}, pubstate = {published}, tppubtype = {article} } A premise of dual-arm robots is increased efficiency relative to single-arm counterparts in manipulation challenges. Nevertheless, moving two high-dimensional arms simultaneously in the same space is challenging and care must be taken so that collisions are avoided. Given trajectories for two arms to pick two objects, velocity tuning over a coordination diagram can resolve collisions. When multiple objects need to be moved, a scheduling challenge also arises. It involves finding the order with which objects should be manipulated. This paper considers two ways to approach this combination of scheduling and coordination challenges: (i) a ``batch'' approach, where an ordering of objects is selected first; for the given ordering, velocity tuning is performed over a matrix of coordination diagrams that considers all pairs of pick-and-place trajectories; and (ii) an incremental approach, where the ordering of objects is discovered on the fly given the subset of coordination diagrams that arise depending on which object one of the arms is currently manipulating. Simulated experiments for a Baxter robot show that both methods return significantly more efficient trajectories relative to the naive ``round-robin'' schedule, where only one arm moves at a time. Furthermore, the incremental approach is computationally faster, it implicitly provides a good schedule for picking objects and can be used effectively when objects appear dynamically. |
Krontiris, A; Bekris, K International Conference on Robotics and Automation (ICRA), Stockholm, Sweden, 2016. @conference{Krontiris:2016ab, title = {Efficiently Solving General Rearrangement Tasks: A Fast Extension Primitive for an Incremental Sampling-Based Planner}, author = {A Krontiris and K Bekris}, url = {http://www.cs.rutgers.edu/~kb572/pubs/fast_object_rearrangement.pdf}, year = {2016}, date = {2016-05-01}, booktitle = {International Conference on Robotics and Automation (ICRA)}, address = {Stockholm, Sweden}, abstract = {Manipulating movable obstacles is a hard problem that involves searching high-dimensional configuration spaces. A milestone method for this problem by Stilman et al. was able to compute solutions for monotone instances. These are problems where every object needs to be transferred at most once to achieve a desired arrangement of all objects. The method uses backtracking search to find the order with which objects should be moved in the environment. This paper first proposes an approximate but significantly faster alternative for monotone rearrangement instances. The method defines a dependency graph between objects given the minimum constraint removal paths (Minimum Constraint Removal) to transfer each object to its target. From this graph it is possible to discover the order of moving the objects by performing topological sorting without the need for backtracking search. The approximation arises from the limitation to consider only the Minimum Constraint Removal paths for moving the objects. Such paths, however, minimize the number of conflicts between the objects. To solve non-monotone instances, this primitive is incorporated in a higher-level incremental search algorithm for general rearrangement planning, that operates similar to Bi-RRT. Given a start and a goal arrangement of objects, tree structures of reachable new arrangements are generated by using the primitive as an expansion procedure. The integrated solution achieves probabilistic completeness for the general non-monotone case and based on simulated experiments it achieves very good success ratios, solution times and path quality relative to all the alternatives considered.}, keywords = {}, pubstate = {published}, tppubtype = {conference} } Manipulating movable obstacles is a hard problem that involves searching high-dimensional configuration spaces. A milestone method for this problem by Stilman et al. was able to compute solutions for monotone instances. These are problems where every object needs to be transferred at most once to achieve a desired arrangement of all objects. The method uses backtracking search to find the order with which objects should be moved in the environment. This paper first proposes an approximate but significantly faster alternative for monotone rearrangement instances. The method defines a dependency graph between objects given the minimum constraint removal paths (Minimum Constraint Removal) to transfer each object to its target. From this graph it is possible to discover the order of moving the objects by performing topological sorting without the need for backtracking search. The approximation arises from the limitation to consider only the Minimum Constraint Removal paths for moving the objects. Such paths, however, minimize the number of conflicts between the objects. To solve non-monotone instances, this primitive is incorporated in a higher-level incremental search algorithm for general rearrangement planning, that operates similar to Bi-RRT. Given a start and a goal arrangement of objects, tree structures of reachable new arrangements are generated by using the primitive as an expansion procedure. The integrated solution achieves probabilistic completeness for the general non-monotone case and based on simulated experiments it achieves very good success ratios, solution times and path quality relative to all the alternatives considered. |
Rennie, C; Shome, R; Bekris, K; Souza, A A Dataset for Improved RGBD-Based Object Detection and Pose Estimation for Warehouse Pick-and-Place Journal Article IEEE Robotics and Automation Letters (RA-L) [Also accepted to appear at the 2016 IEEE International Conference on Robotics and Automation (ICRA)], 1 , pp. 1179–1185, 2016. @article{Rennie:2016aa, title = {A Dataset for Improved RGBD-Based Object Detection and Pose Estimation for Warehouse Pick-and-Place}, author = {C Rennie and R Shome and K Bekris and A Souza}, url = {http://www.cs.rutgers.edu/~kb572/pubs/icra16_pose_estimation.pdf}, year = {2016}, date = {2016-02-01}, journal = {IEEE Robotics and Automation Letters (RA-L) [Also accepted to appear at the 2016 IEEE International Conference on Robotics and Automation (ICRA)]}, volume = {1}, pages = {1179--1185}, address = {Stockholm, Sweden}, abstract = {An important logistics application of robotics involves manipulators that pick-and-place objects placed in warehouse shelves. A critical aspect of this task corresponds to detecting the pose of a known object in the shelf using visual data. Solving this problem can be assisted by the use of an RGB-D sensor, which also provides depth information beyond visual data. Nevertheless, it remains a challenging problem since multiple issues need to be addressed, such as low illumination inside shelves, clutter, texture-less and reflective objects as well as the limitations of depth sensors. This paper provides a new rich data set for advancing the state-of-the-art in RGBD-based 3D object pose estimation, which is focused on the challenges that arise when solving warehouse pick-and-place tasks. The publicly available data set includes thousands of images and corresponding ground truth data for the objects used during the first Amazon Picking Challenge at different poses and clutter conditions. Each image is accompanied with ground truth information to assist in the evaluation of algorithms for object detection. To show the utility of the data set, a recent algorithm for RGBD-based pose estimation is evaluated in this paper. Based on the measured performance of the algorithm on the data set, various modifications and improvements are applied to increase the accuracy of detection. These steps can be easily applied to a variety of different methodologies for object pose detection and improve performance in the domain of warehouse pick-and-place.}, keywords = {}, pubstate = {published}, tppubtype = {article} } An important logistics application of robotics involves manipulators that pick-and-place objects placed in warehouse shelves. A critical aspect of this task corresponds to detecting the pose of a known object in the shelf using visual data. Solving this problem can be assisted by the use of an RGB-D sensor, which also provides depth information beyond visual data. Nevertheless, it remains a challenging problem since multiple issues need to be addressed, such as low illumination inside shelves, clutter, texture-less and reflective objects as well as the limitations of depth sensors. This paper provides a new rich data set for advancing the state-of-the-art in RGBD-based 3D object pose estimation, which is focused on the challenges that arise when solving warehouse pick-and-place tasks. The publicly available data set includes thousands of images and corresponding ground truth data for the objects used during the first Amazon Picking Challenge at different poses and clutter conditions. Each image is accompanied with ground truth information to assist in the evaluation of algorithms for object detection. To show the utility of the data set, a recent algorithm for RGBD-based pose estimation is evaluated in this paper. Based on the measured performance of the algorithm on the data set, various modifications and improvements are applied to increase the accuracy of detection. These steps can be easily applied to a variety of different methodologies for object pose detection and improve performance in the domain of warehouse pick-and-place. |
Correll, N; Bekris, K; Berenson, D; Brock, O; Causo, A; Hauser, K; Okada, K; Rodriguez, A; Romano, J; Wurman, P Analysis and Observations from the First Amazon Picking Challenge Journal Article IEEE Transactions on Automation Science and Engineering (T-ASE), 15 (1), pp. 172-188, 2016. @article{Correll:2016aa, title = {Analysis and Observations from the First Amazon Picking Challenge}, author = {N Correll and K Bekris and D Berenson and O Brock and A Causo and K Hauser and K Okada and A Rodriguez and J Romano and P Wurman}, url = {http://arxiv.org/abs/1601.05484}, year = {2016}, date = {2016-01-01}, journal = {IEEE Transactions on Automation Science and Engineering (T-ASE)}, volume = {15}, number = {1}, pages = {172-188}, abstract = {This paper presents a overview of the inaugural Amazon Picking Challenge along with a summary of a survey conducted among the 26 participating teams. The challenge goal was to design an autonomous robot to pick items from a warehouse shelf. This task is currently performed by human workers, and there is hope that robots can someday help increase efficiency and throughput while lowering cost. We report on a 28-question survey posed to the teams to learn about each team's background, mechanism design, perception apparatus, planning and control approach. We identify trends in this data, correlate it with each team's success in the competition, and discuss observations and lessons learned based on survey results and the authors' personal experiences during the challenge.}, keywords = {}, pubstate = {published}, tppubtype = {article} } This paper presents a overview of the inaugural Amazon Picking Challenge along with a summary of a survey conducted among the 26 participating teams. The challenge goal was to design an autonomous robot to pick items from a warehouse shelf. This task is currently performed by human workers, and there is hope that robots can someday help increase efficiency and throughput while lowering cost. We report on a 28-question survey posed to the teams to learn about each team's background, mechanism design, perception apparatus, planning and control approach. We identify trends in this data, correlate it with each team's success in the competition, and discuss observations and lessons learned based on survey results and the authors' personal experiences during the challenge. |
2015 |
Dobson, A; Bekris, K Planning Representations and Algorithms for Prehensile Multi-Arm Manipulation Conference IEEE/RSJ International Conference on Intelligent Robots and Systems, Hamburg, Germany, 2015. @conference{Dobson:2015ab, title = {Planning Representations and Algorithms for Prehensile Multi-Arm Manipulation}, author = {A Dobson and K Bekris}, url = {http://www.cs.rutgers.edu/~kb572/pubs/Dobson_Bekris_multi_arm_iros15.pdf}, year = {2015}, date = {2015-09-01}, booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems}, address = {Hamburg, Germany}, abstract = {This paper describes the topology of general multi-arm prehensile manipulation by extending work on the single and dual-arm cases. Reasonable assumptions are applied to reduce the number of manipulation modes, which results in an explicit graphical representation for multi-arm manipulation that is computationally manageable to store and search for solution paths. In this context, it is also possible to take advantage of preprocessing steps to significantly speed up online query resolution. The approach is evaluated in simulation for multiple arms showing it is possible to quickly compute multi-arm manipulation paths of high-quality on the fly.}, keywords = {}, pubstate = {published}, tppubtype = {conference} } This paper describes the topology of general multi-arm prehensile manipulation by extending work on the single and dual-arm cases. Reasonable assumptions are applied to reduce the number of manipulation modes, which results in an explicit graphical representation for multi-arm manipulation that is computationally manageable to store and search for solution paths. In this context, it is also possible to take advantage of preprocessing steps to significantly speed up online query resolution. The approach is evaluated in simulation for multiple arms showing it is possible to quickly compute multi-arm manipulation paths of high-quality on the fly. |
Krontiris, A; Bekris, K Dealing with Difficult Instances of Object Rearrangement Conference Robotics: Science and Systems (RSS), 1123 , [Best Paper &amp; Best Student Paper Award Finalists] [Best Paper &amp; Best Student Paper Award Finalists], Rome, Italy, 2015. @conference{Krontiris:2015ab, title = {Dealing with Difficult Instances of Object Rearrangement}, author = {A Krontiris and K Bekris}, url = {https://people.cs.rutgers.edu/~kb572/pubs/Krontiris_Bekris_rearrangement_RSS2015.pdf}, year = {2015}, date = {2015-07-01}, booktitle = {Robotics: Science and Systems (RSS)}, volume = {1123}, publisher = {[Best Paper &amp; Best Student Paper Award Finalists]}, address = {Rome, Italy}, organization = {[Best Paper &amp; Best Student Paper Award Finalists]}, abstract = {An important skill for robots is the effective rearrangement of multiple objects so as to deal with clutter in human spaces. This paper proposes a simple but general primitive for rearranging multiple objects and its use in task planning frameworks. Rearrangement is a challenging problem as it involves combinatorially large, continuous C-spaces for multiple movable bodies and with kinematic constraints. This work starts by reviewing an existing search-based approach, which quickly computes solutions for monotone challenges, i.e., when objects need to be grasped only once so as to be rearranged. This work focuses on non-monotone challenges, as well as labeled problems, which some of the related efforts do not address. The first contribution is the extension of the monotone solution to a method that addresses a subset of non-monotone challenges. Then, this work proposes the use of the resulting non-monotone solver as a local planner in the context of a higher-level task planner that searches the space of object placements and for which stronger guarantees can be provided. The paper aims to emphasize the benefit of using more powerful motion primitives in the context of task planning for object rearrangement. Experiments in simulation using a model of a Baxter robot arm show the capability of solving difficult instances of rearrangement problems and evaluate the methods in terms of success ratio, computational requirements, scalability and path quality.}, keywords = {}, pubstate = {published}, tppubtype = {conference} } An important skill for robots is the effective rearrangement of multiple objects so as to deal with clutter in human spaces. This paper proposes a simple but general primitive for rearranging multiple objects and its use in task planning frameworks. Rearrangement is a challenging problem as it involves combinatorially large, continuous C-spaces for multiple movable bodies and with kinematic constraints. This work starts by reviewing an existing search-based approach, which quickly computes solutions for monotone challenges, i.e., when objects need to be grasped only once so as to be rearranged. This work focuses on non-monotone challenges, as well as labeled problems, which some of the related efforts do not address. The first contribution is the extension of the monotone solution to a method that addresses a subset of non-monotone challenges. Then, this work proposes the use of the resulting non-monotone solver as a local planner in the context of a higher-level task planner that searches the space of object placements and for which stronger guarantees can be provided. The paper aims to emphasize the benefit of using more powerful motion primitives in the context of task planning for object rearrangement. Experiments in simulation using a model of a Baxter robot arm show the capability of solving difficult instances of rearrangement problems and evaluate the methods in terms of success ratio, computational requirements, scalability and path quality. |
Krontiris, A; Bekris, K Computational Tradeoffs of Search Methods for Minimum Constraint Removal Paths Conference Symposium on Combinatorial Search (SoCS), Dead Sea, Israel, 2015. @conference{Krontiris:2015aa, title = {Computational Tradeoffs of Search Methods for Minimum Constraint Removal Paths}, author = {A Krontiris and K Bekris}, url = {http://www.cs.rutgers.edu/~kb572/pubs/Krontiris_SoCS2015_MCR.pdf}, year = {2015}, date = {2015-06-01}, booktitle = {Symposium on Combinatorial Search (SoCS)}, address = {Dead Sea, Israel}, abstract = {The typical objective of path planning is to find the shortest feasible path. Many times, however, there may be no solution given the existence of constraints, such as obstacles. In these cases, the minimum constraint removal problem asks for the minimum set of constraints that need to be removed from the state space to find a solution. For instance, in manipulation planning, it is desirable to compute the minimum set of obstacles to be cleared from the workspace to manipulate a target object. Unfortunately, minimum constraint removal paths do not exhibit dynamic programming properties, i.e., subsets of optimum solutions are not necessarily optimal. Thus, searching for such solutions is computationally expensive. This leads to approximate methods, which balance the cost of computing a solution and its quality. This work investigates alternatives in this context and evaluates their performance in terms of such tradeoffs. Solutions that follow a bounded-length approach, i.e., searching for paths up to a certain length, seem to provide a good balance between minimizing constraints, computational cost and path length.}, keywords = {}, pubstate = {published}, tppubtype = {conference} } The typical objective of path planning is to find the shortest feasible path. Many times, however, there may be no solution given the existence of constraints, such as obstacles. In these cases, the minimum constraint removal problem asks for the minimum set of constraints that need to be removed from the state space to find a solution. For instance, in manipulation planning, it is desirable to compute the minimum set of obstacles to be cleared from the workspace to manipulate a target object. Unfortunately, minimum constraint removal paths do not exhibit dynamic programming properties, i.e., subsets of optimum solutions are not necessarily optimal. Thus, searching for such solutions is computationally expensive. This leads to approximate methods, which balance the cost of computing a solution and its quality. This work investigates alternatives in this context and evaluates their performance in terms of such tradeoffs. Solutions that follow a bounded-length approach, i.e., searching for paths up to a certain length, seem to provide a good balance between minimizing constraints, computational cost and path length. |
2014 |
Zhao, M Identifying Features of Legible Manipulation Paths Masters Thesis Rutgers, the State University of New Jersey, 2014. @mastersthesis{Zhao:2014ab, title = {Identifying Features of Legible Manipulation Paths}, author = {M Zhao}, url = {http://www.cs.rutgers.edu/~kb572/pubs/Zhao_Min_MSThesis_legibility.pdf}, year = {2014}, date = {2014-10-01}, volume = {MS}, address = {Piscataway, New Jersey}, school = {Rutgers, the State University of New Jersey}, abstract = {This work performs an experimental study on the legibility of paths executed by a manipulation arm available on a Baxter robot. In this context, legibility is defined as the ability of people to effectively predict the target of the armtextquoterights motion. Paths that are legible can improve the collaboration of robots with humans since they allow people to intuitively understand the robottextquoterights intentions. Each experimental trial in this study reproduces manipulator motions to one of many targets in front of the robot. An appropriate experimental setup was developed in order to collect the responses of people in terms of the perceived robottextquoterights target during the execution of a trajectory by Baxter. The objective of the experimental setup was to minimize the cognitive load of the human subjects during the collection of data. The extensive experimental data provide insights into the features of motion that make certain paths more legible for humans than other paths. For instance, motions where the end-effector is oriented towards the intended target appear to be better in terms of legibility than alternatives.}, keywords = {}, pubstate = {published}, tppubtype = {mastersthesis} } This work performs an experimental study on the legibility of paths executed by a manipulation arm available on a Baxter robot. In this context, legibility is defined as the ability of people to effectively predict the target of the armtextquoterights motion. Paths that are legible can improve the collaboration of robots with humans since they allow people to intuitively understand the robottextquoterights intentions. Each experimental trial in this study reproduces manipulator motions to one of many targets in front of the robot. An appropriate experimental setup was developed in order to collect the responses of people in terms of the perceived robottextquoterights target during the execution of a trajectory by Baxter. The objective of the experimental setup was to minimize the cognitive load of the human subjects during the collection of data. The extensive experimental data provide insights into the features of motion that make certain paths more legible for humans than other paths. For instance, motions where the end-effector is oriented towards the intended target appear to be better in terms of legibility than alternatives. |
Krontiris, A; Shome, R; Dobson, A; Kimmel, A; Bekris, K Rearranging Similar Objects with a Manipulator Using Pebble Graphs Conference IEEE-RAS International Conference on Humanoid Robots (HUMANOIDS), Madrid, Spain, 2014. @conference{Krontiris:2014aa, title = {Rearranging Similar Objects with a Manipulator Using Pebble Graphs}, author = {A Krontiris and R Shome and A Dobson and A Kimmel and K Bekris}, url = {http://www.cs.rutgers.edu/~kb572/pubs/Krontiris_Humanoids14_rearrangement.pdf}, year = {2014}, date = {2014-01-01}, booktitle = {IEEE-RAS International Conference on Humanoid Robots (HUMANOIDS)}, address = {Madrid, Spain}, abstract = {This work proposes a method for effectively computing manipulation paths to rearrange similar objects in a cluttered space. Rearrangement is a challenging problem as it involves combinatorially large, continuous configuration spaces due to the presence of multiple bodies and kinematically complex manipulators. This work leverages ideas from algorithmic theory, multi-robot motion planning and manipulation planning to propose appropriate graphical representations for this challenge. These representations allow to quickly reason whether manipulation paths allow the transition between entire sets of object arrangements without having to explicitly store these arrangements. The proposed method also allows to take advantage of precomputation given a manipulation roadmap for transferring a single object in the same cluttered space. The resulting approach is probabilistically complete for a wide set of problem instances. It is evaluated in simulation for a realistic model of a Baxter robot and executed on the real system, showing that the approach solves complex instances and is promising in terms of scalability and success ratio.}, keywords = {}, pubstate = {published}, tppubtype = {conference} } This work proposes a method for effectively computing manipulation paths to rearrange similar objects in a cluttered space. Rearrangement is a challenging problem as it involves combinatorially large, continuous configuration spaces due to the presence of multiple bodies and kinematically complex manipulators. This work leverages ideas from algorithmic theory, multi-robot motion planning and manipulation planning to propose appropriate graphical representations for this challenge. These representations allow to quickly reason whether manipulation paths allow the transition between entire sets of object arrangements without having to explicitly store these arrangements. The proposed method also allows to take advantage of precomputation given a manipulation roadmap for transferring a single object in the same cluttered space. The resulting approach is probabilistically complete for a wide set of problem instances. It is evaluated in simulation for a realistic model of a Baxter robot and executed on the real system, showing that the approach solves complex instances and is promising in terms of scalability and success ratio. |
Krontiris, A; Shome, R; Dobson, A; Kimmel, A; Yochelson, I; Bekris, K Similar Part Rearrangement with Pebble Graphs Journal Article CoRR, abs/1404.6573 , 2014. @article{Krontiris:2014ab, title = {Similar Part Rearrangement with Pebble Graphs}, author = {A Krontiris and R Shome and A Dobson and A Kimmel and I Yochelson and K Bekris}, url = {https://arxiv.org/abs/1404.6573}, year = {2014}, date = {2014-01-01}, journal = {CoRR}, volume = {abs/1404.6573}, abstract = {This work proposes a method for effectively computing manipulation paths to rearrange similar objects in a cluttered space. The solution can be used to place similar products in a factory floor in a desirable arrangement or for retrieving a particular object from a shelf blocked by similarly sized objects. These are challenging problems as they involve combinatorially large, continuous configuration spaces due to the presence of multiple moving bodies and kinematically complex manipulators. This work leverages ideas from algorithmic theory, multi-robot motion planning and manipulation planning to propose appropriate graphical representations for this challenge. These representations allow to quickly reason whether manipulation paths allow the transition between entire sets of objects arrangements without having to explicitly enumerate the path for each pair of arrangements. The proposed method also allows to take advantage of precomputation given a manipulation roadmap for transferring a single object in the same cluttered space. The resulting approach is evaluated in simulation for a realistic model of a Baxter robot and executed in open-loop on the real system, showing that the approach solves complex instances and is promising in terms of scalability and success ratio.}, keywords = {}, pubstate = {published}, tppubtype = {article} } This work proposes a method for effectively computing manipulation paths to rearrange similar objects in a cluttered space. The solution can be used to place similar products in a factory floor in a desirable arrangement or for retrieving a particular object from a shelf blocked by similarly sized objects. These are challenging problems as they involve combinatorially large, continuous configuration spaces due to the presence of multiple moving bodies and kinematically complex manipulators. This work leverages ideas from algorithmic theory, multi-robot motion planning and manipulation planning to propose appropriate graphical representations for this challenge. These representations allow to quickly reason whether manipulation paths allow the transition between entire sets of objects arrangements without having to explicitly enumerate the path for each pair of arrangements. The proposed method also allows to take advantage of precomputation given a manipulation roadmap for transferring a single object in the same cluttered space. The resulting approach is evaluated in simulation for a realistic model of a Baxter robot and executed in open-loop on the real system, showing that the approach solves complex instances and is promising in terms of scalability and success ratio. |