TY - JOUR
T1 - A Deep Learning Approach to Grasping the Invisible
AU - Yang, Yang
AU - Liang, Hengyue
AU - Choi, Changhyun
N1 - Publisher Copyright:
© 2020 IEEE.
Copyright:
Copyright 2020 Elsevier B.V., All rights reserved.
PY - 2020/4
Y1 - 2020/4
N2 - We study an emerging problem named 'grasping the invisible' in robotic manipulation, in which a robot is tasked to grasp an initially invisible target object via a sequence of pushing and grasping actions. In this problem, pushes are needed to search for the target and rearrange cluttered objects around it to enable effective grasps. We propose to solve the problem by formulating a deep learning approach in a critic-policy format. The target-oriented motion critic, which maps both visual observations and target information to the expected future rewards of pushing and grasping motion primitives, is learned via deep Q-learning. We divide the problem into two subtasks, and two policies are proposed to tackle each of them, by combining the critic predictions and relevant domain knowledge. A Bayesian-based policy accounting for past action experience performs pushing to search for the target; once the target is found, a classifier-based policy coordinates target-oriented pushing and grasping to grasp the target in clutter. The motion critic and the classifier are trained in a self-supervised manner through robot-environment interactions. Our system achieves a 93% and 87% task success rate on each of the two subtasks in simulation and an 85% task success rate in real robot experiments on the whole problem, which outperforms several baselines by large margins. Supplementary material is available at http://sites.google.com/umn.edu/grasping-invisible.
AB - We study an emerging problem named 'grasping the invisible' in robotic manipulation, in which a robot is tasked to grasp an initially invisible target object via a sequence of pushing and grasping actions. In this problem, pushes are needed to search for the target and rearrange cluttered objects around it to enable effective grasps. We propose to solve the problem by formulating a deep learning approach in a critic-policy format. The target-oriented motion critic, which maps both visual observations and target information to the expected future rewards of pushing and grasping motion primitives, is learned via deep Q-learning. We divide the problem into two subtasks, and two policies are proposed to tackle each of them, by combining the critic predictions and relevant domain knowledge. A Bayesian-based policy accounting for past action experience performs pushing to search for the target; once the target is found, a classifier-based policy coordinates target-oriented pushing and grasping to grasp the target in clutter. The motion critic and the classifier are trained in a self-supervised manner through robot-environment interactions. Our system achieves a 93% and 87% task success rate on each of the two subtasks in simulation and an 85% task success rate in real robot experiments on the whole problem, which outperforms several baselines by large margins. Supplementary material is available at http://sites.google.com/umn.edu/grasping-invisible.
KW - Dexterous manipulation
KW - computer vision for automation
KW - deep learning in robotics and automation
UR - http://www.scopus.com/inward/record.url?scp=85081093993&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85081093993&partnerID=8YFLogxK
U2 - 10.1109/LRA.2020.2970622
DO - 10.1109/LRA.2020.2970622
M3 - Article
AN - SCOPUS:85081093993
VL - 5
SP - 2232
EP - 2239
JO - IEEE Robotics and Automation Letters
JF - IEEE Robotics and Automation Letters
SN - 2377-3766
IS - 2
M1 - 8976257
ER -