Manipulation Planning for Object Re-Orientation Based on Semantic Segmentation Keypoint Detection

Wong, Ching-Chang; Yeh, Li-Yu; Liu, Chih-Cheng; Tsai, Chi-Yi; Aoyama, Hisasuki

doi:10.3390/s21072280

Submit to this Journal Review for this Journal Edit a Special Issue

Article Menu

...

Open AccessArticle

Manipulation Planning for Object Re-Orientation Based on Semantic Segmentation Keypoint Detection

^1,*,

¹,

¹,

¹ and

Department of Electrical and Computer Engineering, Tamkang University, New Taipei City 25137, Taiwan

Department of Mechanical and Intelligent Systems Engineering, University of Electro-Communications, Tokyo 182-8585, Japan

Author to whom correspondence should be addressed.

Academic Editor: Hanseok Ko

Sensors 2021, 21(7), 2280; https://doi.org/10.3390/s21072280

Received: 31 January 2021 / Revised: 21 March 2021 / Accepted: 22 March 2021 / Published: 24 March 2021

(This article belongs to the Special Issue Sensor Fusion for Object Detection, Classification and Tracking)

View Full-Text Download PDF

Browse Figures

Citation Export

Abstract

In this paper, a manipulation planning method for object re-orientation based on semantic segmentation keypoint detection is proposed for robot manipulator which is able to detect and re-orientate the randomly placed objects to a specified position and pose. There are two main parts: (1) 3D keypoint detection system; and (2) manipulation planning system for object re-orientation. In the 3D keypoint detection system, an RGB-D camera is used to obtain the information of the environment and can generate 3D keypoints of the target object as inputs to represent its corresponding position and pose. This process simplifies the 3D model representation so that the manipulation planning for object re-orientation can be executed in a category-level manner by adding various training data of the object in the training phase. In addition, 3D suction points in both the object’s current and expected poses are also generated as the inputs of the next operation stage. During the next stage, Mask Region-Convolutional Neural Network (Mask R-CNN) algorithm is used for preliminary object detection and object image. The highest confidence index image is selected as the input of the semantic segmentation system in order to classify each pixel in the picture for the corresponding pack unit of the object. In addition, after using a convolutional neural network for semantic segmentation, the Conditional Random Fields (CRFs) method is used to perform several iterations to obtain a more accurate result of object recognition. When the target object is segmented into the pack units of image process, the center position of each pack unit can be obtained. Then, a normal vector of each pack unit’s center points is generated by the depth image information and pose of the object, which can be obtained by connecting the center points of each pack unit. In the manipulation planning system for object re-orientation, the pose of the object and the normal vector of each pack unit are first converted into the working coordinate system of the robot manipulator. Then, according to the current and expected pose of the object, the spherical linear interpolation (Slerp) algorithm is used to generate a series of movements in the workspace for object re-orientation on the robot manipulator. In addition, the pose of the object is adjusted on the z-axis of the object’s geodetic coordinate system based on the image features on the surface of the object, so that the pose of the placed object can approach the desired pose. Finally, a robot manipulator and a vacuum suction cup made by the laboratory are used to verify that the proposed system can indeed complete the planned task of object re-orientation. View Full-Text

Keywords: object re-orientation; pick-and-place; Mask R-CNN; semantic segmentation; 3D keypoint detection

▼ Show Figures

Figure 1

This is an open access article distributed under the Creative Commons Attribution License which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited

Share and Cite

MDPI and ACS Style

Wong, C.-C.; Yeh, L.-Y.; Liu, C.-C.; Tsai, C.-Y.; Aoyama, H. Manipulation Planning for Object Re-Orientation Based on Semantic Segmentation Keypoint Detection. Sensors 2021, 21, 2280. https://doi.org/10.3390/s21072280

AMA Style

Wong C-C, Yeh L-Y, Liu C-C, Tsai C-Y, Aoyama H. Manipulation Planning for Object Re-Orientation Based on Semantic Segmentation Keypoint Detection. Sensors. 2021; 21(7):2280. https://doi.org/10.3390/s21072280

Chicago/Turabian Style

Wong, Ching-Chang, Li-Yu Yeh, Chih-Cheng Liu, Chi-Yi Tsai, and Hisasuki Aoyama. 2021. "Manipulation Planning for Object Re-Orientation Based on Semantic Segmentation Keypoint Detection" Sensors 21, no. 7: 2280. https://doi.org/10.3390/s21072280

Find Other Styles

Note that from the first issue of 2016, MDPI journals use article numbers instead of page numbers. See further details here.

Article Metrics

Abstract views Pdf views Html views

Article Access Map by Country/Region

Author Biographies

Ching-Chang Wong

Ching-Chang Wong received the B.S. degree from the Department of Electronic Engineering, Tamkang University, Taiwan, in 1984, and the M.S. and Ph.D. degrees from the Department of Electrical Engineering, Tatung Institute of Technology, Taiwan, in 1986 and 1989, respectively. In 1989, he joined the Department of Electrical and Computer Engineering, Tamkang University (TKU), where he served as the Department Chairman from 2006 to 2010. In 2007, he established the Robotics Engineering Institute. He is currently a Distinguished Professor. From 2009 to 2010, he served the Chair of the IEEE Robotics & Automation Taipei Chapter. In 2011, he established the Intelligent Automation and Robotics Center. He was elevated as a Fellow of The Institution of Engineering and Technology (IET), the Chinese Automatic Control Society (CACS), and the Robotics Society of Taiwan (RST), in 2009, 2015, and 2019, respectively. He was a recipient of the Outstanding Automatic Control Award from the CACS, Taiwan, in 2009, and the Outstanding Robotics Engineering Award from the RST in 2018. He received the Outstanding and Premium Research Awards (2011–2020) from the Ministry of Science and Technology (MOST), Taiwan. He has published and co-authored over 300 technical articles and 20 patents. His current research interests include intelligent control, humanoid robot, mobile robot manipulator, and deep reinforcement learning for robotic applications.

Li-Yu Yeh

Li-Yu Yeh received the B.S. degree in information and telecommunications engineering from Ming Chuan University, Taoyuan City, Taiwan, in 2017, and the M.S. degree in electrical and computer engineering from Tamkang University, New Taipei City, Taiwan, in 2020. His research interests include robot manipulator, robotic applications, deep learning, and computer vision.

Chih-Cheng Liu

Chih-Cheng Liu received the B.S., M.S., and Ph.D. degrees in electrical engineering from Tamkang University, Taiwan, in 2004, 2006, and 2014, respectively. He joined the Department of Electrical Engineering of Tamkang University in 2017 and now is an assistant professor. His research interests include humanoid robots, robotic applications, FPGA design, and SOPC design.

Chi-Yi Tsai

Chi-Yi Tsai received the B.S. and M.S. degrees in electrical engineering from the National Yunlin University of Science and Technology, Yunlin, Taiwan, in 2000 and 2002, respectively, and the Ph. D. degree in electrical and control engineering from National Chiao Tung University, Hsinchu, Taiwan, in 2008. In 2010, he joined the Department of Electrical Engineering, Tamkang University, New Taipei City, Taiwan, where he is currently a Professor. His research interests include image processing, color image enhancement processing, visual tracking, visual servoing, computer vision, and deep learning.

Hisasuki Aoyama

Hisayuki Aoyama was born in 1958 in Japan. He receive his bachelor degree in Mechanical Engineering for Production from Tokyo Institute of Technology, Tokyo, Japan in 1981. He received his master degree in Precision Machinery in 1983 and doctor degree in Precision Engineering in 1988 from Tokyo Institute of Technology. He worked at Research Laboratory of Precision Machinery and Electronics, Tokyo Institute of Technology from 1983 to 1988. He became an Associate Professor, Department of Precision Engineering, Shizuoka University and an Associate Professor, Department of Mechanical Engineering and Intelligent Systems, University of Electro-Communications in 1997. He was a visiting researcher at College of Manufacturing, Cranfield Institute of Technology, U.K. during 1989–1990, Department of E.C.E, North Carolina State University, U.S.A.in 1996 and King Mongkut's Institute of Technology Ladkrabang, Thailand in 2017. In 2002, he became Professor, Director of Micro Robotics & Mechatronics Group and Director of Global Alliance Laboratory Project. He is interested in micro/precision engineering and micro metrology, as well as its industrial and biomedical applications.

Article Menu

Manipulation Planning for Object Re-Orientation Based on Semantic Segmentation Keypoint Detection

Abstract

Share and Cite

Article Metrics

Article Access Statistics

Article Access Map by Country/Region

Author Biographies

Further Information

Guidelines

MDPI Initiatives

Follow MDPI