A mango picking vision algorithm on instance segmentation and key point detection from RGB images in an open orchard. (June 2021)

• Record Type: Journal Article
• Authors: Zheng, Chan; Chen, Pengfei; Pang, Jing; Yang, Xiaofan; Chen, Changxin; Tu, Shuqin; Xue, Yueju
• Is Part Of: Biosystems engineering. Volume 206(2021)
• Page Start: 32
• Page End: 54
• Publication Date: 2021-06
• Subjects: Fruit picking robot -- Instance segmentation -- Picking-point detection -- End-to-end framework -- Complex scene -- Occlusion and overlap; Bioengineering -- Periodicals; Agricultural engineering -- Periodicals; Biological systems -- Periodicals; Génie rural -- Périodiques; Systèmes biologiques -- Périodiques; 631
• DOI: 10.1016/j.biosystemseng.2021.03.012
• Languages: English
• ISSNs: 1537-5110

Abstract

Abstract : The vision system of a fruit picking robot must perform two difficult tasks, the accurate pixel-wise instance segmentation of fruit and the correct detection of picking points. Due to the unsatisfactory performance of the vision system on these tasks, the picking performance of the robot is insufficient and its application is not prevalent. In this study, performing fruit instance segmentation and the localization of picking points, a vision algorithm using RGB images was designed in a deep learning framework for a potential visual system of mango picking robots. The algorithm not only performed these two tasks successfully, but also integrated them into an end-to-end network, in which parallel branches performed the two tasks simultaneously. The average precision (at IoU of 0.75) and the average recall of instance segmentation reached 0.947 and 0.929 respectively, and the best precision and recall of picking-point detection reached 0.984 and 0.908 respectively. In addition, the tasks face various illumination and background interference in outdoor orchards, along with complex problems in terms of occlusion, overlap and object scale. In this study, the performance of the vision system was analysed in detail on several datasets and subsets of the various complex conditions, and the major factors that affect the performance were discussed. The results demonstrated that this system was robust against various illuminations and complex backgrounds, and yielded satisfactory segmentation and picking-point detection performances for minor and medium occlusion or overlap, and for medium and large mangoes. The model visualization and the influence analysis of model training demonstrated the training process and modelling effect of the deep learning network. Highlights: Designed an end-to-end vision system for a mango picking robot. Instance segmentation and pick point detection are performed simultaneously. It is robust to various illumination and background interference scenarios. Good performances in minor and medium occlusion, various overlap and object scale.