Canopy Segmentation Using ResNet for Mechanical Harvesting of Apples. Issue 30 (2019)

• Record Type: Journal Article
• Authors: Zhang, Xin; Fu, Longsheng; Karkee, Manoj; D. Whiting, Matthew; Zhang, Qin
• Is Part Of: IFAC-PapersOnLine. Volume 52:Issue 30(2019)
• Page Start: 300
• Page End: 305
• Publication Date: 2019
• Subjects: Semantic segmentation -- deep learning -- convolutional neural networks (CNNs) -- mass harvesting -- foliage canopy -- formally trained tree architecture; Automatic control -- Periodicals; 629.805
• DOI: 10.1016/j.ifacol.2019.12.550
• Languages: English
• ISSNs: 2405-8963

Abstract

Abstract: Fresh market apple is the number one premium fruit crop in Washington state, accounting for more than 60% of U.S. national production every year. With the rapid increment of agricultural labor cost and the decrement of labor availability, mechanical apple harvesting is considered as an alternative solution. To further improve the efficiency of the harvester by automatically locating the target tree trunk and/or branch, the tree canopy under the full foliage condition needs to be analyzed. In this study, convolutional neural network (CNN)-based semantic segmentation method was adopted to segment the tree canopy using a pre-trained and modified ResNet-18 implementation of CNN. In total, 253 images were acquired using a Kinect V2 camera in a commercial "Fuji" apple orchard (trained in the formal architecture with V-axis) during 2018 harvest season near Prosser, Washington. Among those images, 152 (60%), 51 (20%), and 50 (20%) were used for network training, validating, and testing, respectively. Three different classes of pixels were defined including 'trunk/branch', 'apples', and 'leaves' (background) for each image. Then, three commonly adopted evaluation measures were employed to examine the performance of the model; i) normalized confusion matrix (per-class accuracy), ii) intersection over union (IoU), and iii) boundary-F1 score (BFScore) on the image basis. Test results showed that all three classes achieved reasonably high per-class accuracies of 94.8% (trunk/branch), 97.5% (apples), and 94.5% (leaves), respectively. In addition, IoUs for each class in the same sequence were 0.408, 0.717, and 0.944 whereas BFScore for each class were 0.761, 0.915, and 0.887, respectively. Among all the three classes, a generally poor result was achieved for 'trunk/branch' compared to the same for other two classes, which was primarily due to proportionally smaller number of pixels in each image. The results from this study indicated that the efficacy of the mechanical harvesting technique of apples could be potentially improved by automatically locating and shaking the trunk/branch under full foliage canopy during the harvest seasons.