Robotic kiwifruit harvesting using machine vision, convolutional neural networks, and robotic arms. (May 2019)

• Record Type: Journal Article
• Authors: Williams, Henry A.M.; Jones, Mark H.; Nejati, Mahla; Seabright, Matthew J.; Bell, Jamie; Penhall, Nicky D.; Barnett, Josh J.; Duke, Mike D.; Scarfe, Alistair J.; Ahn, Ho Seok; Lim, JongYoon; MacDonald, Bruce A.
• Is Part Of: Biosystems engineering. Volume 181(2019)
• Page Start: 140
• Page End: 156
• Publication Date: 2019-05
• Subjects: Horticulture -- Robotics -- Neural Networking -- Machine Vision -- Harvesting -- Convolution Neural Networks -- Orchard; 00-01 -- 99-00; Bioengineering -- Periodicals; Agricultural engineering -- Periodicals; Biological systems -- Periodicals; Génie rural -- Périodiques; Systèmes biologiques -- Périodiques; 631
• DOI: 10.1016/j.biosystemseng.2019.03.007
• Languages: English
• ISSNs: 1537-5110

Abstract

Abstract : As labour requirements in horticultural become more challenging, automated solutions are becoming an effective approach to maintain productivity and quality. This paper presents the design and performance evaluation of a novel multi-arm kiwifruit harvesting robot designed to operate autonomously in pergola style orchards. The harvester consists of four robotic arms that have been designed specifically for kiwifruit harvesting, each with a novel end-effector developed to enable safe harvesting of the kiwifruit. The vision system leverages recent advances in deep neural networks and stereo matching for reliably detecting and locating kiwifruit in real-world lighting conditions. Furthermore, a novel dynamic fruit scheduling system is presented that has been developed to coordinate the four arms throughout the harvesting process. The performance of the harvester has been measured through a comprehensive and realistic field-trial in a commercial orchard environment. The results show that the presented harvester is capable of successfully harvesting 51.0% of the total number of kiwifruit within the orchard with an average cycle time of 5.5s/fruit. Highlights: This paper has reported the design and performance of a novel robotic kiwifruit harvesting system. Detection rate of 89.6% of reachable kiwifruit with a deep network based vision system. Measurement of its in-orchard performance shows that it is capable of picking 51.0% of kiwifruit in the three test orchards. It is estimated that with further development the unit may be capable of harvesting 70% of the kiwifruit