2-D/3-D fusion-based robust pose normalisation of 3-D livestock from multiple RGB-D cameras. (November 2022)

• Record Type: Journal Article
• Authors: Lu, Jie; Guo, Hao; Du, Ao; Su, Yang; Ruchay, Alexey; Marinello, Francesco; Pezzuolo, Andrea
• Is Part Of: Biosystems engineering. Volume 223(2022)Part B
• Page Start: 129
• Page End: 141
• Publication Date: 2022-11
• Subjects: Precision livestock farming -- Body measurement -- 2-D/3-D fusion -- Pose normalisation -- point clouds; Bioengineering -- Periodicals; Agricultural engineering -- Periodicals; Biological systems -- Periodicals; Génie rural -- Périodiques; Systèmes biologiques -- Périodiques; 631
• DOI: 10.1016/j.biosystemseng.2021.12.013
• Languages: English
• ISSNs: 1537-5110

Abstract

Abstract : Recently, 3-D scanning data are becoming increasingly important for precision livestock farming. In particular, RGB-D (red, green, blue – depth) data of livestock have come to play a critical role in the field of livestock body measurement. However, the latest livestock pose normalisation methods rely on purely 3-D geometric data and are therefore prone to errors due to noise and missing data. To achieve adequate performance, particularly for different livestock species in practical applications a 2-D/3-D fusion-based robust livestock pose normalisation method is proposed. Firstly, based on advanced 2-D object detection techniques, the proposed approach makes the best use of 2-D information to determine the accurate orientation of livestock in 3-D. Secondly, the 2-D detection results are used to generate frustums in 3-D space to locate livestock targets, which markedly reduces the search space and improves segmentation. Finally, based on a bilateral symmetry-based pose normalisation framework, a more robust pose normalisation algorithm is applied. Compared to existing pose normalization methods that operate in 3-D, extensive experiments with multiple view RGB-D data of livestock show that the proposed method is more robust and practical than existing methods. The proposed algorithm provides pose normalisation in an automatic body measurement system for livestock. This study proposes that the idea that 2-D/3-D fusion-based strategies in 3-D should be explored in more detail, particularly for cases in which the 3-D input captured by consumer designed RGB-D cameras which are often noisy and miss values at certain pixels. All the training databases and codes used in the study can be downloaded freely. Highlights: Optimised automatic pose normalisation framework is proposed. New method of 2D object detection fused with 3D point cloud technology. New method shown to enhance pose normalisation in pigs and cattle. Improved livestock segmentation and estimation of forward direction. Proposed normalisation approach is freely available for PLF users.