Spark-based adaptive Mapreduce data processing method for remote sensing imagery. Issue 1 (2nd January 2021)

• Record Type: Journal Article
• Authors: Tan, Xicheng; Di, Liping; Zhong, Yanfei; Yao, Yayu; Sun, Ziheng; Ali, Yahya
• Is Part Of: International journal of remote sensing. Volume 42:Issue 1(2021)
• Page Start: 191
• Page End: 207
• Publication Date: 2021-01-02
• Subjects: Remote sensing -- Periodicals; Télédétection -- Périodiques; 621.3678
• DOI: 10.1080/01431161.2020.1804087
• Languages: English
• ISSNs: 0143-1161

Abstract

ABSTRACT: Existing Hadoop-based remote sensing data processing approaches are insufficient for efficiently meeting the requirements of applications, especially when large remote sensing datasets are involved. This paper proposes an adaptive Spark-based remote sensing data processing method on the cloud that achieves improved efficiency and stability. The method includes a remote sensing data storage scheme on the cloud that employs the Hadoop Distributed File System (HDFS) and adaptive MapReduce mechanisms for use with remote sensing data; specifically, a mapping strategy for use with image tiles, a reducing strategy for use with adjacent tiles, and a mechanism for merging the results are proposed. An image classification experiment is conducted using Land Remote-Sensing Satellite System (Landsat) Thematic Mapper (TM) data, and the proposed method displays improved performance, stability and scalability compared to the existing Hadoop-based method. Hence, the proposed method is more suitable for processing large volumes of remote sensing data.