An efficient attribute reduction algorithm using MapReduce. (February 2021)

• Record Type: Journal Article
• Authors: Yin, Linzi; Li, Jing; Jiang, Zhaohui; Ding, Jiafeng; Xu, Xuemei
• Is Part Of: Journal of information science. Volume 47:Number 1(2021)
• Page Start: 101
• Page End: 117
• Publication Date: 2021-02
• Subjects: Attribute reduction -- MapReduce -- rough set -- shuffle mechanism -- sort technology; Information science -- Periodicals; Information science; Periodicals; 020.5
• DOI: 10.1177/0165551519874617
• Languages: English
• ISSNs: 0165-5515

Abstract

Classical attribute reduction algorithms based on attribute significance initiate too many jobs ( O (| C | 2 )) when they run in MapReduce. To improve the efficiencies of these algorithms, we proposed a novel reduction algorithm. Instead of focusing on attribute significance, the notion of a core attribute was applied to construct a new heuristic reduction algorithm, and only | C | jobs were considered to obtain a reduct. The algorithm only included two basic operations: compare and sort. The latter was optimised using the shuffle mechanism in MapReduce, which provided an efficient sorting ability for big data. In particular, we connected jobs in an iterative form to transfer the processing result of the former job to the latter job. Finally, experimental results demonstrated that the proposed attribute reduction algorithm was efficient and significantly improved upon the classical algorithms in runtime and number of jobs.