The development of a low-cost big data cluster using Apache Hadoop and Raspberry Pi. A complete guide. (December 2022)

• Record Type: Journal Article
• Authors: Neto, Antônio José Alves; Neto, José Aprígio Carneiro; Moreno, Edward David
• Is Part Of: Computers & electrical engineering. Volume 104:Part A(2022)
• Publication Date: 2022-12
• Subjects: Apache Hadoop -- Big data -- Cluster -- Grafana -- Raspberry Pi -- Step-by-step -- Terasort -- TestDFSIO -- Zabbix; Computer engineering -- Periodicals; Electrical engineering -- Periodicals; Electrical engineering -- Data processing -- Periodicals; Ordinateurs -- Conception et construction -- Périodiques; Électrotechnique -- Périodiques; Électrotechnique -- Informatique -- Périodiques; Computer engineering; Electrical engineering; Electrical engineering -- Data processing; Periodicals; Electronic journals; 621.302854
• DOI: 10.1016/j.compeleceng.2022.108403
• Languages: English
• ISSNs: 0045-7906

Abstract

Abstract: This paper provides a complete guide to the development, testing, and monitoring of a low-cost big data cluster through a detailed step-by-step configuration and installation of Apache Hadoop using 9 Raspberry Pis 4B. For the tests and performance evaluation, were used the Terasort and TestDFSIO benchmarks. The benchmarks were performed in different sizes of data files (250 MB up to 1 GB) and different slaves nodes quantity (2, 4, and 8). The results showed that the combination of Raspberry Pi and Apache Hadoop can be a very efficient and robust solution to get a low-cost big data cluster, considering its costs/benefits. Using a Raspberry Pi 3B+ as a monitoring server, we installed the Zabbix and Grafana tools, making it possible to collect important information in real-time, helping to better monitoring of the cluster's devices and better visualization of the behavior and performance of the cluster. Graphical abstract: Highlights: Development of a low-cost big data cluster using Apache Hadoop and Raspberry Pi 4B. Detailed step-by-step to guide the cluster development. Cluster evaluation using the Terasort and TestDFSIO Benchmarks. Cluster monitoring using a Raspberry Pi 3B+ as monitoring server with Zabbix and Grafana tools.