The development of a low-cost big data cluster using Apache Hadoop and Raspberry Pi. A complete guide. (December 2022) - Neto, Antônio José Alves; Neto, José Aprígio Carneiro; Moreno, Edward David
Abstract: This paper provides a complete guide to the development, testing, and monitoring of a low-cost big data cluster through a detailed step-by-step configuration and installation of Apache Hadoop using 9 Raspberry Pis 4B. For the tests and performance evaluation, were used the Terasort and TestDFSIO benchmarks. The benchmarks were performed in different sizes of data files (250 MB up to 1 GB) and different slaves nodes quantity (2, 4, and 8). The results showed that the combination of Raspberry Pi and Apache Hadoop can be a very efficient and robust solution to get a low-cost big data cluster, considering its costs/benefits. Using a Raspberry Pi 3B+ as a monitoring server, we installed the Zabbix and Grafana tools, making it possible to collect important information in real-time, helping to better monitoring of the cluster's devices and better visualization of the behavior and performance of the cluster. Graphical abstract: Highlights: Development of a low-cost big data cluster using Apache Hadoop and Raspberry Pi 4B. Detailed step-by-step to guide the cluster development. Cluster evaluation using the Terasort and TestDFSIO Benchmarks. Cluster monitoring using a Raspberry Pi 3B+ as monitoring server with Zabbix and Grafana tools.