A HDFS dynamic load balancing strategy using improved niche PSO algorithm in cloud storage. (7th April 2021)

• Record Type: Journal Article
• Authors: Jian, Zhiyu; Jian, Yiwei
• Is Part Of: International journal of autonomous and adaptive communications systems. Volume 14:Number 1/2(2021)
• Page Start: 163
• Page End: 178
• Publication Date: 2021-04-07
• Subjects: cloud storage -- urban surveillance video -- improved niche -- PSO -- particle swarm optimisation -- dynamic load balancing -- big data -- HDFS -- Hadoop distributed file system; Adaptive computing systems -- Periodicals; Wireless communication systems -- Periodicals; Computer networks -- Periodicals; 004.6
• Languages: English
• ISSNs: 1754-8632

Abstract

A Hadoop distributed file system (HDFS) NameNode dynamic load balancing strategy (NDLBT) using improved niche particle swarm optimisation (PSO) is proposed, which is used to solve the load balancing problem of urban surveillance video big data. Firstly, the relationship between bandwidth consumption and the bit rate of video files, the size of data blocks and access hotspots of online video files are analysed. Then, adaptive backup is realised by dynamic multiple copies of heterogeneous nodes. The dynamic distribution of metadata is implemented to ensure the performance of metadata server clusters. Finally, we propose a new improved niche PSO algorithm to achieve load balancing scheduling. In the experimental scenario where high bandwidth consumption video files are used as service access hotspots, the proposed method, which can reduce the bandwidth peak value of bottleneck nodes in data node clusters by 20%, is superior to the original load balancing method in 90% scenarios.