Multi-source streaming-based data accesses for MapReduce systems. (1st January 2014)

• Record Type: Journal Article
• Authors: Wu, Jiadong; Hong, Bo
• Is Part Of: International journal of big data intelligence. Volume 1:Number 1/2(2014)
• Page Start: 36
• Page End: 49
• Publication Date: 2014-01-01
• Subjects: MapReduce -- multi-source streaming -- remote data access; Big data -- Periodicals; 005.705
• Languages: English
• ISSNs: 2053-1389

Abstract

The MapReduce programming model, along with its open-source implementation – Hadoop – has provided a cost effective solution for many data intensive applications. Hadoop stores data distributively and exploits data locality by assigning tasks to where data is stored. In many cases, however, accessing remote data (rack-local and off-rack) is inevitable. In this paper we are evaluating the possibility of improving the remote data accessing performance by streaming data from multiple available replicas. The proposed design consists of a circular buffer, a slice reader and an enhanced DataNode. Such system is capable of adapting to both the static performance variance caused by network topology as well as dynamic variance caused by congestion. Extensive experiments show that multisource streaming can significantly improve the throughput of remote data access and accelerate the related map tasks by 10%–20%. For systems with heterogenous network links, upto 4× speedup was observed.