Hierarchical MapReduce: towards simplified cross‐domain data processing. (24th September 2012)

• Record Type: Journal Article
• Authors: Luo, Yuan; Plale, Beth; Guo, Zhenhua; Li, Wilfred W.; Qiu, Judy; Sun, Yiming; Qiu, Judy; Foster, Ian; Taylor, Ronald; Loidl, Hans‐Wolfgang; Singer, Jeremy
• Is Part Of: Concurrency and computation. Volume 26:Number 4(2014:Mar.)
• Page Start: 878
• Page End: 893
• Publication Date: 2012-09-24
• Subjects: Parallel processing (Electronic computers) -- Periodicals; Parallel computers -- Periodicals; 004.35
• DOI: 10.1002/cpe.2929
• Languages: English
• ISSNs: 1532-0626

Abstract

<abstract abstract-type="main" id="cpe2929-abs-0001"> <title>SUMMARY</title> <p id="cpe2929-para-0001">The MapReduce programming model has proven useful for data‐driven high throughput applications. However, the conventional MapReduce model limits itself to scheduling jobs within a single cluster. As job sizes become larger, single‐cluster solutions grow increasingly inadequate. We present a hierarchical MapReduce framework that utilizes computation resources from multiple clusters simultaneously to run MapReduce job across them. The applications implemented in this framework adopt the <italic>Map–Reduce–GlobalReduce</italic> model where computations are expressed as three functions: Map, Reduce, and GlobalReduce. Two scheduling algorithms are proposed, one that targets compute‐intensive jobs and another data‐intensive jobs, evaluated using a life science application, AutoDock, and a simple Grep. Data management is explored through analysis of the Gfarm file system.Copyright © 2012 John Wiley &amp; Sons, Ltd.</p> </abstract>