Hierarchical MapReduce: towards simplified cross‐domain data processing. (24th September 2012) - Luo, Yuan; Plale, Beth; Guo, Zhenhua; Li, Wilfred W.; Qiu, Judy; Sun, Yiming; Qiu, Judy; Foster, Ian; Taylor, Ronald; Loidl, Hans‐Wolfgang; Singer, Jeremy
SUMMARY The MapReduce programming model has proven useful for data‐driven high throughput applications. However, the conventional MapReduce model limits itself to scheduling jobs within a single cluster. As job sizes become larger, single‐cluster solutions grow increasingly inadequate. We present a hierarchical MapReduce framework that utilizes computation resources from multiple clusters simultaneously to run MapReduce job across them. The applications implemented in this framework adopt the Map–Reduce–GlobalReduce model where computations are expressed as three functions: Map, Reduce, and GlobalReduce. Two scheduling algorithms are proposed, one that targets compute‐intensive jobs and another data‐intensive jobs, evaluated using a life science application, AutoDock, and a simple Grep. Data management is explored through analysis of the Gfarm file system.Copyright © 2012 John Wiley & Sons, Ltd.