Scalable data management for map-reduce-based data-intensive applications: a view for cloud and hybrid infrastructures. (1st January 2013)

• Record Type: Journal Article
• Authors: Antoniu, Gabriel; Costan, Alexandru; Bigot, Julien; Desprez, Frédéric; Fedak, Gilles; Gault, Sylvain; Pérez, Christian; Simonet, Anthony; Tang, Bing; Blanchet, Christophe; Terreux, Raphael; Bougé, Luc; Briant, François; Cappello, Franck; Keahey, Kate; Nicolae, Bogdan; Suter, Frédéric
• Is Part Of: International journal of cloud computing. Volume 2:Number 2/3(2013)
• Page Start: 150
• Page End: 170
• Publication Date: 2013-01-01
• Subjects: map-reduce -- cloud computing -- desktop grids -- hybrid infrastructures -- bio-informatics -- task scheduling -- fault tolerance; Cloud computing -- Periodicals; 004.678205
• Languages: English
• ISSNs: 2043-9989

Abstract

As map-reduce emerges as a leading programming paradigm for data-intensive computing, today's frameworks which support it still have substantial shortcomings that limit its potential scalability. In this paper, we discuss several directions where there is room for such progress: they concern storage efficiency under massive data access concurrency, scheduling, volatility and fault-tolerance. We place our discussion in the perspective of the current evolution towards an increasing integration of large-scale distributed platforms (clouds, cloud federations, enterprise desktop grids, etc.). We propose an approach which aims to overcome the current limitations of existing map-reduce frameworks, in order to achieve scalable, concurrency-optimised, fault-tolerant map-reduce data processing on hybrid infrastructures. This approach will be evaluated with real-life bio-informatics applications on existing Nimbus-powered cloud testbeds interconnected with desktop grids.