Cloud agnostic Big Data platform focusing on scalability and cost-efficiency. (November 2018)

• Record Type: Journal Article
• Authors: Lovas, Róbert; Nagy, Enikő; Kovács, József
• Is Part Of: Advances in engineering software. Volume 125(2018)
• Page Start: 167
• Page End: 177
• Publication Date: 2018-11
• Subjects: Cloud computing -- Big Data -- Orchestration -- Hadoop -- Occopus -- Contextualisation -- Scalability -- Cost analysis; Computer-aided engineering -- Periodicals; Engineering -- Computer programs -- Periodicals; Engineering -- Software -- Periodicals; Periodicals; 620.0028553
• DOI: 10.1016/j.advengsoft.2018.05.002
• Languages: English
• ISSNs: 0965-9978

Abstract

Highlights: There is a growing demand for easily deployable big data platform supporting the execution of map-reduce applications. Apache Hadoop cluster powered by the Occopus cloud orchestrator tool implements an open-source, cloud-agnostic solution. Combining with service discovery and monitoring tools auto-scaling has been also realised to dynamise resource capacity. Based on cost analysis, the solution leads to cost reduction in certain use cases compared to commercial solutions. Short-term and long-term deployments as well as fine-tuning configuration are supported. Abstract: Nowadays a significant part of the cloud applications processes a large amount of data to provide the desired analytics, simulation and other results. Cloud computing is becoming a widely used IT model to address the needs of many scientific and commercial Big Data applications. In this paper, we present a Hadoop platform deployment method for various cloud infrastructures with the Occopus cloud orchestrator tool. Our automated solution provides an easy-to-use, portable and scalable way to deploy the popular Hadoop platform with the main goal to avoid vendor locking issues, i.e. there is no dependency on any cloud provider prepared and offered virtual machine image or "black-box" Platform-as-a-Service mechanism. The paper presents promising performance measurements results and cost analysis.