A transparent approach to performance analysis and comparison of infrastructure as a service providers. (July 2018)

• Record Type: Journal Article
• Authors: Muhammad-Bello, Bilkisu Larai; Aritsugi, Masayoshi
• Is Part Of: Computers & electrical engineering. Volume 69(2018)
• Page Start: 317
• Page End: 333
• Publication Date: 2018-07
• Subjects: Cloud performance analysis -- Workload correlations -- Cloud service selection -- Cloud benchmarking; Computer engineering -- Periodicals; Electrical engineering -- Periodicals; Electrical engineering -- Data processing -- Periodicals; Ordinateurs -- Conception et construction -- Périodiques; Électrotechnique -- Périodiques; Électrotechnique -- Informatique -- Périodiques; Computer engineering; Electrical engineering; Electrical engineering -- Data processing; Periodicals; Electronic journals; 621.302854
• DOI: 10.1016/j.compeleceng.2017.12.034
• Languages: English
• ISSNs: 0045-7906

Abstract

Highlights: A transparent methodology for comparing public cloud virtual machines is proposed. Established benchmarks are used to obtain sufficiently informative performance evaluation of public cloud virtual machines. Empirical results show significant performance differences for comparable instances on the public cloud. In depth performance/cost analysis and comparison via the devised Cloud Provider Value (CPV) metric gives deeper insights. Workload correlation of the adapted benchmarks' workloads with three real analytics application workloads. Abstract: This study proposes a transparent approach to performance analysis and comparison of Infrastructure as a Service (IaaS) providers. Using established benchmarks, we focused on obtaining deeper insights into the performance of the public cloud infrastructure as reflected by the CPU, memory and disk I/O subsystems. We conducted experiments on a real public cloud environment and demonstrated how prospective cloud users can use our transparent methodology to discover how well virtualized public cloud resources meet their application requirements. Our transparent approach is unique in the sense that it helps prospective cloud users to decipher public cloud benchmarking data and appraise the performance of public cloud services relative to their application's performance objectives. Furthermore, we show workload correlations to the benchmarks using three real-life analytics applications. Our empirical results show significant performance differences for comparable instances on the public cloud, underscoring the need for thoughtful and transparent IaaS provider selection.