A transparent approach to performance analysis and comparison of infrastructure as a service providers. (July 2018) - Muhammad-Bello, Bilkisu Larai; Aritsugi, Masayoshi
Highlights: A transparent methodology for comparing public cloud virtual machines is proposed. Established benchmarks are used to obtain sufficiently informative performance evaluation of public cloud virtual machines. Empirical results show significant performance differences for comparable instances on the public cloud. In depth performance/cost analysis and comparison via the devised Cloud Provider Value (CPV) metric gives deeper insights. Workload correlation of the adapted benchmarks' workloads with three real analytics application workloads. Abstract: This study proposes a transparent approach to performance analysis and comparison of Infrastructure as a Service (IaaS) providers. Using established benchmarks, we focused on obtaining deeper insights into the performance of the public cloud infrastructure as reflected by the CPU, memory and disk I/O subsystems. We conducted experiments on a real public cloud environment and demonstrated how prospective cloud users can use our transparent methodology to discover how well virtualized public cloud resources meet their application requirements. Our transparent approach is unique in the sense that it helps prospective cloud users to decipher public cloud benchmarking data and appraise the performance of public cloud services relative to their application's performance objectives. Furthermore, we show workload correlations to the benchmarks using three real-life analytics applications. Our empirical results show significant performance differences for comparable instances on the public cloud, underscoring the need for thoughtful and transparent IaaS provider selection.