An unsupervised approach to online noisy-neighbor detection in cloud data centers. (15th December 2017)

• Record Type: Journal Article
• Authors: Lorido-Botran, Tania; Huerta, Sergio; Tomás, Luis; Tordsson, Johan; Sanz, Borja
• Is Part Of: Expert systems with applications. Volume 89(2017)
• Page Start: 188
• Page End: 204
• Publication Date: 2017-12-15
• Subjects: Anomaly detection -- Virtual machine -- Cloud computing -- DPGMM -- Noisy-neighbor effect -- Similarity distances; Expert systems (Computer science) -- Periodicals; Systèmes experts (Informatique) -- Périodiques; Electronic journals; 006.33
• DOI: 10.1016/j.eswa.2017.07.038
• Languages: English
• ISSNs: 0957-4174

Abstract

Highlights: An unsupervised, online detection algorithm for the noisy neighbor effect. It occurs when Virtual Machines compete for the same physical resources. Based on Dirichlet Process Gaussian Mixture Models for modeling resource usage. Using statistical distances (Kullback–Leibler) to measure similarity between DPGMMs. An anomaly is a higher difference between short-term models vs long-term ones. Abstract: Resource sharing is an inherent characteristic of cloud data centers. Virtual Machines (VMs) and/or Containers that are co-located in the same physical server often compete for resources leading to interference. The noisy neighbor's effect refers to an anomaly caused by a VM/container limiting resources accessed by another one. Our main contribution is an online, lightweight and application-agnostic solution for anomaly detection, that follows an unsupervised approach. It is based on comparing models for different lags: Dirichlet Process Gaussian Mixture Models to characterize the resource usage profile of the application, and distance measures to score the similarity among models. An alarm is raised when there is an abrupt change in short-term lag (i.e. high distance score for short-term models), while the long-term state remains constant. We test the algorithm for different cloud workloads: websites, periodic batch applications, Spark-based applications, and Memcached server. We are able to detect anomalies in the CPU and memory resource usage with up to 82–96% accuracy (recall) depending on the scenario. Compared to other baseline methods, our approach is able to detect anomalies successfully, while raising low number of false positives, even in the case of applications with unusual normal behavior (e.g. periodic). Experiments show that our proposed algorithm is a lightweight and effective solution to detect noisy neighbor effect without any historical info about the application, that could also be potentially applied to other kind of anomalies.