Anomaly detection in performance regression testing by transaction profile estimation. (9th March 2015)

• Record Type: Journal Article
• Authors: Ghaith, Shadi; Wang, Miao; Perry, Philip; Jiang, Zhen Ming; O'Sullivan, Pat; Murphy, John
• Is Part Of: Software testing, verification & reliability. Volume 26:Number 1(2016)
• Page Start: 4
• Page End: 39
• Publication Date: 2015-03-09
• Subjects: software update -- performance models -- performance regression testing; Computer software -- Testing -- Periodicals; Computer software -- Verification -- Periodicals; Computer software -- Reliability -- Periodicals; 005.14
• DOI: 10.1002/stvr.1573
• Languages: English
• ISSNs: 0960-0833

Abstract

Summary: As part of the process to test a new release of an application, the performance testing team need to confirm that the existing functionalities do not perform worse than those in the previous release, a problem known as performance regression anomaly. Most existing approaches to analyse performance regression testing data vary according to the applied workload, which usually leads to the need for an extra performance testing run. To ease such lengthy tasks, we propose a new workload‐independent, automated technique to detect anomalies in performance regression testing data using the concept known as transaction profile (TP). The TP is inferred from the performance regression testing data along with the queueing network model of the testing system. Based on a case study conducted against two web applications, one open source and one industrial, we have been able to automatically generate the 'TP run report' and verify that it can be used to uncover performance regression anomalies caused by software updates. In particular, the report helped us to isolate the real anomaly issues from those caused by workload changes with an average F1 measure of 85% for the open source application and 90% for the industrial application. Such results support our proposal to use the TP as a more efficient technique in identifying performance regression anomalies than the state of the art industry and research techniques. Copyright © 2015 John Wiley & Sons, Ltd. Abstract : We showed how the transaction profile (TP) can be used, via the TP run report, by regression testing teams as an automated way to discover performance regression anomalies. Furthermore, the TP is immune to changes in the load applied to the system, which will save time to repeat runs with the previous release load. Moreover, we proposed a novel way to infer the TP from already available performance data, mainly the resources utilizations and the transaction response times, and the queueing network model of the testing system.