Detecting and analyzing I/O performance regressions. Issue 12 (17th July 2014)

• Record Type: Journal Article
• Authors: Bezemer, C.; Milon, E.; Zaidman, A.; Pouwelse, J.
• Is Part Of: Journal of software. Volume 26:Issue 12(2014:Dec.)
• Page Start: 1193
• Page End: 1212
• Publication Date: 2014-07-17
• Subjects: Software engineering -- Periodicals; Computer software -- Development -- Periodicals; Software maintenance -- Periodicals; 005.1
• DOI: 10.1002/smr.1657
• Languages: English
• ISSNs: 2047-7473

Abstract

<abstract abstract-type="main"> <title>SUMMARY</title> <p>Regression testing can be performed by reexecuting a test suite on different software versions and comparing the outcome. For functional testing, the outcome of such tests is either pass (correct behavior) or fail (incorrect behavior). For nonfunctional testing, such as performance testing, this is more challenging as correct and incorrect are not clearly defined concepts for these types of testing.</p> <p>In this paper, we present an approach for detecting and analyzing input/output (I/O) performance regressions. Our method is supplemental to existing profilers, and its goal is to analyze the effect of source code changes on the performance of a system. In this paper, we focus on analyzing the amount of I/O writes being performed. The open source implementation of our approach, <sc>SPECTRAPERF</sc>, is available for download. We evaluate our approach in a field user study on Tribler, an open source peer‐to‐peer client and its decentralized solution for synchronizing messages, Dispersy. In this evaluation, we show that our approach can guide the performance optimization process, as it helps developers to find performance bottlenecks on the one hand and, on the other, allows them to validate the effect of performance optimizations. In addition, we perform a feasibility study on Django, the most popular Python project on Github, to demonstrate our applicability on other projects. Copyright © 2014 John Wiley & Sons, Ltd.</p> </abstract>