Fight silent horror unit test methods by consulting a TestWizard. Issue 4 (23rd November 2021)

• Record Type: Journal Article
• Authors: Cerioli, Maura; Lagorio, Giovanni; Leotta, Maurizio; Ricca, Filippo
• Other Names: Bertolino Antonia guestEditor.; Hong Shin guestEditor.; Mathur Aditya P. guestEditor.
• Is Part Of: Journal of software. Volume 35:Issue 4(2023)
• Page Start: n/a
• Page End: n/a
• Publication Date: 2021-11-23
• Subjects: automated testing -- code review -- false‐negative unit tests -- test code -- test quality -- test specifications; Software engineering -- Periodicals; Computer software -- Development -- Periodicals; Software maintenance -- Periodicals; 005.1
• DOI: 10.1002/smr.2396
• Languages: English
• ISSNs: 2047-7473

Abstract

Abstract: Tests, when not correctly implemented, can pass on incorrect system implementations rather than fail. In this case, they are named silent horrors or false‐negative tests. They make releasing low‐quality (buggy) versions of the software system more probable. Furthermore, faithfully implementing test specifications is crucial when they play the role of documentation, like when documenting components or services or driving legacy systems' re‐engineering. This paper presents TestWizard, a novel approach and tool for automatically assessing individual tests' quality from the point of view of their coherence to specifications. TestWizard automatically assesses the quality of each individual test case w.r.t. its specification, providing detailed reports on why a single test is a false negative, hence helping testers fix them. Thus, TestWizard can help to automate the test code review process, which is still mainly manual today. The analysis of 1012 test implementations, developed by 123 students in three experiments, shows that TestWizard is (1) by far more accurate than code review performed by multiple students, (2) slightly better than code review performed by three senior experts, and (3) always able to detect a significant percentage of false‐negative test methods (up to 21.22%). Abstract :