How detrimental is coincidental correctness to coverage‐based fault detection and localization? An empirical study. (9th January 2021)

• Record Type: Journal Article
• Authors: Abou Assi, Rawad; Masri, Wes; Trad, Chadi
• Is Part Of: Software testing, verification & reliability. Volume 31:Number 5(2021)
• Page Start: n/a
• Page End: n/a
• Publication Date: 2021-01-09
• Subjects: coincidental correctness -- defect detection -- failed error propagation -- fault masking -- spectrum‐based fault localization -- test case prioritization -- test suite reduction/minimization; Computer software -- Testing -- Periodicals; Computer software -- Verification -- Periodicals; Computer software -- Reliability -- Periodicals; 005.14
• DOI: 10.1002/stvr.1762
• Languages: English
• ISSNs: 0960-0833

Abstract

Abstract: According to the reachability–infection–propagation (RIP) model, three conditions must be satisfied for program failure to occur: (1) the defect's location must be r eached, (2) the program's state must become i nfected and (3) the infection must p ropagate to the output. Weak coincidental correctness (or weak CC ) occurs when the program produces the correct output, while condition (1) is satisfied but conditions (2) and (3) are not satisfied. Strong coincidental correctness (or strong CC ) occurs when the output is correct, while both conditions (1) and (2) are satisfied but not (3). The prevalence of CC was previously recognized. In addition, the potential for its negative effect on spectrum‐based fault localization (SBFL) was analytically demonstrated; however, this was not empirically validated. Using Defects4J, this paper empirically studies the impact of weak and strong CC on three well‐researched coverage‐based fault detection and localization techniques, namely, test suite reduction (TSR), test case prioritization (TCP) and SBFL. Our study, which involved 52 SBFL metrics, provides the following empirical evidence. (i) The negative impact of CC tests on TSR and TCP is very significant. In addition, cleansing the CC tests was observed to yield (a) a 100% TSR defect detection rate for all subject programs and (b) an improvement of TCP for over 92% of the subjects. (ii) The impact of CC tests on SBFL varies widely w.r.t. the metric used. The negative impact was strong for 11 metrics, mild for 37, non‐measurable for 1 and non‐existent for 3 metrics. Interestingly, the negative impact was mild for the 9 most popular and/or most effective SBFL metrics. In addition, cleansing the CC tests resulted in the deterioration of SBFL for a considerable number of subject programs. (iii) Increasing the proportion of CC tests has a limited impact on TSR, TCP and SBFL. Interestingly, for TSR and TCP and 11 SBFL metrics, small and large proportions of CC tests are strongly harmful. (iv) Lastly, weak and strong CC are equally detrimental in the context of TSR, TCP and SBFL.