RVprio: A tool for prioritizing runtime verification violations. (7th March 2022)

• Record Type: Journal Article
• Authors: Cabral, Lucas; Miranda, Breno; Lima, Igor; d'Amorim, Marcelo
• Other Names: Păsăreanu Corina guestEditor.; Zeller Andreas guestEditor.
• Is Part Of: Software testing, verification & reliability. Volume 32:Number 5(2022)
• Page Start: n/a
• Page End: n/a
• Publication Date: 2022-03-07
• Subjects: prioritization -- runtime verification -- testing; Computer software -- Testing -- Periodicals; Computer software -- Verification -- Periodicals; Computer software -- Reliability -- Periodicals; 005.14
• DOI: 10.1002/stvr.1813
• Languages: English
• ISSNs: 0960-0833

Abstract

Summary: Runtime verification (RV) helps to find software bugs by monitoring formally specified properties during testing. A key problem in using RV during testing is how to reduce the manual inspection effort for checking whether property violations are true bugs. To date, there was no automated approach for determining the likelihood that property violations were true bugs to reduce tedious and time‐consuming manual inspection. We present RVprio, the first automated approach for prioritizing RV violations in order of likelihood of being true bugs. RVprio uses machine learning classifiers to prioritize violations. For training, we used a labelled dataset of 1170 violations from 110 projects. On that dataset, (1) RVprio reached 90% of the effectiveness of a theoretically optimal prioritizer that ranks all true bugs at the top of the ranked list, and (2) 88.1% of true bugs were in the top 25% of RVprio ‐ranked violations; 32.7% of true bugs were in the top 10%. RVprio was also effective when we applied it to new unlabelled violations, from which we found previously unknown bugs—54 bugs in 8 open‐source projects. Our dataset is publicly available online. Abstract : This paper proposes RVPRIO, an automated approach for reducing developer's manual inspection effort by prioritizing violations in order of likelihood of these violations being true bugs. RVPRIO achieves 90% of the effectiveness of a theoretically optimal prioritizer that ranks all true bugs at the top. By focusing on the top 25% of the RVPRIO prioritized list, the developer would have captured 88.1% of the true bugs, saving 75% of the inspection time at the expense of missing 11.9% of true bugs.