Assessing expert system-assisted literature reviews with a case study. (15th August 2022)

• Record Type: Journal Article
• Authors: Yu, Zhe; Carver, Jeffrey C.; Rothermel, Gregg; Menzies, Tim
• Is Part Of: Expert systems with applications. Volume 200(2022)
• Publication Date: 2022-08-15
• Subjects: Systematic literature review -- Expert systems -- Software engineering -- Active learning -- Primary study selection -- Test case prioritization; Expert systems (Computer science) -- Periodicals; Systèmes experts (Informatique) -- Périodiques; Electronic journals; 006.33
• DOI: 10.1016/j.eswa.2022.116958
• Languages: English
• ISSNs: 0957-4174

Abstract

Abstract: Given the large numbers of publications in software engineering, frequent literature reviews are required to keep current on work in specific areas. One tedious work in literature reviews is to find relevant studies amongst thousands of non-relevant search results. In theory, expert systems can assist in finding relevant work but those systems have primarily been tested in simulations rather than in application to actual literature reviews. Hence, few researchers have faith in such expert systems. Accordingly, using a realistic case study, this paper assesses how well our state-of-the-art expert system can help with literature reviews. The assessed literature review aimed at identifying test case prioritization techniques for automated UI testing, specifically from 8, 349 papers on IEEE Xplore. This corpus was studied with an expert system that incorporates an incrementally updated human-in-the-loop active learning tool. Using that expert system, in three hours, we found 242 relevant papers from which we identified 12 techniques representing the state-of-the-art in test case prioritization when source code information is not available. These results were then validated by six other graduate students manually exploring the same corpus. Without the expert system, this task would have required 53 h and would have found 27 additional papers. That is, our expert system achieved 90% recall with 6% of the human effort cost when compared to a conventional manual method. Significantly, the same 12 state-of-the-art test case prioritization techniques were identified by both the expert system and the manual method. That is, the 27 papers missed by the expert system would not have changed the conclusion of the literature review. Hence, if this result generalizes, it endorses the use of our expert system to assist in literature reviews. Highlights: First paper reporting a literature review assisted by FAST2. FAST2 saved 50 man-hours of human effort while missing 10% relevant papers. The missing papers do not change the conclusions of the literature review. FAST2 imposed a bias onto the distribution of the included papers. The literature review process was scripted online for reproduction and experiments.