Choosing the fitness function for the job: Automated generation of test suites that detect real faults. (18th June 2019)

• Record Type: Journal Article
• Authors: Salahirad, Alireza; Almulla, Hussein; Gay, Gregory
• Is Part Of: Software testing, verification & reliability. Volume 29:Number 4/5(2019)
• Page Start: n/a
• Page End: n/a
• Publication Date: 2019-06-18
• Subjects: search‐based test generation -- automated test generation -- unit testing -- adequacy criteria -- search‐based software engineering; Computer software -- Testing -- Periodicals; Computer software -- Verification -- Periodicals; Computer software -- Reliability -- Periodicals; 005.14
• DOI: 10.1002/stvr.1701
• Languages: English
• ISSNs: 0960-0833

Abstract

Summary: Search‐based unit test generation, if effective at fault detection, can lower the cost of testing. Such techniques rely on fitness functions to guide the search. Ultimately, such functions represent test goals that approximate—but do not ensure—fault detection. The need to rely on approximations leads to two questions— can fitness functions produce effective tests and, if so, which should be used to generate tests? To answer these questions, we have assessed the fault‐detection capabilities of unit test suites generated to satisfy eight white‐box fitness functions on 597 real faults from the Defects4J database. Our analysis has found that the strongest indicators of effectiveness are a high level of code coverage over the targeted class and high satisfaction of a criterion's obligations. Consequently, the branch coverage fitness function is the most effective. Our findings indicate that fitness functions that thoroughly explore system structure should be used as primary generation objectives—supported by secondary fitness functions that explore orthogonal, supporting scenarios. Our results also provide further evidence that future approaches to test generation should focus on attaining higher coverage of private code and better initialization and manipulation of class dependencies. Abstract : Search‐based test generation relies on fitness functions. Which are most effective? Our analysis found that the strongest indicators of effectiveness are a high level of code coverage and high fitness. Our findings indicate that fitness functions that explore system structure should be used as primary generation objectives‐supported by secondary functions that explore targeted scenarios. Our results provide evidence that future approaches should focus on higher coverage of private code and initialization and manipulation of class dependencies.