Test suite completeness and black box testing. (9th January 2017)

• Record Type: Journal Article
• Authors: Bonifácio, Adilson Luiz; Moura, Arnaldo Vieira
• Is Part Of: Software testing, verification & reliability. Volume 27:Number 1/2(2017)
• Page Start: n/a
• Page End: n/a
• Publication Date: 2017-01-09
• Subjects: completeness -- isomorphism -- perfectness -- test suite properties; Computer software -- Testing -- Periodicals; Computer software -- Verification -- Periodicals; Computer software -- Reliability -- Periodicals; 005.14
• DOI: 10.1002/stvr.1626
• Languages: English
• ISSNs: 0960-0833

Abstract

Summary: Model‐based testing has been widely studied and successfully applied to generate and verify completeness of test suites. Roughly, test suite completeness guarantees that, in a deterministic setting, any nonequivalent implementation under test will always be identified. Several approaches discuss sufficient, and sometimes also necessary, conditions on specification models and test suites to be able to guarantee test suite completeness. In these studies, usually, test cases are required to be nonblocking—that is, they are required to run to completion—on both the specification and the implementation models. However, often, it is desirable to have blocking test cases, and in some situations the presence of blocking test cases cannot be circumvented. In the present work test cases are allowed to block, both in the specification and in the implementation models, and a natural variant of completeness is then studied, here called perfectness. Perfectness guarantees that noncompliance between a specification and an implementation will always be detected, even in the presence of blocking test cases. This work characterizes perfectness in isomorphisms and establishes a relationship between the classical notions of completeness and perfectness. Also, a sharp upper bound is given on the number of states in implementations, beyond which no test suite can be completed. Abstract : This work allows for test cases to block giving rise to a natural variant of completeness called perfectness. Perfectness guarantees that non‐compliance between a specification and an implementation will always be detected, even in the presence of blocking test cases. This study characterizes perfectness by isomorphisms, and establishes a relationship between the notion of completeness and perfectness. Further, a sharp upper bound is given on the number of states in implementations, beyond which no test suite can be complete.