Data‐driven benchmarking in software development effort estimation: The few define the bulk. Issue 9 (3rd March 2020)

• Record Type: Journal Article
• Authors: Mittas, Nikolaos; Angelis, Lefteris
• Is Part Of: Journal of software. Volume 32:Issue 9(2020)
• Page Start: n/a
• Page End: n/a
• Publication Date: 2020-03-03
• Subjects: archetypal analysis -- benchmarks -- performance measures -- software development effort estimation; Software engineering -- Periodicals; Computer software -- Development -- Periodicals; Software maintenance -- Periodicals; 005.1
• DOI: 10.1002/smr.2258
• Languages: English
• ISSNs: 2047-7473

Abstract

Abstract: Context: The rapid evolvement of software development effort estimation models created the need for empirical evaluation of their quality. The empirical evaluation is based either on hypothesis tests with respect to a single criterion or on aggregating methods for multiple criteria. However, a model can be considered as a multidimensional entity performing differently on alternative datasets and its performance can be divergent when expressed by alternative criteria. Objective: In this study, we explore this multidimensional nature of models by considering them as points in two different spaces (domain and criteria spaces). Method: Introducing an alternative approach for data‐driven benchmarking, a new framework based on archetypal analysis is proposed for evaluation purposes of multiple models. Results: The benefits of the framework are illustrated through a large‐scale experimental setup on a set of 93 effort estimation models, trained and tested on 10 datasets under 8 criteria providing answers to critical research questions. Conclusion: The results indicate that a small minority of reference models is enough to define the performance of the bulk of all models. The framework focuses on models that have behavior close to archetypes and especially those that are close to a "best" archetype. Abstract : In this paper, we propose a data‐driven approach for the evaluation of the multifaceted nature of SDEE models' performance summarized in a three‐step process: (a) exploration of all feasible solutions for the extraction of few reference profiles, (b) characterization of the reference profiles, and (c) evaluation of candidate models in the basis of the extracted reference profiles. The empirical findings indicate that a small minority of reference models is enough to define the performance of the bulk of all models.