A comparative study of existing quality measures for process discovery. (November 2017)

• Record Type: Journal Article
• Authors: Janssenswillen, Gert; Donders, Niels; Jouck, Toon; Depaire, Benoît
• Is Part Of: Information systems. Volume 71(2017)
• Page Start: 1
• Page End: 15
• Publication Date: 2017-11
• Subjects: Process discovery -- Process quality -- Conformance checking -- Process metric; Database management -- Periodicals; Electronic data processing -- Periodicals; Bases de données -- Gestion -- Périodiques; Informatique -- Périodiques; Database management; Electronic data processing; Periodicals; 005.7
• DOI: 10.1016/j.is.2017.06.002
• Languages: English
• ISSNs: 0306-4379

Abstract

Highlights: The feasibility of quality metrics for discovered processes is analysed. It is analysed whether metrics related to the same dimension agree with each other. It is analysed whether metrics are related to each other, or independent. It is analysed to which extent some metrics are more optimistic. It is analysed to which extent some metrics are more sensitive. Abstract: Evaluating the quality of discovered process models is an important task in many process mining analyses. Currently, several metrics measuring the fitness, precision and generalization of a discovered model are implemented. However, there is little empirical evidence how these metrics relate to each other, both within and across these different quality dimensions. In order to better understand these relationships, a large-scale comparative experiment was conducted. The statistical analysis of the results shows that, although fitness and precision metrics behave very similar within their dimension, some are more pessimistic while others are more optimistic. Furthermore, it was found that there is no agreement between generalization metrics. The results of the study can be used to inform decisions on which quality metrics to use in practice. Moreover, they highlight issues which give rise to new directions for future research in the area of quality measurement.