A Review of the Metrics Used to Assess Auto-Contouring Systems in Radiotherapy. Issue 6 (June 2023)

• Record Type: Journal Article
• Authors: Mackay, K.; Bernstein, D.; Glocker, B.; Kamnitsas, K.; Taylor, A.
• Is Part Of: Clinical oncology. Volume 35:Issue 6(2023)
• Page Start: 354
• Page End: 369
• Publication Date: 2023-06
• Subjects: Artificial intelligence -- Auto-contour -- Auto-segmentation -- Deep learning -- Evaluation -- Radiotherapy; AI Artificial Intelligence -- STAPLE Simultaneous Truth and Performance Level Estimation -- DVH Dose Volume Histogram -- AAPM American Association of Physicists in Medicine -- CTV Clinical Target Volume -- PTV Planning Target Volume; Oncology -- Periodicals; Tumors -- Periodicals; Cancer -- Treatment -- Periodicals; Radiotherapy -- Periodicals; Neoplasms -- Periodicals; Cancer -- Radiotherapy; Cancer -- Treatment; Oncology; Medical radiology; Radiotherapy; Tumors; Electronic journals; Periodicals; 616.994
• DOI: 10.1016/j.clon.2023.01.016
• Languages: English
• ISSNs: 0936-6555

Abstract

Abstract: Auto-contouring could revolutionise future planning of radiotherapy treatment. The lack of consensus on how to assess and validate auto-contouring systems currently limits clinical use. This review formally quantifies the assessment metrics used in studies published during one calendar year and assesses the need for standardised practice. A PubMed literature search was undertaken for papers evaluating radiotherapy auto-contouring published during 2021. Papers were assessed for types of metric and the methodology used to generate ground-truth comparators. Our PubMed search identified 212 studies, of which 117 met the criteria for clinical review. Geometric assessment metrics were used in 116 of 117 studies (99.1%). This includes the Dice Similarity Coefficient used in 113 (96.6%) studies. Clinically relevant metrics, such as qualitative, dosimetric and time-saving metrics, were less frequently used in 22 (18.8%), 27 (23.1%) and 18 (15.4%) of 117 studies, respectively. There was heterogeneity within each category of metric. Over 90 different names for geometric measures were used. Methods for qualitative assessment were different in all but two papers. Variation existed in the methods used to generate radiotherapy plans for dosimetric assessment. Consideration of editing time was only given in 11 (9.4%) papers. A single manual contour as a ground-truth comparator was used in 65 (55.6%) studies. Only 31 (26.5%) studies compared auto-contours to usual inter- and/or intra-observer variation. In conclusion, significant variation exists in how research papers currently assess the accuracy of automatically generated contours. Geometric measures are the most popular, however their clinical utility is unknown. There is heterogeneity in the methods used to perform clinical assessment. Considering the different stages of system implementation may provide a framework to decide the most appropriate metrics. This analysis supports the need for a consensus on the clinical implementation of auto-contouring. Highlights: A systematic review of auto-contouring assessment publications was performed. Variation in the metrics used in auto-contouring research is demonstrated. Geometric metrics are the most popular, but these may not be clinically meaningful. There is significant heterogeneity in the "clinically relevant" assessment metrics. There is a need to standardise auto-contouring assessment, to enable clinical use.