Creativity Assessment over Time: Examining the Reliability of CAT Ratings. (25th June 2020)

• Record Type: Journal Article
• Authors: Barth, Philipp; Stadtmann, Georg
• Is Part Of: Journal of creative behavior. Volume 55:Number 2(2021)
• Page Start: 396
• Page End: 409
• Publication Date: 2020-06-25
• Subjects: consensual assessment technique -- creativity assessment -- fashion -- Instagram -- reliability -- temporal stability -- test‐retest reliability; Creative ability -- Periodicals; Creativeness -- Abstracts; Creativeness -- Periodicals; Education -- Abstracts; Education -- Periodicals; Créativité -- Périodiques; 153.35
• DOI: 10.1002/jocb.462
• Languages: English
• ISSNs: 0022-0175

Abstract

Abstract: The consensual assessment technique (CAT) is a reliable and valid method to measure (product) creativity and often considered the gold standard of creativity assessment. The reliability measure traditionally applied in CAT studies—inter‐rater reliability—cannot capture time‐sampling error, which is a particular relevant source of error for specific applications of the CAT. Therefore, the present study intended to investigate the test–retest reliability of CAT ratings. We asked raters ( N = 61) for their creativity assessment of the same set of 90 fashion outfits at an initial rating session and a follow‐up session either 2 or 4 weeks later. We found that mean product ratings—the actual focus of interest in the CAT—were highly stable over time, as evidenced by consistency and agreement ICCs clearly exceeding levels of .90. However, individual raters (partially) lacked temporal stability, indicating a drift in rater tendencies over time. Our findings support the CAT's reputation as a highly reliable measurement method, but question the temporal rating stability of the CAT's actual "measurement instrument, " namely individual judges.