Examining the assessment of creativity with generalizability theory: An analysis of creative problem solving assessment tasks✰. (March 2022)

• Record Type: Journal Article
• Authors: Van Hooijdonk, Mare; Mainhard, Tim; Kroesbergen, Evelyn H.; Van Tartwijk, Jan
• Is Part Of: Thinking skills and creativity. Volume 43(2022)
• Publication Date: 2022-03
• Subjects: Assessment -- Creative problem solving -- Creativity -- Generalizability theory -- G-study -- D-study; Thought and thinking -- Periodicals; Critical thinking -- Study and teaching -- Periodicals; Creative thinking -- Study and teaching -- Periodicals; Thinking -- Periodicals; Creativeness -- Periodicals; Teaching -- Periodicals; Pensée -- Étude et enseignement -- Périodiques; Créativité (Éducation) -- Étude et enseignement -- Périodiques; 370.15205
• DOI: 10.1016/j.tsc.2021.100994
• Languages: English
• ISSNs: 1871-1871

Abstract

Highlights: Generalizability theory is applied step-by-step on CPS assessment data. The impact of elements of the assessment system differed slightly across aspects. Three to five tasks and two raters may produce reliable relative CPS scores. Implications for the assessment of CPS and creativity in general are discussed. The non-technical manual assists in applying G-theory in future creativity studies. Abstract: The assessment of creative problem solving (CPS) is challenging. Elements of an assessment procedure, such as the tasks that are used and the raters who assess those tasks, introduce variation in student scores that do not necessarily reflect actual differences in students' creative problem solving abilities. When creativity researchers evaluate assessment procedures, they often inspect these elements such as tasks and raters separately. We show the use of Generalizability Theory allows researchers to investigate creativity assessment procedures - and CPS assessments in particular - in a comprehensive and integrated way. In this paper, we first introduce this statistical framework and the choices creativity researchers need to make before applying Generalizability Theory to their data. Then, Generalizability Theory is applied in an analysis of CPS assessment tasks. We highlight how alterations in the nature of the assessment procedure, such as changing the number of tasks or raters, may affect the quality of CPS scores. Furthermore, we present implications for the assessment of CPS and for creativity research in general.