Combining factor analysis with writing analytics for the formative assessment of written reflection. (July 2021)

• Record Type: Journal Article
• Authors: Liu, Ming; Kitto, Kirsty; Buckingham Shum, Simon
• Is Part Of: Computers in human behavior. Volume 120(2021)
• Publication Date: 2021-07
• Subjects: Reflection assessment -- Writing analytics -- Factor analysis; Interactive computer systems -- Periodicals; Man-machine systems -- Periodicals; 004.019
• DOI: 10.1016/j.chb.2021.106733
• Languages: English
• ISSNs: 0747-5632

Abstract

Abstract: The formative assessment of written reflection provides opportunities for students to improve their practice in an iterative manner using reflective writing. However, manual formative assessment of written reflection is time consuming and subjective. While progress has been made in deploying writing analytics tools to provide automated, formative feedback, few approaches to automated assessment are grounded in a validated, theory-based, formative assessment model. To address this, we propose a five-factor model of the Capability for Written Reflection (CWRef), grounded in the scholarship of reflective writing pedagogy . This paper uses Confirmatory Factor Analysis to validate the CWRef model by examining the relative contributions of textual features, derived from writing analytics, to each factor in the model, and their contributions to CWRef. The model was evaluated with two reflective writing corpora, showing which textual features, derived using Academic Writing Analytics and Linguistic Inquiry & Word Count, were significant indicators of factors in both corpora. In addition, it was found that the reflective writing context was an important factor influencing the validity of the CWRef model. Finally, we consider how this new analytical assessment model could enable improved tracking of progression in reflective writing, providing the basis for improved formative feedback. Highlights: Presentation of a five-factor assessment model (CWRef) for written reflection. Alignment of factors with textual features extracted from writing analytics. Empirical evaluation of CWRef using factor analysis in reflective writing contexts. . Implication for automatic formative assessment/feedback of written reflection.