Assessing cognitive presence in online inquiry‐based discussion through text classification and epistemic network analysis. (14th November 2022)

• Record Type: Journal Article
• Authors: Ba, Shen; Hu, Xiao; Stein, David; Liu, Qingtang
• Is Part Of: British journal of educational technology. Volume 54:Number 1(2023)
• Page Start: 247
• Page End: 266
• Publication Date: 2022-11-14
• Subjects: analytics for assessment -- cognitive presence -- epistemic network analysis -- online discussion -- text classification; Audio-visual education -- Periodicals; 371.33
• DOI: 10.1111/bjet.13285
• Languages: English
• ISSNs: 0007-1013

Abstract

Abstract: Providing coaching to participants in inquiry‐based online discussions contributes to developing cognitive presence (CP) and higher‐order thinking. However, a primary issue limiting quality and timely coaching is instructors' lack of tools to efficiently identify CP phases in massive discussion transcripts and effectively assess learners' cognitive development. This study examined a computational approach integrating text mining and co‐occurrence analysis for assessing CP and cognitive development in online discussions based on the community of inquiry (CoI) framework. First, text classifiers trained on different language models were evaluated for identifying and coding the CP phases. Second, epistemic network analysis (ENA) was employed to model cognitive patterns reflected by co‐occurrences between the coding elements. Results indicated that text classifiers trained on the state‐of‐the‐art language model Bidirectional Encoder Representations from Transformers (BERT) can address the efficiency issue in coding CP phases in discussion transcripts and obtain substantial agreements (Cohen's k = 0.76) with humans, which outperformed other baseline classifiers. Furthermore, compared to traditional quantitative content analysis, ENA can effectively model the temporal characteristics of online discourse and detect fine‐grained cognitive patterns. Overall, the findings suggest a feasible path for applying learning analytics to tracking learning progression and informing theory‐based assessments. Practitioner notes What is already known about this topic Cognitive presence is an important construct describing the progression of thinking in online inquiry‐based discussions. Most studies used self‐report instruments or quantitative content analysis to measure and assess cognitive presence. More efficient and effective approaches were needed by instructors to support assessment of cognitive development and determine coaching strategies. What this paper adds An integrated computational approach for the developmental and formative assessment of cognitive presence was proposed and evaluated. A BERT‐based text classification model could efficiently code massive transcripts and achieve substantial agreements with human coders. Epistemic network analysis effectively revealed the process of cognitive development and identified representative discussion patterns and behaviours. Implications for practice and/or policy The proposed approach can considerably reduce the pressure on instructors, enabling them to focus on quality coaching and feedback. Compared to frequencies of individual codes, the connective features between codes carry more insights for assessing cognitive patterns. Learners in a discussion group play different roles and produce diverse paths of cognitive development.