Feedback at scale: designing for accurate and timely practical digital skills evaluation. Issue 2 (3rd March 2020)

• Record Type: Journal Article
• Authors: Piccoli, Gabriele; Rodriguez, Joaquin; Palese, Biagio; Bartosiak, Marcin Lukasz
• Is Part Of: European journal of information systems. Volume 29:Issue 2(2020)
• Page Start: 114
• Page End: 133
• Publication Date: 2020-03-03
• Subjects: Pär Ågerfalk; Iris Junglas; Design science research -- feedback systems -- digital skills evaluation; Information technology -- Periodicals; Information technology -- Europe -- Periodicals; Management information systems -- Periodicals; Management information systems -- Europe -- Periodicals; 658.403805
• DOI: 10.1080/0960085X.2019.1701955
• Languages: English
• ISSNs: 0960-085X

Abstract

ABSTRACT: The global demand for digital proficiency has resulted in increasing pressure to "massify" education. As practical digital skills development becomes more important, there is a need to design accurate and timely performance feedback systems that can scale to a large number of learners. This paper contributes meta-requirements and design principles for designing a socio-technical artefact that offers a solution to the general problem of providing performance feedback at scale. The artefact evaluation provides interesting results for achieving the three objectives of a) scalability to a large number of learners, b) validity and reliability of the feedback, and c) positive impact on learners' behaviour and engagement with the feedback system. These results are obtained through the synergistic contribution of pedagogical prioritisation (i.e., what skills to cover), assignment design (i.e., what tasks to use to evaluate mastery) and automated measurement (i.e., grading engine functionalities for error detection).