Are intermediate constraint question formats useful for evaluating student thinking and promoting learning in formative assessments?. (November 2019)

• Record Type: Journal Article
• Authors: Meir, Eli; Wendel, Daniel; Pope, Denise S.; Hsiao, Ling; Chen, Deborah; Kim, Kerry J.
• Is Part Of: Computers & education. Volume 141(2019)
• Publication Date: 2019-11
• Subjects: Authoring tools and methods -- Improving classroom teaching -- Interactive learning environments -- Teaching/learning strategies -- Assessment techniques; Education -- Data processing -- Periodicals; Education -- Periodicals; Computers -- Periodicals; Computer-Assisted Instruction -- Periodicals; Éducation -- Informatique -- Périodiques; Electronic journals; 370.285
• DOI: 10.1016/j.compedu.2019.103606
• Languages: English
• ISSNs: 0360-1315

Abstract

Abstract: There is a broad movement in science classes to emphasize higher-order thinking and complex skills over simple accumulation of facts. A major barrier to making this shift is a lack of good assessment tools for evaluating higher-order thinking that can be used with low effort by instructors. Computerized assessments aid instructors and increase efficiency, and potentially allow for immediate feedback. But easily-used question formats tend to be highly constrained (e.g. multiple-choice) and thus less informative on student thinking, while low constraint open-response formats (e.g. essay questions) are difficult to grade automatically. This is particularly problematic for formative assessments where immediate feedback specific to a student's current understanding is important to their learning. Here we explore two question formats that are intermediate in degree of constraint between open-response and multiple-choice formats. Intermediate constraint formats have the potential to allow some response construction while still enabling automatic feedback. We examine 15 questions in the two formats, with data from student interviews as well as from use in dozens of classrooms (both high school and university) as part of their regular curriculum. We find that the intermediate constraint questions enable automatic scoring of answers into specific categories of understanding and confusion, thus allowing for specific feedback to each student; that the intermediate constraint formats do encourage students to construct their own answers to a greater degree than multiple-choice; that they are usable by students; and that questions in those formats can show similar ranges of student answers as open-response. Further, using a split-class design in high school classrooms, we show that intermediate constraint questions can impact student learning to a greater degree than short answer questions. We conclude that these two question formats, and by extension perhaps other question formats with intermediate levels of constraint, are a useful addition to the toolbox of the educational software developer, but because they require a higher effort to construct than other formats, they should be reserved for key points in an online lesson. Graphical abstract: Image 1 Highlights: The diversity of answers is higher in intermediate constraint formats than in multiple-choice. Answers in intermediate constraint formats show substantial overlap with open-response formats. Answers in intermediate constraint formats can show a mix of correct and incorrect conceptions. Intermediate constraint formats can positively impact student understanding to a greater degree than open-response. Feedback and constraint both aid student learning.