A Critical Look into the Beuk Standard‐Setting Method. Issue 1 (26th February 2020)

• Record Type: Journal Article
• Authors: Wyse, Adam E.
• Is Part Of: Educational measurement. Volume 39:Issue 1(2020)
• Page Start: 52
• Page End: 60
• Publication Date: 2020-02-26
• Subjects: beuk method -- cut scores -- importance weights -- slope -- standard setting; Educational tests and measurements -- Periodicals; Educational tests and measurements -- United States -- Periodicals; 371.262
• DOI: 10.1111/emip.12317
• Languages: English
• ISSNs: 0731-1745

Abstract

Abstract: One commonly used compromise standard‐setting method is the Beuk (1984) method. A key assumption of the Beuk method is that the emphasis given to the pass rate and the percent correct ratings should be proportional to the extent that the panelists agree on their ratings. However, whether the slope of Beuk line reflects the emphasis that panelists believe should be assigned to the pass rate and the percentage correct ratings has not be fully tested. In this article, I evaluate this critical assumption of the Beuk method by asking panelists to assign importance weights to their percentage correct and pass rate judgments. I show that in several cases that the emphasis suggested by the Beuk slope is noticeably different from what one would expect and is inconsistent with importance weight ratings. I also suggest two ways that the importance weights can be used to calculate alternate cut scores, and I show that one of the ways of calculating cut scores using the importance weights leads to larger potential differences in cut score estimates. I suggest that practitioners should consider collecting importance weights when the Beuk method is used for determining cut scores.