Anchored Graphical Representations: A Graphical Alternative to Traditional Just Qualified Candidate Descriptors for Licensure Tests. Issue 1 (12th November 2018)

• Record Type: Journal Article
• Authors: Kannan, Priya; Tannenbaum, Richard J.; Hebert, Delano
• Is Part Of: ETS research report series. Issue 1(2018)
• Page Start: 1
• Page End: 17
• Publication Date: 2018-11-12
• Subjects: Standard setting -- performance level descriptors -- educator licensure; Universities and colleges -- Entrance examinations; Universities and colleges -- Graduate work -- Examinations; Universities and colleges -- United States -- Entrance examinations; Universities and colleges -- United States -- Graduate work -- Examinations; Graduate Record Examination; Educational tests and measurements; Social sciences; Education -- Research; Education -- Research; Educational tests and measurements; Graduate Record Examination; Social sciences; Universities and colleges -- Entrance examinations; Universities and colleges -- Graduate work -- Examinations; United States; 378
• DOI: 10.1002/ets2.12228
• Languages: English
• ISSNs: 2330-8516

Abstract

Abstract: A well‐constructed just qualified candidate (JQC) description is needed to arrive at a reasonable passing score for licensure tests. Traditionally, such descriptions consist of a list of knowledge and skill statements without sufficient context to internalize its intended meaning, allowing the standard‐setting panelists to make idiosyncratic interpretations. In a series of studies, we evaluated an alternative JQC description called anchored graphical representation (AGR). AGRs are intended to provide both text and visual organizers by each tested domain to contextualize the meaning of the JQC. In Study 1, 22 mathematics educators participated in a mock standard‐setting study and were randomly assigned to conditions in which they used either the AGR‐based or the text‐based definitions in making standard‐setting judgments. In Study 2, 17 social studies educators were randomly assigned to either the AGR‐based or text‐based condition. While the overall passing score did not noticeably vary between the conditions, consistent with our expectations, results from both studies showed that the domain passing scores for the AGR groups reflected the relative importance of each test domain for the JQC and that the inter‐panelist variability was smaller for the AGR groups. Collectively, our results indicate that use of the AGR resulted in standard‐setting ratings that were more aligned with the panelists' differential expectations of JQCs' performance for each domain, and the results add to the procedural validity evidence supporting the reasonableness of standard‐setting recommendations. Abstract : Report Number: ETS RR–18‐40