A Generalizability Theory Study to Examine Sources of Score Variance in Third‐Party Evaluations Used in Decision‐Making for Graduate School Admissions. Issue 1 (25th October 2018)

• Record Type: Journal Article
• Authors: McCaffrey, Daniel F.; Oliveri, Maria Elena; Holtzman, Steven
• Is Part Of: ETS research report series. Issue 1(2018)
• Page Start: 1
• Page End: 17
• Publication Date: 2018-10-25
• Subjects: measurement error -- inter‐rater reliability -- sources of error -- generalizability coefficient -- D study; Universities and colleges -- Entrance examinations; Universities and colleges -- Graduate work -- Examinations; Universities and colleges -- United States -- Entrance examinations; Universities and colleges -- United States -- Graduate work -- Examinations; Graduate Record Examination; Educational tests and measurements; Social sciences; Education -- Research; Education -- Research; Educational tests and measurements; Graduate Record Examination; Social sciences; Universities and colleges -- Entrance examinations; Universities and colleges -- Graduate work -- Examinations; United States; 378
• DOI: 10.1002/ets2.12225
• Languages: English
• ISSNs: 2330-8516

Abstract

Abstract: Scores from noncognitive measures are increasingly valued for their utility in helping to inform postsecondary admissions decisions. However, their use has presented challenges because of faking, response biases, or subjectivity, which standardized third‐party evaluations (TPEs) can help minimize. Analysts and researchers using TPEs, however, need to be mindful of the potential for construct‐irrelevant differences that may arise in TPEs due to differences in evaluators' rating approaches, which introduces measurement error. Research on sources of construct‐irrelevant variance in TPEs is scarce. We address this paucity by conducting generalizability theory (G theory) analyses using TPE data that informs postsecondary admissions decisions. We also demonstrate an approach to assess the size of interevaluator variability and conduct a decision study to determine the number of evaluators necessary to achieve the desired generalizability coefficient. We illustrate these approaches using a TPE whereby applicants select their evaluators, leading to a situation where most evaluators solely rate one applicant. We conclude by presenting strategies to improve the design of TPEs to help increase confidence in their use. Abstract : Report Number: ETS RR–18‐37