Exploring Methods for Developing Behaviorally Anchored Rating Scales for Evaluating Structured Interview Performance. Issue 1 (5th June 2017)

• Record Type: Journal Article
• Authors: Kell, Harrison J.; Martin‐Raugh, Michelle P.; Carney, Lauren M.; Inglese, Patricia A.; Chen, Lei; Feng, Gary
• Is Part Of: ETS research report series. Issue 1(2017)
• Page Start: 1
• Page End: 26
• Publication Date: 2017-06-05
• Subjects: Amazon Mechanical Turk -- behaviorally anchored rating scales -- crowdsourcing -- employment interviews -- performance appraisal -- social skills -- structured interviews; Universities and colleges -- Entrance examinations; Universities and colleges -- Graduate work -- Examinations; Universities and colleges -- United States -- Entrance examinations; Universities and colleges -- United States -- Graduate work -- Examinations; Graduate Record Examination; Educational tests and measurements; Social sciences; Education -- Research; Education -- Research; Educational tests and measurements; Graduate Record Examination; Social sciences; Universities and colleges -- Entrance examinations; Universities and colleges -- Graduate work -- Examinations; United States; 378
• DOI: 10.1002/ets2.12152
• Languages: English
• ISSNs: 2330-8516

Abstract

Abstract: Behaviorally anchored rating scales (BARS) are an essential component of structured interviews. Use of BARS to evaluate interviewees' performance is associated with greater predictive validity and reliability and less bias. BARS are time‐consuming and expensive to construct, however. This report explores the feasibility of gathering participants' responses to structured interview questions through an online crowdsourcing platform and using those responses to develop BARS. We describe the development of 12 structured interview questions to assess four applied social skills, elicitation of responses to these questions in the form of critical incidents from 68 respondents, and the creation of BARS from these critical incidents. Results indicate online participants are able to produce responses of sufficient quality to generate BARS for evaluating structured interview performance. We conclude by discussing limitations to this approach and future directions for research and practice. Abstract : Report Number: ETS RR‐17–28