Enhancing efficiency, reliability, and rigor in competency model analysis using natural language processing. Issue 3 (12th June 2018) - Garman, Andrew N.; Standish, Melanie P.; Kim, Dae Hyun
Abstract : Background: Competency modeling is frequently used in higher education and workplace settings to inform a variety of learning and performance improvement programs. However, approaches commonly taken to modeling tasks can be very labor‐intensive, and are vulnerable to perceptual and experience biases of raters. Aims: The present study assesses the potential for natural language processing (NLP) to support competency‐related tasks, by developing a baseline comparison of results generated by NLP to results generated by human raters. Methods: Two raters separately conducted cross‐walks for leadership competency models of graduate healthcare management programs from eight universities against a newly validated competency model from the National Center for Healthcare Leadership containing 28 competencies, to create 224 cross‐walked pairs of "best matches". Results: Results indicated that the NLP model performed at least as accurately as human raters, who required a total of 16 work hours to complete, versus the NLP calculations which were nearly instantaneous. Conclusion: Based on these findings, we conclude that NLP has substantial promise as a high‐efficiency adjunct to human evaluations in competency cross‐walks.