Enhancing efficiency, reliability, and rigor in competency model analysis using natural language processing. Issue 3 (12th June 2018)

• Record Type: Journal Article
• Authors: Garman, Andrew N.; Standish, Melanie P.; Kim, Dae Hyun
• Is Part Of: Journal of competency-based education. Volume 3:Issue 3(2018)
• Page Start: n/a
• Page End: n/a
• Publication Date: 2018-06-12
• Subjects: competency models -- leadership -- natural language processing -- validation; Competency-based education -- Periodicals; 371.3
• DOI: 10.1002/cbe2.1164
• Languages: English
• ISSNs: 2379-6154

Abstract

Abstract : Background: Competency modeling is frequently used in higher education and workplace settings to inform a variety of learning and performance improvement programs. However, approaches commonly taken to modeling tasks can be very labor‐intensive, and are vulnerable to perceptual and experience biases of raters. Aims: The present study assesses the potential for natural language processing (NLP) to support competency‐related tasks, by developing a baseline comparison of results generated by NLP to results generated by human raters. Methods: Two raters separately conducted cross‐walks for leadership competency models of graduate healthcare management programs from eight universities against a newly validated competency model from the National Center for Healthcare Leadership containing 28 competencies, to create 224 cross‐walked pairs of "best matches". Results: Results indicated that the NLP model performed at least as accurately as human raters, who required a total of 16 work hours to complete, versus the NLP calculations which were nearly instantaneous. Conclusion: Based on these findings, we conclude that NLP has substantial promise as a high‐efficiency adjunct to human evaluations in competency cross‐walks.