An approach for selecting and using a method of inter-coder reliability in information management research. (October 2020)

• Record Type: Journal Article
• Authors: Nili, Alireza; Tate, Mary; Barros, Alistair; Johnstone, David
• Is Part Of: International journal of information management. Volume 54(2020)
• Publication Date: 2020-10
• Subjects: Inter-coder reliability -- Content analysis -- Information -- Management; Social sciences -- Information services -- Periodicals; Social sciences -- Research -- Periodicals; Information science -- Periodicals; Management information systems -- Periodicals; Knowledge management -- Periodicals; Sciences sociales -- Documentation, Services de -- Périodiques; Sciences sociales -- Recherche -- Périodiques; Sciences de l'information -- Périodiques; Systèmes d'information de gestion -- Périodiques; Information science; Management information systems; Social sciences -- Information services; Social sciences -- Research; Periodicals; Electronic journals; 025.52068
• DOI: 10.1016/j.ijinfomgt.2020.102154
• Languages: English
• ISSNs: 0268-4012

Abstract

Highlights: Qualitative codes can be trusted only after ensuring their reliability. Checking the inter-coder reliability prevents extra time, effort and financial costs. Percent agreement is the least and Krippendorff's alpha is the most flexible method. The right method is that matches the characteristics of a specific project. The process of selecting and using the right method includes five major steps. Abstract: Qualitative researchers in information management research often need to evaluate inter-coder reliability (ICR) to test the trustworthiness of their content analysis. A suitable method of evaluating ICR enables researchers to rigorously assess the degree of agreement among two or more independent qualitative coders. This allows researchers to identify mistakes in the content analysis before the codes are used in developing and testing a theory or a measurement model and avoid any associated time, effort and financial cost. Different methods have been proposed, but little guidance is available on which approach to evaluating ICR should be used. In this paper, we review and compare leading ICR methods that are suitable for qualitative information management research. We propose an approach for selecting and using an ICR method, supported by an illustrative example. The five steps in our proposed approach include: selecting an ICR method based on its characteristics and requirements of a project; developing a coding scheme; selecting and training independent coders; calculating the ICR coefficient and resolving discrepancies; and reporting the process of evaluating ICR and its results.