Developing a contributing factor classification scheme for Rasmussen's AcciMap: Reliability and validity evaluation. (October 2017)

• Record Type: Journal Article
• Authors: Goode, N.; Salmon, P.M.; Taylor, N.Z.; Lenné, M.G.; Finch, C.F.
• Is Part Of: Applied ergonomics. Volume 64(2017)
• Page Start: 14
• Page End: 26
• Publication Date: 2017-10
• Subjects: Systems thinking -- Reliability -- Validity -- Incident classification; Human engineering -- Periodicals; 620.82
• DOI: 10.1016/j.apergo.2017.04.014
• Languages: English
• ISSNs: 0003-6870

Abstract

Abstract: One factor potentially limiting the uptake of Rasmussen's (1997) Accimap method by practitioners is the lack of a contributing factor classification scheme to guide accident analyses. This article evaluates the intra- and inter-rater reliability and criterion-referenced validity of a classification scheme developed to support the use of Accimap by led outdoor activity (LOA) practitioners. The classification scheme has two levels: the system level describes the actors, artefacts and activity context in terms of 14 codes; the descriptor level breaks the system level codes down into 107 specific contributing factors. The study involved 11 LOA practitioners using the scheme on two separate occasions to code a pre-determined list of contributing factors identified from four incident reports. Criterion-referenced validity was assessed by comparing the codes selected by LOA practitioners to those selected by the method creators. Mean intra-rater reliability scores at the system ( M = 83.6%) and descriptor ( M = 74%) levels were acceptable. Mean inter-rater reliability scores were not consistently acceptable for both coding attempts at the system level ( M T1 = 68.8%; M T2 = 73.9%), and were poor at the descriptor level ( M T1 = 58.5%; M T2 = 64.1%). Mean criterion referenced validity scores at the system level were acceptable ( M T1 = 73.9%; M T2 = 75.3%). However, they were not consistently acceptable at the descriptor level ( M T1 = 67.6%; M T2 = 70.8%). Overall, the results indicate that the classification scheme does not currently satisfy reliability and validity requirements, and that further work is required. The implications for the design and development of contributing factors classification schemes are discussed. Highlights: Evaluation of a contributing factor classification scheme to support the use of Accimap by practitioners. The scheme had acceptable intra-rater reliability. The scheme did not satisfy inter-reliability and criterion-referenced validity requirements. The key methodological issues for designing reliability and validity studies are discussed. Illustrates the challenges involved in developing a valid, reliable and useful classification scheme.