Training and interobserver reliability in qualitative scoring of skeletal samples. (February 2017)

• Record Type: Journal Article
• Authors: Wilczak, Cynthia A; Mariotti, Valentina; Pany-Kucera, Doris; Villotte, Sébastien; Henderson, Charlotte Yvette
• Is Part Of: Journal of archaeological science. Volume 11(2017)
• Page Start: 69
• Page End: 79
• Publication Date: 2017-02
• Subjects: Enthesis -- Entheseal change (EC) -- Qualitative scoring -- Interobserver error -- Reliability; Archaeology -- Periodicals; Archaeology -- Research -- Periodicals; 930.1
• DOI: 10.1016/j.jasrep.2016.11.033
• Languages: English
• ISSNs: 2352-409X

Abstract

Abstract: This article reports on inter- and intra-observer reliability testing of the Coimbra method for fibrocartilaginous entheseal changes and discusses the results in the context of reliability for other qualitative methods that score skeletal variation. Four tests of interobserver and one test of intraobserver reliability were conducted. Three of the interobserver tests were conducted in person (Geneva, Coimbra A, Coimbra B) and one was conducted on-line. Before each test, the observers trained together and refinements were made to the method. Inter- and intra-observer reliability was assessed using percentage agreement, kappa scores, and Krippendorff's alpha. On-line testing had the lowest reliability (60% agreement, κ = 0.292, α = 0.369). Final reliability results in the Coimbra B test increased from the initial Geneva test (68.6 to 80.0% agreement, κ = 0.522 to 0.589, α = 0.604 to 0.666). Factors such as training, observation conditions, visual acuity, and differences in interpretation of scoring criteria impact reliability. Suggestions are made to address the general problem of reliability in qualitative methods. Highlights: Testing reliability of the Coimbra Method for recording entheseal change On-line collaboration is inadequate to improve reliability. In person training is essential to calibrate scoring. Reliability testing without prior training is not a fair test of a method. Suggestions to address reliability in qualitative scoring methods