On the Equivalence of Multirater Kappas Based on 2-Agreement and 3-Agreement with Binary Scores. (15th October 2012)

• Record Type: Journal Article
• Authors: Warrens, Matthijs J.
• Other Names: Hu J. Academic Editor.; Pons O. Academic Editor.
• Is Part Of: ISRN probability and statistics. Volume 2012(2012)
• Publication Date: 2012-10-15
• Subjects: Probabilities -- Periodicals; Mathematical statistics -- Periodicals; Mathematical statistics; Probabilities; Electronic journals; Periodicals; 519
• DOI: 10.5402/2012/656390
• Languages: English
• ISSNs: 2090-4711

Abstract

Abstract : Cohen's kappa is a popular descriptive statistic for summarizing agreement between the classifications of two raters on a nominal scale. With m ≥ 3 raters there are several views in the literature on how to define agreement. The concept of g -agreement ( g ∈ { 2, 3, …, m } ) refers to the situation in which it is decided that there is agreement if g out of m raters assign an object to the same category. Given m ≥ 2 raters we can formulate m − 1 multirater kappas, one based on 2-agreement, one based on 3-agreement, and so on, and one based on m -agreement. It is shown that if the scale consists of only two categories the multi-rater kappas based on 2-agreement and 3-agreement are identical.