A statistical test for detecting discordance in rankings between k groups. Issue 10 (27th July 2019)

• Record Type: Journal Article
• Authors: Fischer, Katinka; Bothung, Christoph; Lieder, Felix; Wolfart, Stefan; Schwender, Holger
• Is Part Of: Journal of applied statistics. Volume 46:Issue 10(2019)
• Page Start: 1822
• Page End: 1842
• Publication Date: 2019-07-27
• Subjects: Ranking data -- disagreement -- average ranking -- inhomogeneous intra-group concordance -- permutation test; Statistics -- Periodicals; 519.5
• DOI: 10.1080/02664763.2019.1572720
• Languages: English
• ISSNs: 0266-4763

Abstract

ABSTRACT: In several research areas such as psychology, social science, and medicine, studies are conducted in which objects should be ranked by different judges/raters and the concordance of the different rankings is then analyzed. In such studies, it is also frequently of interest to compare the rankings between different groups of judges, e.g. female vs. male judges or judges from different professions. In the two-group case, the two-group concordance test of Schucany & Frawley can be employed for such a comparison. In this article, we propose an extension of this test enabling the comparison of rankings from more than two groups of judges. This test aims to detect disagreement in the average rankings of the objects between k groups with an at least moderate intra-group concordance. We evaluate this test in an extensive simulation study and in an application to data from an aesthetics study. This simulation study shows that the proposed test is able to detect differences between average rankings and performs well even in situations in which the disagreement is comparably small or the intra-group concordance is inhomogeneous.