The inter-rater reliability and individual reviewer performance of the 2012 world heart federation guidelines for the echocardiographic diagnosis of latent rheumatic heart disease. (1st April 2021)

• Record Type: Journal Article
• Authors: Scheel, Amy; Mirabel, Mariana; Nunes, Maria Carmo Pereira; Okello, Emmy; Sarnacki, Rachel; Steer, Andrew C.; Engelman, Daniel; Zimmerman, Meghan; Zühlke, Liesl; Sable, Craig; Beaton, Andrea
• Is Part Of: International journal of cardiology. Volume 328(2021)
• Page Start: 146
• Page End: 151
• Publication Date: 2021-04-01
• Subjects: Rheumatic heart disease -- Echocardiography; WHF World Heart Federation -- RHD Rheumatic Heart Disease -- ARF Acute Rheumatic Fever -- CW Continuous Wave -- AMVL Anterior Mitral Valve Leaflet -- MR mitral regurgitation -- AR aortic regurgitation; Cardiology -- Periodicals; Electronic journals; 616.12
• DOI: 10.1016/j.ijcard.2020.11.013
• Languages: English
• ISSNs: 0167-5273

Abstract

Abstract: Background: In 2012, the World Heart Federation (WHF) published guidelines for the echocardiographic diagnosis of rheumatic heart disease (RHD). This study assesses individual reviewer performance and inter-rater agreement and reliability on the presence of any RHD, as well classification of RHD based on the 2012 WHF criteria. Methods: Four cardiologists individually reviewed echocardiograms in the context of a randomized clinical trial (ClinicalTrials.gov :NCT03346525 ) and participated in a blinded adjudication panel. Panel decision was the reference standard for diagnosis. Performance of individual reviewers to panel adjudication was compared through sensitivity and specificity analyses and inter-rater reliability was assessed between individual panelists using Fleiss free marginal multirater kappa. Results: Echocardiograms from 784 children had two independent reports and panel adjudication. The accuracy of independent reviewers for any RHD had high sensitivity (94%, 95% CI 93–95%) and moderate specificity (62%, 95% CI 53–70%). Sensitivity and specificity for definite RHD was 61.3 (95% CI, 55.3–67.1) and 93.1 (95% CI, 91.6–94.4), with 86.8 (84.7–88.7) and 65.8 (61.0–70.4) for borderline RHD. There was moderate inter-rater agreement (κ = 0.66) on the presence of any RHD while agreement for specific 2012 WHF classification was only fair (κ = 0.51). Conclusions: The 2012 WHF guidelines are moderately reproducible when used by expert cardiologists. More cases of RHD were diagnosed by an consensus panel than by individual reviewers. A revision to the criteria is now warranted to further increase the reliability of the WHF criteria. Highlights: Compared to the panel consensus, individuals were less likely to diagnose any RHD, as well as morphological abnormalities. Inter-rater agreement, even amongst experts, for 2012 WHF classification is fair. Considerable variability remains amongst reviewers utilizing the WHF criteria. Our data support the use of a > 2 reviewer system for more diagnosis of RHD. A revision to the 2012 WHF is now warranted.