Assessment of the variability and accuracy of readability scores across online calculators: Presenter(s): Olivia Mac, The University of Sydney, Australia. (April 2023)

• Record Type: Journal Article
• Authors: Ayre, Julie; Bell, Katy; McCaffery, Kirsten; Muscat, Danielle
• Is Part Of: Patient education and counseling. Volume 109(2023)Supplement
• Page Start: 62
• Publication Date: 2023-04
• Subjects: Patient education -- Periodicals; Health counseling -- Periodicals; Health education -- Periodicals; Counseling -- Periodicals; Patient Education -- Periodicals; Éducation des patients -- Périodiques; Counseling -- Périodiques; Éducation sanitaire -- Périodiques; 615.5071
• DOI: 10.1016/j.pec.2022.10.149
• Languages: English
• ISSNs: 0738-3991

Abstract

Abstract : Background: Readability assessment is the most common approach for evaluating the health literacy demands of written health information. However, there are no uniform guidelines for conducting and reporting readability analyses. Readability scores can vary by up to five reading grades across formulas. They may also vary depending on the online calculator used and subjective decisions about how to prepare the text for analyses. This study aimed to quantify the variability of readability scores, assess the effect of text preparation, and to assess the agreement of online calculated scores with the reference standard (hand calculated scores). Methods: Two readability scores for ten samples of text were obtained by each calculator: one using unedited text and one in which text was prepared (e.g., by removing incomplete sentences). We reported the Simple Measure of Gobbledygook (SMOG), Flesch Kincaid Grade Level (FKGL) and the Automated Readability Index (ARI). We used Bland-Altman plots to assess the agreement of online readability scores with the reference standard. Findings: We identified eight online calculators. There were 16 combinations of calculator and formula (4 for SMOG, 6 for FKGL and 6 for ARI). Across the online calculators, the same text produced scores that varied by up to 12.9 grade reading levels even when the same formula was used. Text preparation markedly decreased this variability between online calculators (range: 2.1 grade levels). SMOG index scores from two calculators, and FKGL scores from one calculator showed good agreement with the reference standard. The remaining calculators showed poor agreement with the reference standard. Discussion: Readability scores are inconsistent and often inaccurate. There is a need for more comprehensive and accessible guidelines for conducting and reporting on readability studies. Those evaluating written health information may consider complementing readability assessment with other tools, especially when the text is largely fragmented.