Automated measures of speech content and speech organization in schizophrenia: Test-retest reliability and generalizability across demographic variables. (February 2023)

• Record Type: Journal Article
• Authors: Minor, Kyle S.; Lundin, Nancy B.; Myers, Evan J.; Fernández-Villardón, Aitana; Lysaker, Paul H.
• Is Part Of: Psychiatry research. Volume 320(2023)
• Publication Date: 2023-02
• Subjects: Speech content -- Speech organization -- Schizophrenia -- Reliability -- Race; Psychiatry -- Periodicals; Psychiatry -- periodicals; Psychiatrie -- Périodiques; 616.89
• DOI: 10.1016/j.psychres.2023.115048
• Languages: English
• ISSNs: 0165-1781

Abstract

Highlights: Test-retest reliability was relatively low for speech content and organization. The lower reliability may reflect the dynamic nature of speech. Less than 5% of speech indices changed over time based on demographic factors. Multiple main effects were found for speech based on race, income, and education. Findings highlight the need to design speech indices that account for demographics. Abstract: Technological advances in artificial intelligence and natural language processing have increased efficiency of assessing speech content and speech organization in schizophrenia. Despite these developments, there has been little focus on the psychometrics of these approaches. Using two common assessments, the current study addressed this gap by: 1) measuring test-retest reliability; and 2) assessing whether speech content and/or speech organization generalize across demographics. To test these aims, we examined psychometric properties of the Linguistic Inquiry Word Count (LIWC), a speech content measure, and the Coh-Metrix, a speech organization measure. Across baseline to six month ( n = 101) and baseline to one year ( n = 47) narrative speech samples, we generally observed fair reliability for speech content measures and fair to good reliability for speech organization measures. Regarding demographics, multiple speech indices varied by race, income, and education. The lack of excellent reliability scores for speech indices holds important implications for examining speech variables in clinical trials and highlights the dynamic nature of speech. This work illustrates the importance of designing speech content and speech organization measures with external validity across demographic factors. Future studies examining speech in schizophrenia should account for potential biases against demographic groups introduced by linguistic analysis tools.