Evaluating the internal consistency of subtraction‐based and residualized difference scores: Considerations for psychometric reliability analyses of event‐related potentials. (21st January 2021)

• Record Type: Journal Article
• Authors: Clayson, Peter E.; Baldwin, Scott A.; Larson, Michael J.
• Is Part Of: Psychophysiology. Volume 58:Number 4(2021)
• Page Start: n/a
• Page End: n/a
• Publication Date: 2021-01-21
• Subjects: difference score -- ERP Reliability Analysis (ERA) Toolbox -- event‐related potentials (ERPs) -- generalizability theory -- psychometric reliability -- classical test theory; Psychophysiology -- Periodicals; 612.8
• DOI: 10.1111/psyp.13762
• Languages: English
• ISSNs: 0048-5772

Abstract

Abstract: In studies of event‐related brain potentials (ERPs), difference scores between conditions in a task are frequently used to isolate neural activity for use as a dependent or independent variable. Adequate score reliability is a prerequisite for studies examining relationships between ERPs and external correlates, but there is no extensive treatment on the suitability of the various available approaches to estimating difference score reliability that focus on ERP research. In the present study, we provide formulas from classical test theory and generalizability theory for estimating the internal consistency of subtraction‐based and residualized difference scores. These formulas are then applied to error‐related negativity (ERN) and reward positivity (RewP) difference scores from the same sample of 117 participants. Analyses demonstrate that ERN difference scores can be reliable, which supports their use in studies of individual differences. However, RewP difference scores yielded poor reliability due to the high correlation between the constituent reward and non‐reward ERPs. Findings emphasize that difference score reliability largely depends on the internal consistency of constituent scores and the correlation between those scores. Furthermore, generalizability theory yields more suitable estimates of internal consistency for subtraction‐based difference scores than classical test theory. We conclude that ERP difference scores can show adequate reliability and be useful for isolating neural activity in studies of individual differences. Abstract : In studies of event‐related brain potentials (ERPs), difference scores are used to isolate neural activity for statistical analysis, despite concerns about their low internal consistency. This manuscript describes the optimal approaches for estimating the internal consistency of ERP difference scores and clarifies the measurement characteristics that improve their reliability.