Type I Error Inflation in DIF Identification With Mantel–Haenszel: An Explanation and a Solution. (August 2014)

• Record Type: Journal Article
• Authors: Magis, David; De Boeck, Paul
• Is Part Of: Educational and psychological measurement. Volume 74:Number 4(2014:Aug.)
• Page Start: 713
• Page End: 728
• Publication Date: 2014-08
• Subjects: differential item functioning -- Mantel–Haenszel -- impact -- Type I error inflation -- dispersion; Educational tests and measurements -- Periodicals; Psychological tests -- Periodicals; 151.205
• DOI: 10.1177/0013164413516855
• Languages: English
• ISSNs: 0013-1644

Abstract

It is known that sum score-based methods for the identification of differential item functioning (DIF), such as the Mantel–Haenszel (MH) approach, can be affected by Type I error inflation in the absence of any DIF effect. This may happen when the items differ in discrimination and when there is item impact. On the other hand, outlier DIF methods have been developed that are robust against this Type I error inflation, although they are still based on the MH DIF statistic. The present article gives an explanation for why the common MH method is indeed vulnerable to the inflation effect whereas the outlier DIF versions are not. In a simulation study, we were able to produce the Type I error inflation by inducing item impact and item differences in discrimination. At the same time and in parallel with the Type I error inflation, the dispersion of the DIF statistic across items was increased. As expected, the outlier DIF methods did not seem sensitive to impact and differences in item discrimination.