Analysing multisource feedback with multilevel structural equation models: Pitfalls and recommendations from a simulation study. (29th January 2019)

• Record Type: Journal Article
• Authors: Mahlke, Jana; Schultze, Martin; Eid, Michael
• Is Part Of: British journal of mathematical & statistical psychology. Volume 72:Part 2(2019)
• Page Start: 294
• Page End: 315
• Publication Date: 2019-01-29
• Subjects: multisource feedback -- multilevel structural equation modelling -- multitrait–multimethod analysis -- convergent and discriminant validity -- simulation study; Psychometrics -- Periodicals; Psychology -- Mathematical models -- Periodicals; Psychology -- Statistical methods -- Periodicals; 150.727
• DOI: 10.1111/bmsp.12149
• Languages: English
• ISSNs: 0007-1102

Abstract

Abstract : When multisource feedback instruments, for example, 360‐degree feedback tools, are validated, multilevel structural equation models are the method of choice to quantify the amount of reliability as well as convergent and discriminant validity. A non‐standard multilevel structural equation model that incorporates self‐ratings (level‐2 variables) and others' ratings from different additional perspectives (level‐1 variables), for example, peers and subordinates, has recently been presented. In a Monte Carlo simulation study, we determine the minimal required sample sizes for this model. Model parameters are accurately estimated even with the smallest simulated sample size of 100 self‐ratings and two ratings of peers and of subordinates. The precise estimation of standard errors necessitates sample sizes of 400 self‐ratings or at least four ratings of peers and subordinates. However, if sample sizes are smaller, mainly standard errors concerning common method factors are biased. Interestingly, there are trade‐off effects between the sample sizes of self‐ratings and others' ratings in their effect on estimation bias. The degree of convergent and discriminant validity has no effect on the accuracy of model estimates. The χ 2 test statistic does not follow the expected distribution. Therefore, we suggest using a corrected level‐specific standardized root mean square residual to analyse model fit and conclude with further recommendations for applied organizational research.