Simulation With PARTS (Phase-Augmented Research and Training Scenarios). Issue 3 (June 2015) - Schick, Carl J.
Abstract Introduction Assessment in simulation is gaining importance, as are scenario design methods increasing opportunity for assessment. We present our approach to improving measurement in complex scenarios using PARTS [Phase-Augmented Research and Training Scenarios], essentially separating cases into clearly delineated phases. Methods We created 7 PARTS with real-time rating instruments and tested these in 63 cases during 4 weeks of simulation. Reliability was tested by comparing real-time rating with postsimulation video-based rating using the same instrument. Validity was tested by comparing preintervention and postintervention total results, by examining the difference in improvement when focusing on the phase-specific results addressed by the intervention, and further explored by trying to demonstrate the discrete improvement expected from proficiency in the rare occurrence of leader inclusive behavior. Results Intraclass correlations [3, 1] between real-time and postsimulation ratings were 0.951 (95% confidence interval [CI], 0.794–0.990), 1.00 (95% CI, — to —), 0.948 (95% CI, 0.783–0.989), and 0.995 (95% CI, 0.977–0.999) for 3 phase-specific scores and total scenario score, respectively. Paired t tests of prelecture-postlecture performance showed an improvement of 14.26% (bias-corrected and accelerated bootstrap [BCa] 95% CI, 4.71–23.82; P = 0.009) for total performance but of 28.57% (BCa 95% CI, 13.84–43.30; P = 0.002) for performance in the respective phase. The correlation of total scenario performance with leader inclusiveness was not significant (rs = 0.228; BCa 95% CI. −0.082 to 0.520; P = 0.119) but significant for specific phase performance (rs = 0.392; BCa 95% CI, 0.118–0.632; P = 0.006). Conclusions The PARTS allowed for improved reliability and validity of measurements in complex scenarios.