The effect of supplemental instruction on academic performance: An encouragement design experiment. (December 2016) - Paloyo, Alfredo R.; Rogan, Sally; Siminski, Peter
Highlights: We report on an experimental evaluation of Supplemental Instruction (PASS). One hour of PASS improved grades by an estimated 0.065 standard deviations. The result is consistent with the large (non-experimental) SI evaluation literature. This estimate is not statistically significant, reflecting limited power. We discuss strengths and challenges of randomized-encouragement-design experiments. Abstract: Supplemental Instruction (SI) or PASS (Peer Assisted Study Sessions) has been widely offered to students at tertiary institutions in many countries with the aim of improving academic performance. The SI/PASS evaluation literature is extensive, but it has not adequately addressed potential selection bias. We evaluate an SI/PASS program at an Australian university through a randomized-encouragement-design experiment. A randomly selected subgroup of students from first-year courses ( N = 6954) was offered large incentives (worth AUD 55, 000) to attend PASS which increased attendance by an estimated 0.47 hours each. This first-stage (inducement) effect did not vary with the size of the incentive and was larger (0.89) for students from disadvantaged backgrounds. Instrumental-variable estimates suggest that 1 hour of PASS improved grades by 0.065 standard deviations, which is consistent with the non-experimental literature. However, this estimate is not statistically significant, reflecting limited statistical power. The estimated effect is largest for students in their first semester at university.