A counterfactual impact evaluation of a bilingual program on students' grade point average at a spanish university. (June 2018)

• Record Type: Journal Article
• Authors: Arco-Tirado, J.L.; Fernández-Martín, F.; Ramos-García, A.M.; Littvay, L.; Villoria, J.; Naranjo, J.A.
• Is Part Of: Evaluation and program planning. Volume 68(2018)
• Page Start: 81
• Page End: 89
• Publication Date: 2018-06
• Subjects: Bilingual program -- Counterfactual impact evaluation -- Causal inference -- Matching -- English as medium of instruction -- Grade point average; Health planning -- Periodicals; Medical care -- Evaluation -- Periodicals; 362.1068
• DOI: 10.1016/j.evalprogplan.2018.02.013
• Languages: English
• ISSNs: 0149-7189

Abstract

Highlights: The paper provides convincing evidence that there is a cost in academic performance of the bilingual program analysed. The paper describes and illustrates thoughtfully the use of regression adjustment and propensity scores for the analysis of observational data. The negative effect of the program on key students outcomes reinforces the need for evaluating the academic impact of HE bilingual programs. These results although in some extent disappointing, hopefully will contribute to mitigate the international scenario of scarcity of data on the success or impact of EMI programs. Abstract: This observational study intends to estimate the causal effects of an English as a Medium of Instruction (EMI) program (as predictor) on students Grade Point Average (GPA) (as outcome) at a particular University in Spain by using a Counterfactual Impact Evaluation (CIE). The need to address the crucial question of causal inferences in EMI programs to produce credible evidences of successful interventions contrasts, however, with the absence of experimental or quasi-experimental research and evaluation designs in the field. CIE approach is emerging as a methodologically viable solution to bridge that gap. The program evaluated here consisted in delivering an EMI program in a Primary Education Teacher Training Degree group. After achieving balance on the observed covariates and recreating a situation that would have been expected in a randomized experiment, three matching approaches such as genetic matching, nearest neighbor matching and Coarsened Exact Matching were used to analyze observational data from a total of 1288 undergraduate students, including both treatment and control group. Results show unfavorable effects of the bilingual group treatment condition. Potential interpretations and recommendations are provided in order to strengthen future causal evidences of bilingual education programs' effectiveness in Higher Education.