On doing relevant and rigorous experiments: Review and recommendations. Issue 1 (17th December 2018)

• Record Type: Journal Article
• Authors: Lonati, Sirio; Quiroga, Bernardo F.; Zehnder, Christian; Antonakis, John
• Is Part Of: Journal of operations management. Volume 64:Issue 1(2018)
• Page Start: 19
• Page End: 40
• Publication Date: 2018-12-17
• Subjects: Experimental methods -- Research methods -- Research impact -- Causality -- Endogeneity -- Quasi‐experimentation; Production management -- Periodicals; Management -- Periodicals; 658.5
• DOI: 10.1016/j.jom.2018.10.003
• Languages: English
• ISSNs: 0272-6963

Abstract

Abstract: Although experiments are the gold standard for establishing causality, several threats can undermine the internal validity of experimental findings. In this article, we first discuss these threats, which include the lack of consequential decisions and outcomes, deception, demand effects and unfair comparisons, as well as issues concerning statistical validity (e.g., minimum sample size per cell, estimating variance correctly). We expose each problem, show potential solutions, and bring to the fore issues of relevance of the findings (i.e., external and ecological validity). Thereafter, we take stock of the state‐of‐the‐science regarding validity threats using a representative sample of 468 recent experiments from 258 articles published in top‐tier journals. We compare research practices in three fields of study—management, social psychology, and economics, which regularly use experimental research—to operations management, which has more recently begun to use the experimental paradigm. Our results underscore the importance for journals and authors to follow what we identify to be best‐practice methodological suggestions (i.e., the "ten commandments" of experimental research). We show that—on average—markers of methodological rigor and generalizability positively and significantly predict the citations received by published articles. Finally, given that experiments are infeasible in some settings, we conclude with a brief review of often overlooked quasi‐experimental designs, which are useful for generating strong counterfactuals and hence allow making causal claims in the field.