Understanding and misunderstanding randomized controlled trials. (August 2018)

• Record Type: Journal Article
• Authors: Deaton, Angus; Cartwright, Nancy
• Is Part Of: Social science & medicine. Volume 210(2018)
• Page Start: 2
• Page End: 21
• Publication Date: 2018-08
• Subjects: RCTs -- Balance -- Bias -- Precision -- External validity -- Transportation of results -- Health -- Economic development; Social medicine -- Periodicals; Medical anthropology -- Periodicals; Public health -- Periodicals; Psychology -- Periodicals; Medicine -- Periodicals; Medicine -- Periodicals; Médecine sociale -- Périodiques; Anthropologie médicale -- Périodiques; Santé publique -- Périodiques; Psychologie -- Périodiques; Médecine -- Périodiques; Electronic journals; 362.105
• DOI: 10.1016/j.socscimed.2017.12.005
• Languages: English
• ISSNs: 0277-9536

Abstract

Abstract: Randomized Controlled Trials (RCTs) are increasingly popular in the social sciences, not only in medicine. We argue that the lay public, and sometimes researchers, put too much trust in RCTs over other methods of investigation. Contrary to frequent claims in the applied literature, randomization does not equalize everything other than the treatment in the treatment and control groups, it does not automatically deliver a precise estimate of the average treatment effect (ATE), and it does not relieve us of the need to think about (observed or unobserved) covariates. Finding out whether an estimate was generated by chance is more difficult than commonly believed. At best, an RCT yields an unbiased estimate, but this property is of limited practical value. Even then, estimates apply only to the sample selected for the trial, often no more than a convenience sample, and justification is required to extend the results to other groups, including any population to which the trial sample belongs, or to any individual, including an individual in the trial. Demanding 'external validity' is unhelpful because it expects too much of an RCT while undervaluing its potential contribution. RCTs do indeed require minimal assumptions and can operate with little prior knowledge. This is an advantage when persuading distrustful audiences, but it is a disadvantage for cumulative scientific progress, where prior knowledge should be built upon, not discarded. RCTs can play a role in building scientific knowledge and useful predictions but they can only do so as part of a cumulative program, combining with other methods, including conceptual and theoretical development, to discover not 'what works', but 'why things work'. Highlights: Randomization does not balance confounders in any single trial. Unbiasedness is of limited practical value compared with precision. Asymmetric distributions of treatment effects pose threats to significance testing. The best method depends on hypothesis tested, what's known, and cost of mistakes. RCT results can serve science but are weak ground for inferring 'what works'.