2014 Rossi Award Lecture: Beyond Internal Validity. (April 2015)

• Record Type: Journal Article
• Authors: Orr, Larry L.
• Is Part Of: Evaluation review. Volume 39:Number 2(2015)
• Page Start: 167
• Page End: 178
• Publication Date: 2015-04
• Subjects: methodological development -- content area -- outcome evaluation (other than economic evaluation) -- design and evaluation of programs and policies; Evaluation research (Social action programs) -- Periodicals; 361.0072
• DOI: 10.1177/0193841X15573659
• Languages: English
• ISSNs: 0193-841X

Abstract

Background: For much of the last 40 years, the evaluation profession has been consumed in a battle over internal validity. Today, that battle has been decided. Random assignment, while still far from universal in practice, is almost universally acknowledged as the preferred method for impact evaluation. It is time for the profession to shift its attention to the remaining major flaws in the "standard model" of evaluation: (i) external validity and (ii) the high cost and low hit rate of experimental evaluations as currently practiced. Recommendations: To raise the profession's attention to external validity, the author recommends some simple, easy steps to be taken in every evaluation. The author makes two recommendations to increase the number of interventions found to be effective within existing resources: First, a two-stage evaluation strategy in which a cheap, streamlined Stage 1 evaluation is followed by a more intensive Stage 2 evaluation only for those interventions found to be effective in a Stage 1 trial and, second, use of random assignment to guide the myriad program management decisions that must be made in the course of routine program operations. This article is not intended as a solution to these issues: It is intended to stimulate the evaluation community to take these issues more seriously and to develop innovative solutions.