Causal inferences on the effectiveness of complex social programs: Navigating assumptions, sources of complexity and evaluation design challenges. (December 2016)

• Record Type: Journal Article
• Authors: Chatterji, Madhabi
• Is Part Of: Evaluation and program planning. Volume 59(2016:Dec.)
• Page Start: 128
• Page End: 140
• Publication Date: 2016-12
• Subjects: Impact evaluations -- Experimental designs -- Mixed methods -- Causal inferences -- Complex social programs; Health planning -- Periodicals; Medical care -- Evaluation -- Periodicals; 362.1068
• DOI: 10.1016/j.evalprogplan.2016.05.009
• Languages: English
• ISSNs: 0149-7189

Abstract

Highlights: This paper navigates assumptions and research design challenges posed by complex social programs (CSPs). CSPs threaten assumptions of textbook-recommended experimental designs for impact studies. Seven (7) major sources of complexity that characterize a CSP are identified. Four methodological strategies are recommended to counter the design challenges. Systems thinking, multiphase mixed methods designs, and evaluation "situation recognition" are key requirements for gauging effects of CSPs. Abstract: This paper explores avenues for navigating evaluation design challenges posed by complex social programs (CSPs) and their environments when conducting studies that call for generalizable, causal inferences on the intervention's effectiveness. A definition is provided of a CSP drawing on examples from different fields, and an evaluation case is analyzed in depth to derive seven (7) major sources of complexity that typify CSPs, threatening assumptions of textbook-recommended experimental designs for performing impact evaluations. Theoretically-supported, alternative methodological strategies are discussed to navigate assumptions and counter the design challenges posed by the complex configurations and ecology of CSPs. Specific recommendations include: sequential refinement of the evaluation design through systems thinking, systems-informed logic modeling; and use of extended term, mixed methods (ETMM) approaches with exploratory and confirmatory phases of the evaluation. In the proposed approach, logic models are refined through direct induction and interactions with stakeholders. To better guide assumption evaluation, question-framing, and selection of appropriate methodological strategies, a multiphase evaluation design is recommended.