Towards program theory validation: Crowdsourcing the qualitative analysis of participant experiences. (February 2018)

• Record Type: Journal Article
• Authors: Harman, Elena; Azzam, Tarek
• Is Part Of: Evaluation and program planning. Volume 66(2018)
• Page Start: 183
• Page End: 194
• Publication Date: 2018-02
• Subjects: Crowdsourcing -- Theory-driven evaluation -- Qualitative analysis -- Stability -- Transcript coding -- Transcript rating -- Mechanical Turk -- MTurk; Health planning -- Periodicals; Medical care -- Evaluation -- Periodicals; 362.1068
• DOI: 10.1016/j.evalprogplan.2017.08.008
• Languages: English
• ISSNs: 0149-7189

Abstract

Highlights: Crowdsourcing is defined as the harnessing of the abilities of many people to complete a specific task or a set of tasks. This study combines a quantitative pattern matching framework traditionally used in theory-driven evaluation with crowdsourcing. The findings indicate that using crowdsourcing to analyze qualitative data can differentiate between program theory components that are supported by a participant's experience and those that are not. The findings indicate that crowdsourcing can help strengthen the theory-driven approach and expanding the range of programs to which it can be applied. Abstract: This exploratory study examines a novel tool for validating program theory through crowdsourced qualitative analysis. It combines a quantitative pattern matching framework traditionally used in theory-driven evaluation with crowdsourcing to analyze qualitative interview data. A sample of crowdsourced participants are asked to read an interview transcript and identify whether program theory components (Activities and Outcomes) are discussed and to highlight the most relevant passage about that component. The findings indicate that using crowdsourcing to analyze qualitative data can differentiate between program theory components that are supported by a participant's experience and those that are not. This approach expands the range of tools available to validate program theory using qualitative data, thus strengthening the theory-driven approach.