Analysing synthesis of evidence in a systematic review in health professions education: observations on struggling beyond Kirkpatrick. Issue 1 (1st January 2020)

• Record Type: Journal Article
• Authors: Maudsley, Gillian; Taylor, David
• Is Part Of: Medical education online. Volume 25:Issue 1(2020)
• Publication Date: 2020-01-01
• Subjects: Best evidence -- cluster analysis -- epistemology -- evidence-based education -- evidence synthesis -- Kirkpatrick levels -- Maxwell dimensions of quality -- medical education -- process analysis -- systematic review; Medical education -- Periodicals; Education, Medical -- Periodicals; 610.71
• DOI: 10.1080/10872981.2020.1731278
• Languages: English
• ISSNs: 1087-2981

Abstract

ABSTRACT: Background : Systematic reviews in health professions education may well under-report struggles to synthesize disparate evidence that defies standard quantitative approaches. This paper reports further process analysis in a previously reported systematic review about mobile devices on clinical placements. Objective : For a troublesome systematic review: (1) Analyse further the distribution and reliability of classifying the evidence to Maxwell quality dimensions (beyond 'Does it work?' ) and their overlap with Kirkpatrick K-levels. (2) Analyse how the abstracts represented those dimensions of the evidence-base. (3) Reflect on difficulties in synthesis and merits of Maxwell dimensions. Design : Following integrative synthesis of 45 K2–K4 primary studies (by combined content–thematic analysis in the pragmatism paradigm): (1) Hierarchical cluster analysis explored overlap between Maxwell dimensions and K-levels. Independent and consensus-coding to Maxwell dimensions compared (using: percentages; kappa; McNemar hypothesis-testing) pre- vs post-discussion and (2) article abstract vs main body. (3) Narrative summary captured process difficulties and merits. Results : (1) The largest cluster (five-cluster dendrogram) was acceptability–accessibility–K1–appropriateness–K3, with K1 and K4 widely separated. For article main bodies, independent coding agreed most for appropriateness (good; adjusted kappa = 0.78). Evidence increased significantly pre–post-discussion about acceptability (p = 0.008; 31/45→39/45), accessibility, and equity-ethics-professionalism. (2) Abstracts suggested efficiency significantly less than main bodies evidenced: 31.1% vs 44.4%, p = 0.031. 3) Challenges and merits emerged for before, during, and after the review. Conclusions : There should be more systematic reporting of process analysis about difficulties synthesizing suboptimal evidence-bases. In this example, Maxwell dimensions were a useful framework beyond K-levels for classifying and synthesizing the evidence-base.