Quality of epidemiological studies: Procedural rules for uncertain science for policy, a case study on bisphenol-A. Issue 84 (June 2018)

• Record Type: Journal Article
• Authors: Maxim, Laura; Van der Sluijs, Jeroen
• Is Part Of: Environmental science & policy. Issue 84(2018)
• Page Start: 80
• Page End: 87
• Publication Date: 2018-06
• Subjects: Quality -- Epidemiologic -- Chemical -- Risk -- Uncertainty -- Advisory; Environmental policy -- Periodicals; Environmental sciences -- Periodicals; Environnement -- Politique gouvernementale -- Périodiques; Sciences de l'environnement -- Périodiques; Environmental policy; Environmental sciences; Periodicals; Electronic journals; 363.70561
• DOI: 10.1016/j.envsci.2018.03.010
• Languages: English
• ISSNs: 1462-9011

Abstract

Highlights: Qualichem_epi allows collective quality assessment of epidemiologic studies. Qualichem_epi reflects both majority and minority judgments in a group. Expert judgments about quality are heterogeneous even using the same criteria. Group consensus can be an artifact of procedural rules in health agencies. Abstract: This paper proposes a method for in-depth mapping of heterogeneity in expert judgment, in the evaluation of the quality of epidemiological studies used in regulatory chemical risk assessment. Whereas consensus in scientific advisory groups provides legitimation for subsequent political action, it can also have unintended effects on the quality of regulatory risk assessment. Based on empirical testing of our method, called Qualichem_epi, with ten experts and two epidemiological case studies about bisphenol A (BPA)'s effects on human health, we have shown that expert judgment plays an essential role in managing uncertainty and deciding what "quality" of a study actually means. We found substantial heterogeneity of scientists' judgments about the quality of epidemiological studies, even if the same criteria were used for the assessment. This heterogeneity is not present anymore in reports produced by expert groups, where results are presented under the collective signature of all the scientists involved. We argue that flattening heterogeneity can be an important problem when it is not the result of true scientific agreement but only a secondary effect of consensus-based working procedures of agencies that experts have to follow. Qualichem_epi provides an easy to understand color-based picture of both majority and minority opinions in a scientific advisory group. We suggest that it could be used on a regular basis for communicating quality assessments of epidemiological studies in regulatory chemical risk assessment.