Accumulating evidence using crowdsourcing and machine learning: A living bibliography about existential risk and global catastrophic risk. (February 2020)

• Record Type: Journal Article
• Authors: Shackelford, Gorm E.; Kemp, Luke; Rhodes, Catherine; Sundaram, Lalitha; ÓhÉigeartaigh, Seán S.; Beard, Simon; Belfield, Haydn; Weitzdörfer, Julius; Avin, Shahar; Sørebø, Dag; Jones, Elliot M.; Hume, John B.; Price, David; Pyle, David; Hurt, Daniel; Stone, Theodore; Watkins, Harry; Collas, Lydia; Cade, Bryony C.; Johnson, Thomas Frederick; Freitas-Groff, Zachary; Denkenberger, David; Levot, Michael; Sutherland, William J.
• Is Part Of: Futures. Volume 116(2020)
• Publication Date: 2020-02
• Subjects: Bibliographic databases -- Crowdsourcing -- Machine learning -- Subject-wide evidence synthesis -- Systematic maps -- Systematic reviews; Economic forecasting -- Periodicals; Technological forecasting -- Periodicals; Economic policy -- Periodicals; Prévision économique -- Périodiques; Prévision technologique -- Périodiques; Politique économique -- Périodiques; Economic forecasting; Economic policy; Technological forecasting; Periodicals; Electronic journals; 330.0112
• DOI: 10.1016/j.futures.2019.102508
• Languages: English
• ISSNs: 0016-3287

Abstract

Highlights: Crowdsourcing: 51 participants assessed 10, 001 publications that are potentially relevant to the study of existential risk. Machine learning: new publications are automatically assessed by a neural network and verified by participants. The "living bibliography" is updated every month, and it is freely available online at www.x-risk.net . Abstract: The study of existential risk — the risk of human extinction or the collapse of human civilization — has only recently emerged as an integrated field of research, and yet an overwhelming volume of relevant research has already been published. To provide an evidence base for policy and risk analysis, this research should be systematically reviewed. In a systematic review, one of many time-consuming tasks is to read the titles and abstracts of research publications, to see if they meet the inclusion criteria. We show how this task can be shared between multiple people (using crowdsourcing) and partially automated (using machine learning), as methods of handling an overwhelming volume of research. We used these methods to create The Existential Risk Research Assessment (TERRA), which is a living bibliography of relevant publications that gets updated each month (www.x-risk.net ). We present the results from the first ten months of TERRA, in which 10, 001 abstracts were screened by 51 participants. Several challenges need to be met before these methods can be used in systematic reviews. However, we suggest that collaborative and cumulative methods such as these will need to be used in systematic reviews as the volume of research increases.