Validity Threats in Quantitative Data Collection With Games: A Narrative Survey. (June 2019)

• Record Type: Journal Article
• Authors: Gundry, David; Deterding, Sebastian
• Other Names: Slegers Karin guest-editor.; Bleumers Lizzy guest-editor.; Maurer Bernhard guest-editor.; Krischkowsky Alina guest-editor.; Blythe Mark guest-editor.
• Is Part Of: Simulation & gaming. Volume 50:Number 3(2019)
• Page Start: 302
• Page End: 328
• Publication Date: 2019-06
• Subjects: applied games -- data collection -- game-based methods -- games with a purpose -- research gamification -- validity -- validity threats; Social sciences -- Study and teaching -- Periodicals; Simulation methods -- Periodicals; Game theory -- Periodicals; Sciences sociales -- Étude et enseignement -- Périodiques; Simulation, Méthodes de -- Périodiques; Jeu de rôle -- Périodiques; Éducation; Jeu d'entreprise; Jeu de simulation; Méthode de simulation; Sciences sociales; Théorie des jeux; Périodique électronique (Descripteur de forme); Ressource Internet (Descripteur de forme); 003.3
• DOI: 10.1177/1046878118805515
• Languages: English
• ISSNs: 1046-8781

Abstract

Background. Games are increasingly used to collectscientific data . Some suggest that game features like high cognitive load may limit the inferences we can draw from such data, yet no systematic overview exists of potential validity threats ofgame-based methods . Aim. We present anarrative survey of documented and potential threats to validity in using games forquantitative data collection . Method. We combined an unsystematic bottom-upliterature review with a systematic top-down application of standard validity threat typologies to games to arrive at asystematisation of game-characteristicvalidity threats . Results. We identify three game characteristics that potentially impact validity: Games arecomplex systems, impeding the predictable control and isolation of treatments. They are rich in unwantedvariance and diversity. And theirsocial framing can differ from and interact with the framing of research studies or non-game situations they are supposed to represent. The diversity of gamers and their differences to general populations bring further complications. Discussion and Conclusions. The wealth of potential validity threats in game-based research is met by a dearth of systematic methodological studies, leading us to outline several futureresearch directions.