Effect of experience sampling schedules on response rate and recall accuracy of objective self-reports. Issue 125 (May 2019)

• Record Type: Journal Article
• Authors: van Berkel, Niels; Goncalves, Jorge; Lovén, Lauri; Ferreira, Denzil; Hosio, Simo; Kostakos, Vassilis
• Is Part Of: International journal of human-computer studies. Issue 125(2019)
• Page Start: 118
• Page End: 128
• Publication Date: 2019-05
• Subjects: Experience sampling method -- ESM -- Ecological momentary assessment -- EMA -- Self-report -- Smartphone -- Contingency -- Response rate -- Accuracy -- Mobile questionnaires -- Data quality -- Validation; Human-machine systems -- Periodicals; Systems engineering -- Periodicals; Human engineering -- Periodicals; Human engineering; Human-machine systems; Systems engineering; Periodicals; Electronic journals; 004.019
• DOI: 10.1016/j.ijhcs.2018.12.002
• Languages: English
• ISSNs: 1071-5819

Abstract

Research highlights: We investigate the effect of random, interval, and smartphone-unlock based questionnaires Scheduling configuration affects participants' response rates Recall accuracy of participants is affected by different scheduling configurations The use of combined scheduling techniques should be considered in future ESM studies Abstract: The Experience Sampling Method is widely used to collect human labelled data in the wild. Using this methodology, study participants repeatedly answer a set of questions, constructing a rich overview of the studied phenomena. One of the methodological decisions faced by researchers is deciding on the question scheduling. The literature defines three distinct schedule types: randomised, interval-based, or event-based (in our case, smartphone unlock). However, little evidence exists regarding the side-effects of these schedules on response rate and recall accuracy, and how they may bias study findings. We evaluate the effect of these three contingency configurations in a 3-week within-subjects study (N = 20). Participants answered various objective questions regarding their phone usage, while we simultaneously establish a ground-truth through smartphone instrumentation. We find that scheduling questions on phone unlock yields a higher response rate and accuracy. Our study provides empirical evidence for the effects of notification scheduling on participant responses, and informs researchers who conduct experience sampling studies on smartphones.