Many Labs 5: Registered Multisite Replication of the Tempting-Fate Effects in Risen and Gilovich (2008). Issue 3 (September 2020)

• Record Type: Journal Article
• Authors: Mathur, Maya B.; Bart-Plange, Diane-Jo; Aczel, Balazs; Bernstein, Michael H.; Ciunci, Antonia M.; Ebersole, Charles R.; Falcão, Filipe; Ashbaugh, Kayla; Hilliard, Rias A.; Jern, Alan; Kellier, Danielle J.; Kessinger, Grecia; Kolb, Vanessa S.; Kovacs, Marton; Lage, Caio Ambrosio; Langford, Eleanor V.; Lins, Samuel; Manfredi, Dylan; Meyet, Venus; Moore, Don A.; Nave, Gideon; Nunnally, Christian; Palinkas, Anna; Parks, Kimberly P.; Pessers, Sebastiaan; Ramos, Tiago; Rudy, Kaylis Hase; Salamon, Janos; Shubella, Rachel L.; Silva, Rúben; Steegen, Sara; Stein, L. A. R.; Szaszi, Barnabas; Szecsi, Peter; Tuerlinckx, Francis; Vanpaemel, Wolf; Vlachou, Maria; Wiggins, Bradford J.; Zealley, David; Zrubka, Mark; Frank, Michael C.;
• Is Part Of: Advances in methods and practices in psychological science. Volume 3:Issue 3(2020)
• Page Start: 394
• Page End: 404
• Publication Date: 2020-09
• Subjects: replication -- reproducibility -- heuristic -- magical thinking -- open data -- open materials -- preregistered; Psychology -- Periodicals; Psychology -- Research -- Periodicals; 150
• DOI: 10.1177/2515245918785165
• Languages: English
• ISSNs: 2515-2459

Abstract

Risen and Gilovich (2008) found that subjects believed that "tempting fate" would be punished with ironic bad outcomes (a main effect), and that this effect was magnified when subjects were under cognitive load (an interaction). A previous replication study (Frank & Mathur, 2016) that used an online implementation of the protocol on Amazon Mechanical Turk failed to replicate both the main effect and the interaction. Before this replication was run, the authors of the original study expressed concern that the cognitive-load manipulation may be less effective when implemented online than when implemented in the lab and that subjects recruited online may also respond differently to the specific experimental scenario chosen for the replication. A later, large replication project, Many Labs 2 (Klein et al. 2018), replicated the main effect (though the effect size was smaller than in the original study), but the interaction was not assessed. Attempting to replicate the interaction while addressing the original authors' concerns regarding the protocol for the first replication study, we developed a new protocol in collaboration with the original authors. We used four university sites ( N = 754) chosen for similarity to the site of the original study to conduct a high-powered, preregistered replication focused primarily on the interaction effect. Results from these sites did not support the interaction or the main effect and were comparable to results obtained at six additional universities that were less similar to the original site. Post hoc analyses did not provide strong evidence for statistical inconsistency between the original study's estimates and our estimates; that is, the original study's results would not have been extremely unlikely in the estimated distribution of population effects in our sites. We also collected data from a new Mechanical Turk sample under the first replication study's protocol, and results were not meaningfully different from those obtained with the new protocol at universities similar to the original site. Secondary analyses failed to support proposed substantive mechanisms for the failure to replicate.