Lack of Replication or Generalization? Cultural Values Explain a Question Wording Effect. (20th August 2021)

• Record Type: Journal Article
• Authors: Silber, Henning; Tvinnereim, Endre; Stark, Tobias H; Blom, Annelies G; Krosnick, Jon A; Bosnjak, Michael; Clement, Sanne Lund; Cornilleau, Anne; Cousteaux, Anne-Sophie; John, Melvin; Jonsdottir, Gudbjorg Andrea; Lawson, Karen; Lynn, Peter; Martinsson, Johan; Shamshiri-Petersen, Ditte; Tu, Su-Hao
• Is Part Of: Journal of Survey Statistics and Methodology. Volume 10:Number 5(2022)
• Page Start: 1121
• Page End: 1147
• Publication Date: 2021-08-20
• Subjects: Cross-cultural comparisons -- Generalization -- Questionnaire design -- Question wording effects -- Replication -- Survey research methods; Surveys -- Methodology -- Periodicals; Surveys -- Evaluation -- Periodicals; Sampling (Statistics) -- Periodicals; 001.433
• DOI: 10.1093/jssam/smab007
• Languages: English
• ISSNs: 2325-0984

Abstract

Abstract: In the context of the current "replication crisis" across the sciences, failures to reproduce a finding are often viewed as discrediting it. This paper shows how such a conclusion can be incorrect. In 1981, Schuman and Presser showed that including the word "freedom" in a survey question significantly increased approval of allowing a speech against religion in the USA. New experiments in probability sample surveys ( n = 23, 370) in the USA and 10 other countries showed that the wording effect replicated in the USA and appeared in four other countries (Canada, Germany, Taiwan, and the Netherlands) but not in the remaining countries. The effect appeared only in countries in which the value of freedom is especially salient and endorsed. Thus, public support for a proposition was enhanced by portraying it as embodying a salient principle of a nation's culture. Instead of questioning initial findings, inconsistent results across countries signal limits on generalizability and identify an important moderator.