Web survey experiments on matrix questions. (February 2017)

• Record Type: Journal Article
• Authors: Liu, Mingnan
• Is Part Of: Computers in human behavior. Volume 67(2017)
• Page Start: 61
• Page End: 72
• Publication Date: 2017-02
• Subjects: Web surveys -- Matrix questions -- Item nonresponse -- Missing data -- Respondent evaluation; Interactive computer systems -- Periodicals; Man-machine systems -- Periodicals; 004.019
• DOI: 10.1016/j.chb.2016.10.017
• Languages: English
• ISSNs: 0747-5632

Abstract

Abstract: Matrix or grid questions are a very common question type in web surveys although previous studies have demonstrated several measurement errors associated with this type of question. This study reports findings from four web survey experiments on testing a new matrix, called a menu matrix. A menu matrix groups several matrix questions into one large question and presents the response options using dropdown menus. The four experiments collectively show that the item nonresponse rate for the menu matrix is higher than for the regular matrix and multiple-choice questions. Also, respondents found the menu matrix to be more difficult than the regular matrix and multiple-choice questions. The time to complete is similar across the three question types. Also, the regular matrix and multiple-choice showed very similar missing data rates. The evidence speaks against the use of complex survey questions, such as the menu matrix, since they reduce data quality and the respondents' survey taking experience. Highlights: Experimentally compared menu matrix, regular matrix, and multiple-choice questions. Menu matrix consistently produced more missing data than the other two question types. Completion time was similar across three question types. Respondents judged the menu matrix to be more difficult than the other two question types.