Identifying experts in the crowd for evaluation of engineering designs. (4th May 2017)

• Record Type: Journal Article
• Authors: Burnap, Alexander; Gerth, Richard; Gonzalez, Richard; Papalambros, Panos Y.
• Is Part Of: Journal of engineering design. Volume 28:Number 5(2017)
• Page Start: 317
• Page End: 337
• Publication Date: 2017-05-04
• Subjects: Design evaluation -- crowdsourcing -- open innovation -- expertise identification; Engineering design -- Periodicals; Design, Industrial -- Periodicals; Industrial engineering -- Periodicals; 620.0042
• DOI: 10.1080/09544828.2017.1316013
• Languages: English
• ISSNs: 0954-4828

Abstract

ABSTRACT: Crowdsourcing offers the opportunity to gather evaluations on concept designs from evaluators that otherwise may not have been considered, thus leveraging additional expertise to improve decision making during early stages of the design process. Previous research has shown that crowdsourcing may fail to evaluate correctly even 'simple' engineering design concepts, because non-expert evaluations overwhelm the entire crowd evaluation. This article proposes using expertise prediction heuristics to automatically identify experts and filter non-experts prior to a crowdsourced evaluation. We conducted an experiment to test four common expertise prediction heuristics: (1) evaluator demographics, (2) evaluation reaction time, (3) mechanical reasoning aptitude, and (4) 'easy and known' versions of the actual 'difficult and unknown' design evaluation task. The results show statistical significance between variables for all four heuristics; however, most predictive power is garnered going from easy to difficult tasks. A combination of these heuristics offers a practical way to identify and filter experts from non-experts, thus improving crowdsourced evaluation of early-stage design concepts.