Scoring vs. Ranking: An Experimental Study of Idea Evaluation Processes. Issue 1 (10th July 2018)

• Record Type: Journal Article
• Authors: Cui, Zhijian; Kumar PM, Shijith; Gonçalves, Dilney
• Is Part Of: Production and operations management. Volume 28:Issue 1(2019)
• Page Start: 176
• Page End: 188
• Publication Date: 2018-07-10
• Subjects: idea evaluation -- experimental study -- data‐driven research -- ranking -- scoring; Production management -- Periodicals; 658.505
• DOI: 10.1111/poms.12910
• Languages: English
• ISSNs: 1059-1478

Abstract

Abstract : Using an online experiment, this study examines the efficacy of two idea evaluation processes: scoring vs. ranking in the context of innovation management. In the scoring process, the evaluators are asked to rate the quality of each idea by assigning it a score (e.g., from 0 to 10), while in the ranking process the evaluator simply orders all ideas according to their perceived qualities. Our results suggest that the scoring process strictly outperforms the ranking process in terms of the likelihood of selecting the highest‐quality ideas. This result remains robust, irrespective of the possibility of allowing ties in the ranking process. However, when the number of ideas to be evaluated is reduced from eight to three, the efficacies of the two idea evaluation processes become similar. Additionally, we find that the efficacy of the ranking process is significantly improved when additional information is provided, yet the efficacy of the scoring process does not change with more information. Based on the observations from the experimental data, we propose and test an explanatory model in which the information becomes a cue that directs the participants' efforts to evaluate the ideas (i.e., the time taken for evaluation).