A meta-evaluation model on science and technology project review experts using IVIF-BWM and MULTIMOORA. (15th April 2021)

• Record Type: Journal Article
• Authors: Wang, Jian; Ma, Qianqian; Liu, Hu-Chen
• Is Part Of: Expert systems with applications. Volume 168(2021)
• Publication Date: 2021-04-15
• Subjects: Science and technology project review -- Meta-evaluation model -- Interval-valued Intuitionistic Fuzzy (IVIF) -- Best Worst Method (BWM) -- MULTIMOORA method; Expert systems (Computer science) -- Periodicals; Systèmes experts (Informatique) -- Périodiques; Electronic journals; 006.33
• DOI: 10.1016/j.eswa.2020.114236
• Languages: English
• ISSNs: 0957-4174

Abstract

Abstract: Meta evaluation theory and methods were used to evaluate the review experts of science and technology projects. Dozens of meta evaluation criteria can be found within two categories: the objective data of the experts' review results, i.e. the coefficient of deviation, the Spearman rank correlation coefficient, the reliability coefficient of binary value; the subjective data of the experts, i.e., the degree of experts' participation, the degree of review punctuality, the qualification of experts, and the degree of review seriousness. How these criteria impact each other has few been examined, and how to integrate the objective and subjective criteria need a comprehensive model with considering the fuzzy characteristics of the subjective criteria. This study targeted these two questions. An empirical study was adopted with hundreds of experts taking part in reviewing hundreds of Sci-Tech projects. The impacting relationships among the criteria were analyzed based on the empirical study. In order to deal with the intuitionistic fuzzy data on the subjective criteria and improve the estimation efficiency, an IVIF-BWM (best worst method under interval-valued intuitionistic fuzzy environment) was proposed by combining IVIF and the classical BWM to generate the importance weight for each criterion. The MULTIMOORA (multi-objective optimization by ratio analysis plus the full multiplicative form) was used to determine how to combine these criteria. At last, the proposed meta-evaluation model based on IVIF-BWM and MULTIMOORA was applied in a real case. The case study results supported the accuracy and reliability of the proposed model. Highlights: A meta-evaluation model was established to evaluate the technology project review experts. Based on the empirical results, seven meta-evaluation criteria were selected and modified for the model. BWM was improved in interval valued intuitionistic fuzzy environment, and a method called as IVIF-BWM was proposed. A comprehensive meta-evaluation model was founded based on IVIF-BWM and MULTIMOORA.