Evaluating research and researchers by the journal impact factor: Is it better than coin flipping?. Issue 1 (February 2019)

• Record Type: Journal Article
• Authors: Brito, Ricardo; Rodríguez-Navarro, Alonso
• Is Part Of: Journal of informetrics. Volume 13:Issue 1(2019)
• Page Start: 314
• Page End: 324
• Publication Date: 2019-02
• Subjects: Research evaluation -- Impact factor -- Failure probability; Library statistics -- Periodicals; Information science -- Statistical methods -- Periodicals; Bibliometrics -- Periodicals; Bibliothèques -- Statistiques -- Périodiques; Sciences de l'information -- Méthodes statistiques -- Périodiques; Bibliométrie -- Périodiques; 020.727
• DOI: 10.1016/j.joi.2019.01.009
• Languages: English
• ISSNs: 1751-1577

Abstract

Highlights: The JIF is used as a surrogate of the citation for paper evaluation. We compare the evaluations of two papers by the JIF and the number of citations. JIF paper evaluation fails when the JIF evaluation contradicts citation evaluation. We find high failure probabilities in JIF evaluations, as high as 0.5 in many cases. Abstract: The journal impact factor (JIF) is the average of the number of citations of the papers published in a journal, calculated according to a specific formula; it is extensively used for the evaluation of research and researchers. The method assumes that all papers in a journal have the same scientific merit, which is measured by the JIF of the publishing journal. This implies that the number of citations measures scientific merits but the JIF does not evaluate each individual paper by its own number of citations. Therefore, in the comparative evaluation of two papers, the use of the JIF implies a risk of failure, which occurs when a paper in the journal with the lower JIF is compared to another with fewer citations in the journal with the higher JIF. To quantify this risk of failure, this study calculates the failure probabilities, taking advantage of the lognormal distribution of citations. In two journals whose JIFs are ten-fold different, the failure probability is low. However, in most cases when two papers are compared, the JIFs of the journals are not so different. Then, the failure probability can be close to 0.5, which is equivalent to evaluating by coin flipping.