Putting Psychology to the Test: Rethinking Model Evaluation Through Benchmarking and Prediction. Issue 3 (September 2021)

• Record Type: Journal Article
• Authors: Rocca, Roberta; Yarkoni, Tal
• Is Part Of: Advances in methods and practices in psychological science. Volume 4:Issue 3(2021)
• Publication Date: 2021-09
• Subjects: psychology -- model evaluation -- benchmarking -- machine learning -- open data -- open materials; Psychology -- Periodicals; Psychology -- Research -- Periodicals; 150
• DOI: 10.1177/25152459211026864
• Languages: English
• ISSNs: 2515-2459

Abstract

Consensus on standards for evaluating models and theories is an integral part of every science. Nonetheless, in psychology, relatively little focus has been placed on defining reliable communal metrics to assess model performance. Evaluation practices are often idiosyncratic and are affected by a number of shortcomings (e.g., failure to assess models' ability to generalize to unseen data) that make it difficult to discriminate between good and bad models. Drawing inspiration from fields such as machine learning and statistical genetics, we argue in favor of introducing common benchmarks as a means of overcoming the lack of reliable model evaluation criteria currently observed in psychology. We discuss a number of principles benchmarks should satisfy to achieve maximal utility, identify concrete steps the community could take to promote the development of such benchmarks, and address a number of potential pitfalls and concerns that may arise in the course of implementation. We argue that reaching consensus on common evaluation benchmarks will foster cumulative progress in psychology and encourage researchers to place heavier emphasis on the practical utility of scientific models.