Emerging Trends: SOTA-Chasing. (March 2022)

• Record Type: Journal Article
• Authors: Church, Kenneth Ward; Kordoni, Valia
• Is Part Of: Natural language engineering. Volume 28:Number 2(2022)
• Page Start: 249
• Page End: 269
• Publication Date: 2022-03
• Subjects: State-of-the-art -- Evaluation -- Benchmarks -- Leaderboards -- Root causes -- Leadership -- Reviewing -- Replication crisis; Natural language processing (Computer science) -- Periodicals; Software engineering -- Periodicals; 006.35
• DOI: 10.1017/S1351324922000043
• Languages: English
• ISSNs: 1351-3249

Abstract

Abstract: Many papers are chasing state-of-the-art (SOTA) numbers, and more will do so in the future. SOTA-chasing comes with many costs. SOTA-chasing squeezes out more promising opportunities such as coopetition and interdisciplinary collaboration. In addition, there is a risk that too much SOTA-chasing could lead to claims of superhuman performance, unrealistic expectations, and the next AI winter. Two root causes for SOTA-chasing will be discussed: (1) lack of leadership and (2) iffy reviewing processes. SOTA-chasing may be similar to the replication crisis in the scientific literature. The replication crisis is yet another example, like evaluation, of over-confidence in accepted practices and the scientific method, even when such practices lead to absurd consequences.