Emerging Trends: SOTA-Chasing. (March 2022)
- Record Type:
- Journal Article
- Title:
- Emerging Trends: SOTA-Chasing. (March 2022)
- Main Title:
- Emerging Trends: SOTA-Chasing
- Authors:
- Church, Kenneth Ward
Kordoni, Valia - Abstract:
- Abstract: Many papers are chasing state-of-the-art (SOTA) numbers, and more will do so in the future. SOTA-chasing comes with many costs. SOTA-chasing squeezes out more promising opportunities such as coopetition and interdisciplinary collaboration. In addition, there is a risk that too much SOTA-chasing could lead to claims of superhuman performance, unrealistic expectations, and the next AI winter. Two root causes for SOTA-chasing will be discussed: (1) lack of leadership and (2) iffy reviewing processes. SOTA-chasing may be similar to the replication crisis in the scientific literature. The replication crisis is yet another example, like evaluation, of over-confidence in accepted practices and the scientific method, even when such practices lead to absurd consequences.
- Is Part Of:
- Natural language engineering. Volume 28:Number 2(2022)
- Journal:
- Natural language engineering
- Issue:
- Volume 28:Number 2(2022)
- Issue Display:
- Volume 28, Issue 2 (2022)
- Year:
- 2022
- Volume:
- 28
- Issue:
- 2
- Issue Sort Value:
- 2022-0028-0002-0000
- Page Start:
- 249
- Page End:
- 269
- Publication Date:
- 2022-03
- Subjects:
- State-of-the-art -- Evaluation -- Benchmarks -- Leaderboards -- Root causes -- Leadership -- Reviewing -- Replication crisis
Natural language processing (Computer science) -- Periodicals
Software engineering -- Periodicals
006.35 - Journal URLs:
- http://journals.cambridge.org/action/displayJournal?jid=NLE ↗
- DOI:
- 10.1017/S1351324922000043 ↗
- Languages:
- English
- ISSNs:
- 1351-3249
- Deposit Type:
- Legaldeposit
- View Content:
- Available online (eLD content is only available in our Reading Rooms) ↗
- Physical Locations:
- British Library HMNTS - ELD Digital store
- Ingest File:
- 20656.xml