Emerging trends: Deep nets thrive on scale. (September 2022)

• Record Type: Journal Article
• Authors: Church, Kenneth Ward
• Is Part Of: Natural language engineering. Volume 28:Number 5(2022)
• Page Start: 673
• Page End: 682
• Publication Date: 2022-09
• Subjects: Deep nets -- Curse of dimensionality -- Blessing -- Model size -- Scale; Natural language processing (Computer science) -- Periodicals; Software engineering -- Periodicals; 006.35
• DOI: 10.1017/S1351324922000365
• Languages: English
• ISSNs: 1351-3249

Abstract

Abstract: Deep nets are becoming larger and larger in practice, with no respect for (non)-factors that ought to limit growth including the so-called curse of dimensionality (CoD). Donoho suggested that dimensionality can be a blessing as well as a curse. Current practice in industry is well ahead of theory, but there are some recent theoretical results from Weinan E's group suggesting that errors may be independent of dimensions $d$ . Current practice suggests an even stronger conjecture: deep nets are not merely immune to CoD, but actually, deep nets thrive on scale.