DeepMoM: Robust Deep Learning With Median-of-Means. Issue 1 (2nd January 2023)

• Record Type: Journal Article
• Authors: Huang, Shih-Ting; Lederer, Johannes
• Is Part Of: Journal of computational and graphical statistics. Volume 32:Issue 1(2023)
• Page Start: 181
• Page End: 195
• Publication Date: 2023-01-02
• Subjects: Deep learning -- Median-of-means -- Robust estimator; Mathematical statistics -- Data processing -- Periodicals; Mathematical statistics -- Graphic methods -- Periodicals; 519.50285
• DOI: 10.1080/10618600.2022.2090947
• Languages: English
• ISSNs: 1061-8600

Abstract

Abstract: Data used in deep learning is notoriously problematic. For example, data are usually combined from diverse sources, rarely cleaned and vetted thoroughly, and sometimes corrupted on purpose. Intentional corruption that targets the weak spots of algorithms has been studied extensively under the label of "adversarial attacks." In contrast, the arguably much more common case of corruption that reflects the limited quality of data has been studied much less. Such "random" corruptions are due to measurement errors, unreliable sources, convenience sampling, and so forth. These kinds of corruption are common in deep learning, because data are rarely collected according to strict protocols—in strong contrast to the formalized data collection in some parts of classical statistics. This article concerns such corruption. We introduce an approach motivated by very recent insights into median-of-means and Le Cam's principle, we show that the approach can be readily implemented, and we demonstrate that it performs very well in practice. In conclusion, we believe that our approach is a very promising alternative to standard parameter training based on least-squares and cross-entropy loss.