Overfitting in linear feature extraction for classification of high-dimensional image data. (May 2016)

• Record Type: Journal Article
• Authors: Liu, Raymond; Gillies, Duncan F.
• Is Part Of: Pattern recognition. Volume 53(2016:May)
• Page Start: 73
• Page End: 86
• Publication Date: 2016-05
• Subjects: Dimensionality reduction -- Feature extraction -- Classification -- High-dimensional datasets -- Overfitting; Pattern perception -- Periodicals; Perception des structures -- Périodiques; Patroonherkenning; 006.4
• DOI: 10.1016/j.patcog.2015.11.015
• Languages: English
• ISSNs: 0031-3203

Abstract

Abstract: Overfitting has been widely studied in the context of classification and regression. In this paper, we study the overfitting in the context of dimensionality reduction. We show that the conventional wisdom of improving classification performance by maximising inter-class discrimination is not valid for high-dimensional datasets, and can lead to severe overfitting. In particular, we prove the theoretical existence of perfectly discriminative subspace projections, and show that for datasets with very high input dimensionality, inter-class discrimination should be reduced rather than maximised. This naturally leads to a simple dimensionality reduction technique, which we call Soft Discriminant Maps, which we use to show a direct relationship between the classification performance and the level of inter-class discrimination of feature extractors. Moreover, Soft Discriminant Maps consistently exhibit better classification performance than other comparable techniques. Abstract : Highlights: The causes of over-fitting in feature extraction for high-dimensional datasets are revealed. We prove the theoretical existence of perfectly discriminative subspace projections. Direct, inverse relationship between the classification performance the levels of inter-class discrimination. Soft Discriminant Maps consistently performs better than other comparable techniques.