On the computation of distribution-free performance bounds: Application to small sample sizes in neuroimaging. (September 2019)

• Record Type: Journal Article
• Authors: Górriz, Juan M.; Ramirez, Javier; Suckling, John
• Is Part Of: Pattern recognition. Volume 93(2019:Sep.)
• Page Start: 1
• Page End: 13
• Publication Date: 2019-09
• Subjects: Resubsitution error estimate -- Lineal classifiers -- Upper bounds -- Neuroimaging -- VC dimension; Pattern perception -- Periodicals; Perception des structures -- Périodiques; Patroonherkenning; 006.4
• DOI: 10.1016/j.patcog.2019.03.032
• Languages: English
• ISSNs: 0031-3203

Abstract

Highlights: Practical and novel upper bounds for the resubstitution error estimate are derived. Based on classical combinatorial geometry with connection to Vapnik's theory. Experiments on synthetic and neuroimaging data demonstrate the performance of resubstitution error estimators. Under heterogeneous scenarios their performance is similar or greater to that obtained by cross-validation method. Abstract: In this paper we derive practical and novel upper bounds for the resubstitution error estimate by assessing the number of linear decision functions within the problem of pattern recognition in neuroimaging. Linear classifiers and regressors have been considered in many fields, where the number of predictors far exceeds the number of training samples available, to overcome the limitations of high complexity models in terms of computation, interpretability and overfitting. Typically in neuroimaging this is the rule rather than the exception, since the dimensionality of each observation (millions of voxels) in relation to the number of available samples (hundred of scans) implies a high risk of overfitting. Based on classical combinatorial geometry, we estimate the number of hyperplanes or linear decision rules and the corresponding distribution-independent performance bounds, comparing it to those obtained by the use of the VC-dimension concept. Experiments on synthetic and neuroimaging data demonstrate the performance of resubstitution error estimators, which are often overlooked in heterogeneous scenarios where their performance is similar to that obtained by cross-validation methods.