Which Is Better: Holdout or Full-Sample Classifier Design?. (12th December 2007)

• Record Type: Journal Article
• Authors: Brun, Marcel; Xu, Qian; Dougherty, Edward R.
• Other Names: Huang Yufei Academic Editor.
• Is Part Of: EURASIP journal on bioinformatics and systems biology. Volume 2008(2008)
• Publication Date: 2007-12-12
• Subjects: Bioinformatics -- Periodicals; Systems biology -- Periodicals; Systems Biology; Signal Processing, Computer-Assisted; Bio-informatique; Biologie systémique; Bioinformatics; Systems biology; Systems Biology; Bioinformatics; Electronic journals; Periodical; Fulltext; Internet Resources; Periodicals; Periodicals; 570.285
• DOI: 10.1155/2008/297945
• Languages: English
• ISSNs: 1687-4145

Abstract

Abstract : Is it better to design a classifier and estimate its error on the full sample or to design a classifier on a training subset and estimate its error on the holdout test subset? Full-sample design provides the better classifier; nevertheless, one might choose holdout with the hope of better error estimation. A conservative criterion to decide the best course is to aim at a classifier whose error is less than a given bound. Then the choice between full-sample and holdout designs depends on which possesses the smaller expected bound. Using this criterion, we examine the choice between holdout and several full-sample error estimators using covariance models and a patient-data model. Full-sample design consistently outperforms holdout design. The relation between the two designs is revealed via a decomposition of the expected bound into the sum of the expected true error and the expected conditional standard deviation of the true error.