A divide-and-conquer strategy to solve the out-of-memory problem of processing thousands of Affymetrix microarrays. (23rd December 2008)

• Record Type: Journal Article
• Authors: Lee, Chia-Ju; Fu, Dong; Du, Pan; Jiang, Hongmei; Lin, Simon; Kibbe, Warren
• Is Part Of: International journal of computational biology and drug design. Volume 1:Number 4(2008)
• Page Start: 396
• Page End: 405
• Publication Date: 2008-12-23
• Subjects: divide-and-conquer -- resampling -- out-of-memory -- microarrays -- Affymetrix arrays -- R&amp -- #47 -- Bioconductor -- supercomputers -- computer clusters -- bioinformatics; Computational biology -- Periodicals; Drugs -- Design -- Periodicals; 570.285
• Languages: English
• ISSNs: 1756-0756

Abstract

Out-of-memory problem was frequently encountered when processing thousands of CEL files using Bioconductor. We propose a divide-and-conquer strategy combined with randomised resampling to solve this problem. The CAMDA 2007 META-analysis data set which contains 5896 CEL files was used to test the approach on a typical commodity computer cluster by running established pre-processing algorithms for Affymetrix arrays in the Bioconductor package. The results were validated against a golden standard obtained by using a supercomputer. In addition to the performance improvement, the general divide-and-conquer strategy can be applied to any other normalisation algorithms without modifying the underlying implementation.