Scalable processing and autocovariance computation of big functional data. (28th September 2017)

• Record Type: Journal Article
• Authors: Brisaboa, Nieves R.; Cao, Ricardo; Paramá, José R.; Silva‐Coira, Fernando
• Is Part Of: Software, practice & experience. Volume 48:Number 1(2018)
• Page Start: 123
• Page End: 140
• Publication Date: 2017-09-28
• Subjects: autocovariance function -- big data -- compact representation -- efficient computation -- functional data -- R package; Computer software -- Periodicals; Computer programming -- Periodicals; Computer programs -- Periodicals; 005.3
• DOI: 10.1002/spe.2524
• Languages: English
• ISSNs: 0038-0644

Abstract

Summary: This paper presents 2 main contributions. The first is a compact representation of huge sets of functional data or trajectories of continuous‐time stochastic processes, which allows keeping the data always compressed even during the processing in main memory. It is oriented to facilitate the efficient computation of the sample autocovariance function without a previous decompression of the data set, by using only partial local decoding. The second contribution is a new memory‐efficient algorithm to compute the sample autocovariance function. The combination of the compact representation and the new memory‐efficient algorithm obtained in our experiments the following benefits. The compressed data occupy in the disk 75% of the space needed by the original data. The computation of the autocovariance function used up to 13 times less main memory, and run 65% faster than the classical method implemented, for example, in the R package.