Optimising the calculation of statistical functions. (2017)

• Record Type: Journal Article
• Authors: Rodrigues, André; Silva, Carla; Borges, Paulo; Silva, Sérgio; Dutra, InÁªs
• Is Part Of: International journal of big data intelligence. Volume 4:Number 2(2017)
• Page Start: 123
• Page End: 138
• Publication Date: 2017
• Subjects: statistical data analysis -- statistical functions -- performance evaluation -- SPSS -- MATLAB -- optimisation; Big data -- Periodicals; 005.705
• Languages: English
• ISSNs: 2053-1389

Abstract

Statistical data analysis methods are well-known for their difficulty in handling large number of instances or large number of parameters. In this paper, we study popular and well-known statistical functions, generally applied to data analysis, and assess their performance as implemented by SPSS, MATLAB, R and our own software, DataIP. We use medium to large datasets and show that DataIP outperforms SPSS, MATLAB and R by several orders of magnitude. We argue that the design and implementation of these functions need to be rethought to adapt to today's data challenges.