A framework for scalable real‐time anomaly detection over voluminous, geospatial data streams. (28th March 2017)

• Record Type: Journal Article
• Authors: Budgaga, Walid; Malensek, Matthew; Lee Pallickara, Sangmi; Pallickara, Shrideep
• Is Part Of: Concurrency and computation. Volume 29:Number 12(2017)
• Page Start: n/a
• Page End: n/a
• Publication Date: 2017-03-28
• Subjects: distributed geospatial anomaly detection -- online anomaly detection -- spatiotemporal data streams time series analytics; Parallel processing (Electronic computers) -- Periodicals; Parallel computers -- Periodicals; 004.35
• DOI: 10.1002/cpe.4106
• Languages: English
• ISSNs: 1532-0626

Abstract

Summary: This study presents a framework to enable distributed detection, storage, and analysis of anomalies in voluminous data streams. Individual observations within these streams are multidimensional, with each dimension corresponding to a feature of interest. We consider time‐series geospatial datasets generated by remote and in situ observational devices. Three aspects make this problem particularly challenging: (1) the cumulative volume and rates of data arrivals, (2) evolution of the datasets over time, and (3) spatiotemporal correlations associated with the data. Further, solutions must minimize user intervention and be amenable to distributed processing to ensure scalability. Our approach achieves accurate, high‐throughput classifications in real time, which we demonstrate with our reference anomaly detector implementations. We also provide interfaces that allow new implementations to be developed and parallelized automatically, ensuring applicability across problem domains. To help quantify the magnitude of anomalous observations, detector implementations provide a corresponding degree of irregularity. We have incorporated these algorithms into our distributed storage platform, Galileo, and profiled their suitability through empirical analysis that demonstrates high throughput (10 000 observations per‐second, per‐node) on a real‐worl Petabyte dataset.