Evaluation of in‐memory storage engine for machine learning analysis of security events‡. (5th March 2016)

• Record Type: Journal Article
• Authors: Sapegin, Andrey; Gawron, Marian; Jaeger, David; Cheng, Feng; Meinel, Christoph
• Other Names: Grosu Daniel guestEditor.; Jin Hai guestEditor.; Maheshwari Ketan guestEditor.; Katz Daniel guestEditor.; Olabarriaga Silvia D. guestEditor.; Wozniak Justin guestEditor.; Thain Douglas guestEditor.
• Is Part Of: Concurrency and computation. Volume 29:Number 2(2017)
• Page Start: n/a
• Page End: n/a
• Publication Date: 2016-03-05
• Subjects: intrusion detection -- SAP HANA -- in memory -- security -- machine learning; Parallel processing (Electronic computers) -- Periodicals; Parallel computers -- Periodicals; 004.35
• DOI: 10.1002/cpe.3800
• Languages: English
• ISSNs: 1532-0626

Abstract

Summary: Modern security information and event management systems should be capable to store and process high amount of events or log messages in different formats and from different sources. This requirement often prevents such systems from usage of computational heavy algorithms for security analysis. To deal with this issue, we built our system based on an in‐memory database with an integrated machine learning library, namely, SAP HANA. Three approaches, that is, (1) deep normalisation of log messages, (2) storing data in the main memory and (3) running data analysis directly in the database, allow us to increase processing speed in such a way that machine learning analysis of security events becomes possible nearly in real time. Besides that, we developed a universal anomaly detection algorithm, which uses vector space model to represent and cluster textual log messages. Together with deep normalisation approach, this algorithm solves the problem of correlation for heterogenous security events containing many text fields. To prove our concepts, we measured the processing speed for the developed system on the data generated using Active Directory testbed, compared it with classical system architecture based on PostgreSQL database and showed the efficiency of our approach for high‐speed analysis of security events. Copyright © 2016 John Wiley & Sons, Ltd.