FlexParser—The adaptive log file parser for continuous results in a changing world. Issue 3 (27th January 2022)
- Record Type:
- Journal Article
- Title:
- FlexParser—The adaptive log file parser for continuous results in a changing world. Issue 3 (27th January 2022)
- Main Title:
- FlexParser—The adaptive log file parser for continuous results in a changing world
- Authors:
- Rücker, Nadine
Maier, Andreas - Abstract:
- Abstract: Any modern system writes events into files, called log files. Those contain crucial information which are subject to various analyses. Examples range from cybersecurity, intrusion detection over usage analyses to trouble shooting. Before data analysis is possible, desired information needs to be extracted first out of the semi‐structured log messages. State‐of‐the‐art event parsing often assumes static log events. However, any modern system is updated consistently and with updates also log file structures can change. We call those changes "mutation" and study parsing performance for different mutation cases. Latest research discovers mutations using anomaly detection post mortem, however, does not cover actual continuous parsing. Thus, we propose a novel and flexible parser, called FlexParser, which can extract desired values despite gradual changes in the log messages. It implies basic text preprocessing followed by a supervised Deep Learning method. We train a stateful LSTM on parsing one event per data set. Statefulness enforces the model to learn log message structures across several examples. Our model was tested on seven different, publicly available log file data sets and various kinds of mutations. Exhibiting an average F1‐Score of 0.98, it outperforms other Deep Learning methods as well as state‐of‐the‐art unsupervised parsers. Abstract : Log files pose an indispensable data source for various analyses. However, with continuous system development, logs areAbstract: Any modern system writes events into files, called log files. Those contain crucial information which are subject to various analyses. Examples range from cybersecurity, intrusion detection over usage analyses to trouble shooting. Before data analysis is possible, desired information needs to be extracted first out of the semi‐structured log messages. State‐of‐the‐art event parsing often assumes static log events. However, any modern system is updated consistently and with updates also log file structures can change. We call those changes "mutation" and study parsing performance for different mutation cases. Latest research discovers mutations using anomaly detection post mortem, however, does not cover actual continuous parsing. Thus, we propose a novel and flexible parser, called FlexParser, which can extract desired values despite gradual changes in the log messages. It implies basic text preprocessing followed by a supervised Deep Learning method. We train a stateful LSTM on parsing one event per data set. Statefulness enforces the model to learn log message structures across several examples. Our model was tested on seven different, publicly available log file data sets and various kinds of mutations. Exhibiting an average F1‐Score of 0.98, it outperforms other Deep Learning methods as well as state‐of‐the‐art unsupervised parsers. Abstract : Log files pose an indispensable data source for various analyses. However, with continuous system development, logs are subject to constant change. Following, rigid parsers cannot extract correct information. Thus, flexible parsing mechanisms are required. We propose FlexParser, a deep learning method which parses changing log files reliably. FlexParser yields average F1‐Score of 98% and greatly outperforms rigid parsers. … (more)
- Is Part Of:
- Journal of software. Volume 34:Issue 3(2022)
- Journal:
- Journal of software
- Issue:
- Volume 34:Issue 3(2022)
- Issue Display:
- Volume 34, Issue 3 (2022)
- Year:
- 2022
- Volume:
- 34
- Issue:
- 3
- Issue Sort Value:
- 2022-0034-0003-0000
- Page Start:
- n/a
- Page End:
- n/a
- Publication Date:
- 2022-01-27
- Subjects:
- deep learning -- flexible parsing -- log parser -- LSTM -- system log
Software engineering -- Periodicals
Computer software -- Development -- Periodicals
Software maintenance -- Periodicals
005.1 - Journal URLs:
- http://onlinelibrary.wiley.com/journal/10.1002/(ISSN)2047-7481 ↗
http://onlinelibrary.wiley.com/ ↗ - DOI:
- 10.1002/smr.2426 ↗
- Languages:
- English
- ISSNs:
- 2047-7473
- Deposit Type:
- Legaldeposit
- View Content:
- Available online (eLD content is only available in our Reading Rooms) ↗
- Physical Locations:
- British Library DSC - BLDSS-3PM
British Library HMNTS - ELD Digital store - Ingest File:
- 21162.xml