NapierOne: A modern mixed file data set alternative to Govdocs1. (March 2022)

• Record Type: Journal Article
• Authors: Davies, Simon R.; Macfarlane, Richard; Buchanan, William J.
• Is Part Of: Forensic science international. Volume 40(2022)
• Publication Date: 2022-03
• Subjects: Corpus -- Mixed file dataset -- Govdocs1 -- Malware -- Ransomware -- Entropy -- Forensics
• DOI: 10.1016/j.fsidi.2021.301330
• Languages: English
• ISSNs: 2666-2817

Abstract

Abstract: It was found when reviewing the ransomware detection research literature that almost no proposal provided enough detail on how the test data set was created, or sufficient description of its actual content, to allow it to be recreated by other researchers interested in reconstructing their environment and validating the research results. A modern cybersecurity mixed file data set called NapierOne is presented, primarily aimed at, but not limited to, ransomware detection and forensic analysis research. NapierOne was designed to address this deficiency in reproducibility and improve consistency by facilitating research replication and repeatability. The methodology used in the creation of this data set is also described in detail. The data set was inspired by the Govdocs1 data set and it is intended that NapierOne be used as a complement to this original data set. An investigation was performed with the goal of determining the common files types currently in use. No specific research was found that explicitly provided this information, so an alternative consensus approach was employed. This involved combining the findings from multiple sources of file type usage into an overall ranked list. After which 5000 real-world example files were gathered, and a specific data subset created, for each of the common file types identified. In some circumstances, multiple data subsets were created for a specific file type, each subset representing a specific characteristic for that file type. For example, there are multiple data subsets for the ZIP file type with each subset containing examples of a specific compression method. Ransomware execution tends to produce files that have high entropy, so examples of file types that naturally have this attribute are also present. The resulting entire data set comprises of nearly 100 separate data subsets divided between 44 distinct file types, resulting in almost 500, 000 unique files in total. A description of the techniques used to gather the files for each file type is provided together with the actions that were performed on the files to confirm that they were of the highest quality and provided an accurate representation of their specific file type. Details are also provided on the content of the entire data set as well as instructions on how researchers can gain free and unlimited access to the final data set. While the data set was initially created to aid research in ransomware detection, it is sufficiently broad and diverse enough to allow for its application in many other areas of research that require a varied mixture of common real-world file examples. The NapierOne data set is an ongoing project and researchers are strongly encouraged to leverage this data set in their own research.