Discovering and merging related analytic datasets. (July 2020)

• Record Type: Journal Article
• Authors: Liu, Rutian; Simon, Eric; Amann, Bernd; Gançarski, Stéphane
• Is Part Of: Information systems. Volume 91(2020)
• Publication Date: 2020-07
• Subjects: Schema augmentation -- Schema complement -- Data quality -- SAP HANA; Database management -- Periodicals; Electronic data processing -- Periodicals; Bases de données -- Gestion -- Périodiques; Informatique -- Périodiques; Database management; Electronic data processing; Periodicals; 005.7
• DOI: 10.1016/j.is.2020.101495
• Languages: English
• ISSNs: 0306-4379

Abstract

Abstract: The production of analytic datasets is a significant big data trend and has gone well beyond the scope of traditional IT-governed dataset development. Analytic datasets are now created by data scientists and data analysts using big data frameworks and agile data preparation tools. However, despite the profusion of available datasets, it remains quite difficult for a data analyst to start from a dataset at hand and customize it with additional attributes coming from other existing datasets. This article describes a model and algorithms that exploit automatically extracted and user-defined semantic relationships for extending analytic datasets with new atomic or aggregated attribute values. Our framework is implemented as a REST service in SAP HANA and includes a careful theoretical analysis and practical solutions for several complex data quality issues. Highlights: Attribute graphs for literal functional dependencies in hierarchical dimensions. Schema augmentation and reduction operations to eliminate tuple multiplication. Quality criteria and automatic repair operations for merging schema augmentations. Detailed description of the implementations within SAP HANA platform. Experimental evaluations on real datasets and usage scenarios.