Automated validation of structured large databases: an illustration of material code bulk validation. (2016)

• Record Type: Journal Article
• Authors: Patankar, Ravindra; Dulluri, Sandeep
• Is Part Of: International journal of big data intelligence. Volume 3:Number 1(2016)
• Page Start: 38
• Page End: 50
• Publication Date: 2016
• Subjects: decision trees -- structured large databases -- automated validation -- big data -- MADlib -- Greenplum -- material codes -- bulk validation -- data validation -- machine learning; Big data -- Periodicals; 005.705
• Languages: English
• ISSNs: 2053-1389

Abstract

The accumulation of data and henceforth the storage is growing at an exponential phase owing to the decrease in the memory costs and increasingly complex business processes. With the increased data, typically there would be an increase in the complexity of validating the data. Often, the complexity and effort in validation of large scale databases grows nonlinearly with the increase in database size (Lee et al., 1999). In this paper, we discuss a novel methodology for bulk validation of large scale structured databases. The approach we propose is generic and has been tested in a real time environment. We present an illustration of validation on a material codes validation problem faced by a Fortune 100 enterprise. The demonstration would highlight the heterogeneity, and scale-scope of data validation related problems and henceforth tackling these problems effectively via application of machine learning techniques on Big Data.