TAQIH, a tool for tabular data quality assessment and improvement in the context of health data. (November 2019)

• Record Type: Journal Article
• Authors: Álvarez Sánchez, Roberto; Beristain Iraola, Andoni; Epelde Unanue, Gorka; Carlin, Paul
• Is Part Of: Computer methods and programs in biomedicine. Volume 181(2020)
• Publication Date: 2019-11
• Subjects: Data quality -- Exploratory data analysis -- Data pre-processing; Medicine -- Computer programs -- Periodicals; Biology -- Computer programs -- Periodicals; Computers -- Periodicals; Medicine -- Periodicals; Médecine -- Logiciels -- Périodiques; Biologie -- Logiciels -- Périodiques; Biology -- Computer programs; Medicine -- Computer programs; Periodicals; Electronic journals; 610.28
• DOI: 10.1016/j.cmpb.2018.12.029
• Languages: English
• ISSNs: 0169-2607

Abstract

Highlights: A web based data quality assessment and improvement tool is implemented considering the data quality dimensions. This tool also includes exploratory data analysis functionalities. Data quality dimensions are reviewed, focusing on those more relevant in the healthcare field. Data quality assessment and improvement techniques are reviewed. The proposed tool has been applied to two datasets related to drug prescriptions and a glucose monitoring system, and results are discussed. Abstract: Background and Objectives: Data curation is a tedious task but of paramount relevance for data analytics and more specially in the health context where data-driven decisions must be extremely accurate. The ambition of TAQIH is to support non-technical users on 1) the exploratory data analysis (EDA) process of tabular health data, and 2) the assessment and improvement of its quality. Methods: A web-based tool has been implemented with a simple yet powerful visual interface. First, it provides interfaces to understand the dataset, to gain the understanding of the content, structure and distribution. Then, it provides data visualization and improvement utilities for the data quality dimensions of completeness, accuracy, redundancy and readability. Results: It has been applied in two different scenarios. (1) The Northern Ireland General Practitioners (GPs) Prescription Data, an open data set containing drug prescriptions. (2) A glucose monitoring tele health system dataset. Findings on (1) include: Features that had significant amount of missing values (e.g. AMP_NM variable 53.39%); instances that have high percentage of variable values missing (e.g. 0.21% of the instances with > 75% of missing values); highly correlated variables (e.g. Gross and Actual cost almost completely correlated (∼ + 1.0)). Findings on (2) include: Features that had significant amount of missing values (e.g. patient height, weight and body mass index (BMI) (> 70%), date of diagnosis 13%)); highly correlated variables (e.g. height, weight and BMI). Full detail of the testing and insights related to findings are reported. Conclusions: TAQIH enables and supports users to carry out EDA on tabular health data and to assess and improve its quality. Having the layout of the application menu arranged sequentially as the conventional EDA pipeline helps following a consistent analysis process. The general description of the dataset and features section is very useful for the first overview of the dataset. The missing value heatmap is also very helpful in visually identifying correlations among missing values. The correlations section has proved to be supportive as a preliminary step before further data analysis pipelines, as well as the outliers section. Finally, the data quality section provides a quantitative value to the dataset improvements.