Empirical analysis of Zipf's law, power law, and lognormal distributions in medical discharge reports. (January 2021)

• Record Type: Journal Article
• Authors: Quiroz, Juan C; Laranjo, Liliana; Tufanaru, Catalin; Kocaballi, Ahmet Baki; Rezazadegan, Dana; Berkovsky, Shlomo; Coiera, Enrico
• Is Part Of: International journal of medical informatics. Volume 145(2021)
• Publication Date: 2021-01
• Subjects: Data mining -- MIMIC-III dataset -- Machine learning -- Maximum likelihood estimation -- Power-law with exponential cut-off -- Statistical distributions; Medical informatics -- Periodicals; Information science -- Periodicals; Computers -- Periodicals; Medical technology -- Periodicals; Medical Informatics -- Periodicals; Technology, Medical -- Periodicals; Computers; Information science; Medical informatics; Medical technology; Electronic journals; Periodicals; Electronic journals; 610.285
• DOI: 10.1016/j.ijmedinf.2020.104324
• Languages: English
• ISSNs: 1386-5056

Abstract

Graphical abstract: Medical discharge reports and subsections are best modeled by the truncated power law (with cut-off) and lognormal distributions. Bayesian nonparametric models for modeling power law behavior are likely to improve modeling efficacy. Highlights: Discharge reports are best modeled by truncated power law and lognormal distributions. Using lemma forms and removing stop words yielded consistent results. Discharge report modeling will benefit from power law Bayesian non-parametric models. Abstract: Background: Bayesian modelling and statistical text analysis rely on informed probability priors to encourage good solutions. Objective: This paper empirically analyses whether text in medical discharge reports follow Zipf's law, a commonly assumed statistical property of language where word frequency follows a discrete power-law distribution. Method: We examined 20, 000 medical discharge reports from the MIMIC-III dataset. Methods included splitting the discharge reports into tokens, counting token frequency, fitting power-law distributions to the data, and testing whether alternative distributions—lognormal, exponential, stretched exponential, and truncated power-law—provided superior fits to the data. Result: Discharge reports are best fit by the truncated power-law and lognormal distributions. Discharge reports appear to be near-Zipfian by having the truncated power-law provide superior fits over a pure power-law. Conclusion: Our findings suggest that Bayesian modelling and statistical text analysis of discharge report text would benefit from using truncated power-law and lognormal probability priors and non-parametric models that capture power-law behavior.