Assessing topic model relevance: Evaluation and informative priors. (2nd May 2019)

• Record Type: Journal Article
• Authors: Fan, Angela; Doshi‐Velez, Finale; Miratrix, Luke
• Is Part Of: Statistical analysis and data mining. Volume 12:Number 3(2019)
• Page Start: 210
• Page End: 222
• Publication Date: 2019-05-02
• Subjects: informative priors -- latent dirichlet allocation -- topic modeling; Data mining -- Statistical methods -- Periodicals; 006.312
• DOI: 10.1002/sam.11415
• Languages: English
• ISSNs: 1932-1864

Abstract

Abstract : Latent Dirichlet allocation (LDA) models trained without stopword removal often produce topics with high posterior probabilities on uninformative words, obscuring the underlying corpus content. Even when canonical stopwords are manually removed, uninformative words common in that corpus will still dominate the most probable words in a topic. In this work, we first show how the standard topic quality measures of coherence and pointwise mutual information act counter‐intuitively in the presence of common but irrelevant words, making it difficult to even quantitatively identify situations in which topics may be dominated by stopwords. We propose an additional topic quality metric that targets the stopword problem, and show that it, unlike the standard measures, correctly correlates with human judgments of quality as defined by concentration of information‐rich words. We also propose a simple‐to‐implement strategy for generating topics that are evaluated to be of much higher quality by both human assessment and our new metric. This approach, a collection of informative priors easily introduced into most LDA‐style inference methods, automatically promotes terms with domain relevance and demotes domain‐specific stop words. We demonstrate this approach's effectiveness in three very different domains: Department of Labor accident reports, online health forum posts, and NIPS abstracts. Overall we find that current practices thought to solve this problem do not do so adequately, and that our proposal offers a substantial improvement for those interested in interpreting their topics as objects in their own right.