Sub-story detection in Twitter with hierarchical Dirichlet processes. Issue 4 (July 2017)

• Record Type: Journal Article
• Authors: Srijith, P.K.; Hepple, Mark; Bontcheva, Kalina; Preotiuc-Pietro, Daniel
• Is Part Of: Information processing & management. Volume 53:Issue 4(2017:Jul.)
• Page Start: 989
• Page End: 1003
• Publication Date: 2017-07
• Subjects: Sub-story detection -- Hierarchical dirichlet process -- Spectral clustering -- Locality sensitive hashing; Information storage and retrieval systems -- Periodicals; Information science -- Periodicals; Systèmes d'information -- Périodiques; Sciences de l'information -- Périodiques; Information science; Information storage and retrieval systems; Periodicals; 658.4038
• DOI: 10.1016/j.ipm.2016.10.004
• Languages: English
• ISSNs: 0306-4573

Abstract

Abstract: Social media has now become the de facto information source on real world events. The challenge, however, due to the high volume and velocity nature of social media streams, is in how to follow all posts pertaining to a given event over time – a task referred to as story detection. Moreover, there are often several different stories pertaining to a given event, which we refer to as sub-stories and the corresponding task of their automatic detection – as sub-story detection . This paper proposes hierarchical Dirichlet processes (HDP), a probabilistic topic model, as an effective method for automatic sub-story detection. HDP can learn sub-topics associated with sub-stories which enables it to handle subtle variations in sub-stories. It is compared with state-of-the-art story detection approaches based on locality sensitive hashing and spectral clustering. We demonstrate the superior performance of HDP for sub-story detection on real world Twitter data sets using various evaluation measures. The ability of HDP to learn sub-topics helps it to recall the sub-stories with high precision. This has resulted in an improvement of up to 60% in the F-score performance of HDP based sub-story detection approach compared to standard story detection approaches. A similar performance improvement is also seen using an information theoretic evaluation measure proposed for the sub-story detection task. Another contribution of this paper is in demonstrating that considering the conversational structures within the Twitter stream can bring up to 200% improvement in sub-story detection performance.