Text Mining with n-gram Variables. (January 2017)

• Record Type: Journal Article
• Authors: Schonlau, Matthias; Guenther, Nick; Sucholutsky, Ilia
• Is Part Of: Stata journal. Volume 17:Number 4(2017)
• Page Start: 866
• Page End: 881
• Publication Date: 2017-01
• Subjects: st0502 -- ngram -- bag of words -- sets of words -- unigram -- gram -- statistical learning -- machine learning; Statistics -- Periodicals; Statistics -- Computer programs -- Periodicals; 001.422
• DOI: 10.1177/1536867X1701700406
• Languages: English
• ISSNs: 1536-867X

Abstract

Text mining is the process of turning free text into numerical variables and then analyzing them with statistical techniques. We introduce the commandngram, which implements the most common approach to text mining, the "bag of words". An n -gram is a contiguous sequence of words in a text. Broadly speaking, ngram creates hundreds or thousands of variables, each recording how often the corresponding n -gram occurs in a given text. This is more useful than it sounds. We illustratengram with the categorization of text answers from two open-ended questions.