Efficient n-gram, Skipgram and Flexgram Modelling with Colibri Core. Issue 1 (2nd August 2016)

• Record Type: Journal Article
• Authors: van Gompel, Maarten; van den Bosch, Antal
• Is Part Of: Journal of open research software. Volume 4:Issue 1(2016)
• Publication Date: 2016-08-02
• Subjects: Natural Language Processing -- n-grams -- skipgrams -- corpus frequency -- corpus analysis; Computer software -- Reusability -- Periodicals; Open source software -- Periodicals; 005
• DOI: 10.5334/jors.105
• Languages: English
• ISSNs: 2049-9647

Abstract

Counting n-grams lies at the core of any frequentist corpus analysis and is often considered a trivial matter. Going beyond consecutive n-grams to patterns such as skipgrams and flexgrams increases the demand for efficient solutions. The need to operate on big corpus data does so even more. Lossless compression and non-trivial algorithms are needed to lower the memory demands, yet retain good speed. Colibri Core is software for the efficient computation and querying of n-grams, skipgrams and flexgrams from corpus data. The resulting pattern models can be analysed and compared in various ways. The software offers a programming library for C++ and Python, as well as command-line tools.