Variation in Word Frequency Distributions: Definitions, Measures and Implications for a Corpus-Based Language Typology. Issue 2 (3rd July 2017)

• Record Type: Journal Article
• Authors: Bentz, Christian; Alikaniotis, Dimitrios; Samardžić, Tanja; Buttery, Paula
• Is Part Of: Journal of quantitative linguistics. Volume 24:Issue 2/3(2017)
• Page Start: 128
• Page End: 162
• Publication Date: 2017-07-03
• Subjects: Linguistics -- Statistical methods -- Periodicals; 410.151
• DOI: 10.1080/09296174.2016.1265792
• Languages: English
• ISSNs: 0929-6174

Abstract

Abstract: Word frequencies are central to linguistic studies investigating processing difficulty, learnability, age of acquisition, diachronic transmission and the relative weight given to a concept in society. However, there are few cross-linguistic studies on entire distributions of word frequencies, and even less on systematic changes within them. Here, we first define and test an exact measure for the relative difference between distributions – the Normalised Frequency Difference (NFD) . We then apply this measure to parallel corpora in overall 19 languages, explaining systematic variation in the frequency distributions within the same language and across different languages. We further establish the NFD between lemmatised and un-lemmatised corpora as a frequency-based measure of inflectional productivity of a language. Finally, we argue that quantitative measures like the NFD can advance language typology beyond abstract, theory-driven expert judgments, towards more corpus-based, empirical and reproducible analyses.