The Probability Distribution of Textual Vocabulary in the English Language. Issue 1 (2nd January 2016)

• Record Type: Journal Article
• Authors: Fengxiang, Fan; Yang, Yu; Yaqin, Wang
• Is Part Of: Journal of quantitative linguistics. Volume 23:Issue 1(2016)
• Page Start: 49
• Page End: 70
• Publication Date: 2016-01-02
• Subjects: Linguistics -- Statistical methods -- Periodicals; 410.151
• DOI: 10.1080/09296174.2015.1071149
• Languages: English
• ISSNs: 0929-6174

Abstract

Abstract: The probability of textual vocabulary is defined as the combined probabilities of the individual lemmas occurring in a text, which sum to 1 in the text but normally less than 1 in another different text. If the text is expanded the probability of the original textual vocabulary would be smaller than 1 in the expanded text. However, the present study reveals that as the text expands continually, instead of monotonically decreasing, the probability of the original textual vocabulary quickly reaches a point from which it stabilizes despite further expansion of the text. In addition, the probability of the textual vocabulary of a text occurring in other texts is not affected by the length of the texts in which they occur. Mathematical models are formulated capturing the distribution of the probability of textual vocabulary in the English language.