A Text Similarity Measurement Based on Semantic Fingerprint of Characteristic Phrases. Issue 2 (1st March 2020)

• Record Type: Journal Article
• Authors: Pang, Shanchen; Yao, Jiamin; Liu, Ting; Zhao, Hua; Chen, Hongqi
• Is Part Of: Chinese journal of electronics. Volume 29:Issue 2(2020)
• Page Start: 233
• Page End: 241
• Publication Date: 2020-03-01
• Subjects: Term frequency‐Inverse document frequency(TF‐IDF) model -- Semantic fingerprint -- Similarity -- Characteristic phrases; computational linguistics -- natural language processing -- text analysis; feature words -- semantic similarities -- HowNet semantic dictionary -- semantic fingerprint -- digital fingerprinting method -- characteristic phrases -- large‐scale similarity detection methods -- digital fingerprint -- detection speed -- Chinese text similarity measurement -- feature phrase semantics -- natural language processing technology -- text preprocessing -- term frequency‐inverse document frequency; Electronics -- Periodicals; Electronics -- China -- Periodicals; Electronics; China; Periodicals; 621.38105
• DOI: 10.1049/cje.2019.12.011
• Languages: English
• ISSNs: 1022-4653

Abstract

Abstract : Text similarity measurements are the basis for measuring the degree of matching between two or more texts. Traditional large‐scale similarity detection methods based on a digital fingerprint have the advantage of high detection speed, which are only suitable for accurate detection. We propose a method of Chinese text similarity measurement based on feature phrase semantics. Natural language processing (NLP) technology is used to pre‐process text and extract the keywords by the Term frequency‐Inverse document frequency (TF‐IDF) model and further screen out the feature words. We get the exact meaning of a word and semantic similarities between words and a HowNet semantic dictionary. We substitute concepts to get the feature phrases and generate a semantic fingerprint and calculate similarity. The experimental results indicate that the method proposed is superior in similarity detection in terms of its accuracy rate, recall rate, and F ‐value to the traditional and digital fingerprinting method.