Improving word embedding quality with innovative automated approaches to hyperparameters. (21st January 2021)

• Record Type: Journal Article
• Authors: Yildiz, Beytullah; Tezgider, Murat
• Other Names: Oh Sangyoon guestEditor.; de Camargo Raphael Y. guestEditor.; Marozzo Fabrizio guestEditor.; Martins Wellington guestEditor.; Kołodziej Joanna guestEditor.; Jaatun Martin Gilje guestEditor.
• Is Part Of: Concurrency and computation. Volume 33:Number 18(2021)
• Page Start: n/a
• Page End: n/a
• Publication Date: 2021-01-21
• Subjects: deep learning -- machine learning -- text analysis -- text classification -- word embedding -- word2vec; Parallel processing (Electronic computers) -- Periodicals; Parallel computers -- Periodicals; 004.35
• DOI: 10.1002/cpe.6091
• Languages: English
• ISSNs: 1532-0626

Abstract

Abstract: Deep learning practices have a great impact in many areas. Big data and significant hardware developments are the main reasons behind deep learning success. Recent advances in deep learning have led to significant improvements in text analysis and classification. Progress in the quality of word representation is an important factor among these improvements. In this study, we aimed to develop word2vec word representation, also called embedding, by automatically optimizing hyperparameters. Minimum word count, vector size, window size, negative sample, and iteration number were used to improve word embedding. We introduce two approaches for setting hyperparameters that are faster than grid search and random search. Word embeddings were created using documents of approximately 300 million words. We measured the quality of word embedding using a deep learning classification model on documents of 10 different classes. It was observed that the optimization of the values of hyperparameters alone increased classification success by 9%. In addition, we demonstrate the benefits of our approaches by comparing the semantic and syntactic relations between word embedding using default and optimized hyperparameters.