Big data and machine learning framework for clouds and its usage for text classification. (21st December 2020)

• Record Type: Journal Article
• Authors: Pintye, István; Kail, Eszter; Kacsuk, Péter; Lovas, Róbert
• Other Names: Ogiela Marek R. guestEditor.; Rahayu Wenny guestEditor.; Palmieri Francesco guestEditor.; Kalyanam Rajesh guestEditor.; Stankovski Vlado guestEditor.
• Is Part Of: Concurrency and computation. Volume 33:Number 19(2021)
• Page Start: n/a
• Page End: n/a
• Publication Date: 2020-12-21
• Subjects: big data -- cloud -- machine learning -- parallel and distributed execution -- reference architectures -- text classification; Parallel processing (Electronic computers) -- Periodicals; Parallel computers -- Periodicals; 004.35
• DOI: 10.1002/cpe.6164
• Languages: English
• ISSNs: 1532-0626

Abstract

Abstract: Reference architectures for big data and machine learning include not only interconnected building blocks but important considerations (among others) for scalability, manageability and usability issues as well. Leveraging on such reference architectures, the automated deployment of distributed toolsets and frameworks on various clouds is still challenging due to the diversity of technologies and protocols. The paper focuses particularly on the widespread Apache Spark cluster with Jupyter as the particularly addressed framework, and the Occopus cloud‐agnostic orchestrator tool for automating its deployment and maintenance stages. The presented approach has been demonstrated and validated with a new, promising text classification application on the Hungarian academic research infrastructure, the OpenStack‐based MTA Cloud. The paper explains the concept, the applied components, and illustrates their usage with real use‐case measurements.