Transparent many‐core partitioning for high‐performance big data I/O. (19th September 2020)

• Record Type: Journal Article
• Authors: Lee, Chan‐Gyu; Cho, Joong‐Yeon; Kim, Jooho; Jin, Hyun‐Wook
• Other Names: Oh Sangyoon guestEditor.; de Camargo Raphael Y. guestEditor.; Marozzo Fabrizio guestEditor.; Martins Wellington guestEditor.; Kołodziej Joanna guestEditor.; Jaatun Martin Gilje guestEditor.
• Is Part Of: Concurrency and computation. Volume 33:Number 18(2021)
• Page Start: n/a
• Page End: n/a
• Publication Date: 2020-09-19
• Subjects: core affinity -- many‐core -- MapReduce -- network I/O -- partitioning -- storage I/O -- system calls; Parallel processing (Electronic computers) -- Periodicals; Parallel computers -- Periodicals; 004.35
• DOI: 10.1002/cpe.6017
• Languages: English
• ISSNs: 1532-0626

Abstract

Summary: As the number of cores equipped in a single computing node is rapidly increasing, utilizing many cores for contemporary applications in an efficient manner is a challenging issue. We need to consider both parallelization and locality to fully exploit many cores for multifarious operations of emerging applications. In particular, big data applications perform computation and I/O intensive operations alternately. For instance, Apache Hadoop MapReduce assumes local persistent storage for each computing node. Thus, unlike traditional parallel programming models, the MapReduce framework performs not only networking but also storage I/O. In this study, we aim to improve the locality of network and storage I/O operations on many‐core systems by partitioning cores for I/O system calls and event handlers. In order to implement fine‐grained many‐core partitioning, we decouple the system call context from the user‐level process by suggesting message‐based system calls. The suggested design provides user‐level transparency and does not require any kernel‐level modifications. In addition, we propose a scheme that dynamically decides the core affinity of system calls and event handlers by considering locality, run‐time loads, and hardware architectures. The experimental results show that the proposed many‐core partitioning can improve the locality of network and storage I/O operations in an integrated manner for MapReduce applications.