Towards higher efficiency in a distributed memory storage system using data compression. (28th December 2022)

• Record Type: Journal Article
• Authors: Yu, Xiaoyang; Lu, Songfeng; Wang, Tongyang; Zhang, Xinfang; Wan, Shaohua
• Is Part Of: International journal of bio-inspired computation. Volume 20:Number 4(2022)
• Page Start: 232
• Page End: 240
• Publication Date: 2022-12-28
• Subjects: data storage systems -- system performance -- data compression -- distributed memory storage; Biologically-inspired computing -- Periodicals; Computational biology -- Periodicals; 572.0285
• Languages: English
• ISSNs: 1758-0366

Abstract

As the amount of data grows, achieving an appropriate trade-off among computation, storage and network transportation will be beneficial for a distributed memory storage system, leading to higher overall efficiency. To this end, we explore a method to achieve this trade-off by introducing data compression technology in a transparent manner. Instead of focusing on specific compressed data structures, we target block level compression for a general-purpose storage system to incorporate a wide range of existing data analysis frameworks and usage scenarios, especially with big data. A prototype is implemented and evaluated based on the memory-centric distributed storage system Alluxio to provide transparent compression and decompression during write/read operations. The extensive experiments for data with different types of compression ratio are conducted and the experimental results prove that our approach can achieve huge write/read throughput.