FATM: A failure‐aware adaptive fault tolerance model for distributed stream processing systems. (4th January 2021)

• Record Type: Journal Article
• Authors: Akber, Syed Muhammad Abrar; Chen, Hanhua; Jin, Hai
• Is Part Of: Concurrency and computation. Volume 33:Number 10(2021)
• Page Start: n/a
• Page End: n/a
• Publication Date: 2021-01-04
• Subjects: checkpoints -- distributed stream processing -- failure prediction -- fault tolerance -- resilience; Parallel processing (Electronic computers) -- Periodicals; Parallel computers -- Periodicals; 004.35
• DOI: 10.1002/cpe.6167
• Languages: English
• ISSNs: 1532-0626

Abstract

Summary: Distributed Stream Processing Systems (DSPS) are very popular to process unbounded data streams in real‐time. Low processing latency is a fundamental requirement for DSPS applications to maintain the real‐time response. This requirement of low processing latency for DSPS is badly affected due to inevitable failures in computing systems. Generally, DSPS grapple with these inevitable failures by triggering periodic checkpoints. The periodic checkpoints pessimistically persist the application state so that the execution may be resumed after the failure. These periodic checkpoints incur high overheads due to the high frequency of checkpoints triggering, which increases the overall execution time. On the other hand, failure occurrences in real‐world systems are not periodic. This sharp contrast between the periodic checkpoints and failure distributions in the real‐world systems makes the periodic checkpoints inefficient. We propose a failure‐aware adaptive fault tolerance model called FATM which triggers the checkpoints inline with the underlying failure rate. Further, we design a model for utility factor and checkpoint overheads to evaluate the performance of fault tolerance models for DSPS. We implement the FATM atop Apache Flink and perform a series of experiments. To validate the effectiveness of FATM, experiment results are compared with the existing checkpoint‐based models of DSPS. The results show that the FATM significantly reduces the checkpoint frequency, increases the utility factor, and reduces the checkpoint overheads by 28%.