Feature‐based high‐availability mechanism for quantile tasks in real‐time data stream processing. (17th December 2013)

• Record Type: Journal Article
• Authors: Ding, Weilong; Han, Yanbo; Wang, Jing; Zhao, Zhuofeng; Ranjan, Rajiv; Buyya, Rajkumar; Leitner, Philipp; Haller, Armin; Tai, Stefan
• Is Part Of: Software, practice & experience. Volume 44:Number 7(2014)
• Page Start: 855
• Page End: 871
• Publication Date: 2013-12-17
• Subjects: Computer software -- Periodicals; Computer programming -- Periodicals; Computer programs -- Periodicals; 005.3
• DOI: 10.1002/spe.2244
• Languages: English
• ISSNs: 0038-0644

Abstract

<abstract abstract-type="main"> <title>SUMMARY</title> <p>Under distributed Cloud environment, the real‐time and continuous data stream makes the availability during processing essential but expensive. For aggregation tasks of data stream processing systems, traditional replica‐based high‐availability mechanisms require large overheads at run‐time and long recovery latency at fail‐time, because of specific nature of aggregations. In this paper, we focus on the typical quantile tasks and propose a feature‐based high‐availability mechanism to reduce related overhead and the latency. With the help of monitor module, quantile feature is maintained incrementally through histogram synopsis over time‐based sliding window, and the failed quantile tasks can be recovered precisely with high probability in an efficient way. The effectiveness has been analyzed theoretically, and meanwhile, the acceptable tradeoff between overheads and performance has been demonstrated by comprehensive experiments on both synthetic and real data. Copyright © 2013 John Wiley &amp; Sons, Ltd.</p> </abstract>