Multiple stream job performance optimization with source operator graph transformations. (6th January 2020)

• Record Type: Journal Article
• Authors: Dayarathna, Miyuru; Suzumura, Toyotaro
• Other Names: Wu Yulei guestEditor.; Yan Zheng guestEditor.; Zhao Zhiwei guestEditor.; Al‐Dubai Ahmed guestEditor.; Li Tonglin guestEditor.; Xie Bing guestEditor.; Zhang Boyu guestEditor.
• Is Part Of: Concurrency and computation. Volume 32:Number 16(2020)
• Page Start: n/a
• Page End: n/a
• Publication Date: 2020-01-06
• Subjects: multiquery optimization -- operator fusion -- parallel streams pattern -- performance optimization -- scalability -- software automatic tuning -- stream processing; Parallel processing (Electronic computers) -- Periodicals; Parallel computers -- Periodicals; 004.35
• DOI: 10.1002/cpe.5658
• Languages: English
• ISSNs: 1532-0626

Abstract

Summary: Multiple distributed stream queries, which are executed on stream processing systems, need to be fine‐tuned to the compute cluster in order to harness the full potential of the hardware they run on. In this paper, we describe an automatic technique for conducting such stream query optimization in the presence of multiple stream jobs. During this autotuning process, we identify the structure of each program and conduct automatic program transformation to generate optimized unified streaming jobs. The operators on the unified secondary sample application are grouped into processing elements considering their performance characteristics and the stream graph topology structure to produce high performance stream query network. We implemented this multiple stream query optimization technique on a mechanism called Tahitica. We demonstrate our approach's ability for producing optimized stream query performance by comparing it to naive deployments using two real‐world stream processing applications in the domains of health care and search advertising. Our stream query optimization approach reported 7.1% throughput performance improvement compared to a naive deployment.