A configurable and executable model of Spark Streaming on Apache YARN. (3rd February 2020)

• Record Type: Journal Article
• Authors: Lin, Jia-Chun; Lee, Ming-Chang; Yu, Ingrid Chieh; Johnsen, Einar Broch
• Is Part Of: International journal of grid and utility computing. Volume 11:Number 2(2020)
• Page Start: 185
• Page End: 195
• Publication Date: 2020-02-03
• Subjects: modelling -- simulation -- Spark Streaming -- Apache YARN -- batch processing -- stream processing -- ABS; Electronic data processing -- Distributed processing -- Periodicals; Electronic commerce -- Management -- Computer programs -- Periodicals; 004.605
• Languages: English
• ISSNs: 1741-847X

Abstract

Streams of data are produced today at an unprecedented scale. Efficient and stable processing of these streams requires a careful interplay between the parameters of the streaming application and of the underlying stream processing framework. Today, finding these parameters happens by trial and error on the complex, deployed framework. This paper shows that high-level models can help to determine these parameters by predicting and comparing the performance of streaming applications running on stream processing frameworks with different configurations. To demonstrate this approach, this paper considers Spark Streaming, a widely used framework to leverage data streams on the fly and provide real-time stream processing. Technically, we develop a configurable and executable model to simulate both the streaming applications and the underlying Spark stream processing framework. Furthermore, we model the deployment of Spark Streaming on Apache YARN, which is a popular open-source distributed software framework for big data processing. We show that the developed model provides a satisfactory accuracy for predicting performance by means of empirical validation.