A topology‐based scaling mechanism for Apache Storm. Issue 3 (13th June 2016)

• Record Type: Journal Article
• Authors: Shieh, Ce‐Kuen; Huang, Sheng‐Wei; Sun, Li‐Da; Tsai, Ming‐Fong; Chilamkurti, Naveen; Batalla, Jordi Mongay; Mastorakis, George; Mavromoustakis, Constandinos X.
• Is Part Of: International journal of network management. Volume 27:Issue 3(2017)
• Page Start: n/a
• Page End: n/a
• Publication Date: 2016-06-13
• Subjects: Computer networks -- Management -- Periodicals; 004.6
• DOI: 10.1002/nem.1933
• Languages: English
• ISSNs: 1055-7148

Abstract

Summary: As more and more well‐known companies, such as Twitter, Yahoo, and Alibaba, start to focus on real‐time big data applications, how to build a platform for processing real‐time data becomes an important issue. Among all the real‐time processing systems, Apache Storm is the most well‐known and representative open‐source, distributed, real‐time computation system. In Storm, the computation is implemented by a topology such as a graph where nodes are operators and edges represent the data flows between operators. In big data processing and analysis systems, scalability is an important issue. Storm provides rebalance mechanism for its scalability property, which can adjust the parallelism of a running topology. However, there are some drawbacks in rebalance command, such as resource usage restriction and topology execution suspension. In this paper, we propose a topology‐based scaling mechanism for Apache Storm. When a topology is overloaded, it scales by adjusting the number of the cloned topologies or replaced by another new topology with more tasks. When scaling by topology‐based mechanism, it eliminates resource usage restriction and execution suspension in the topology, and the procedure is automatically launched. The experimental results show that our topology‐based scaling mechanism can improve the scaling performance of Storm. Copyright © 2016 John Wiley & Sons, Ltd. Abstract : In this paper, we propose a topology‐based scaling mechanism for Apache Storm to scale by adjusting the number of the cloned topologies or replacing another new topology with more tasks. It eliminates resource usage restriction and execution suspension caused by scaling with Storm rebalance command.