DynLW: balancing and scalability for heavy dynamic stream-DB workloads. (1st January 2014)

• Record Type: Journal Article
• Authors: Martins, Pedro; Furtado, Pedro
• Is Part Of: International journal of business intelligence and data mining. Volume 9:Number 1(2014)
• Page Start: 15
• Page End: 30
• Publication Date: 2014-01-01
• Subjects: complex event processing -- CEP -- parallel processing -- distributed systems -- database -- scalability -- load-balance -- algorithm; 006.312
• Languages: English
• ISSNs: 1743-8187

Abstract

In the past, data management research has concentrated in separate data processing issues: heavy database like query processing, and throughput of stream data processing over high-rate data (CEP). However, in many practical contexts, high-rate stream and heavy data processing work together, for correlation, lookup, aggregation, merging or comparison with large amounts of previous data. We refer to these as stream-DB workloads. One way to provide scalability with any off-the-shelf engine is to have multiple machines and/or processor cores, and to parallelise the load (external scheduler), but nodes can still overload. We propose automated control for balancing and scalability over stream-DB workloads. The approach, called DynLW, offers scalability with an integrated mechanism that manages overload (re)scheduling, automated elasticity, shedding, admission control and overload alerts when resources are insufficient. As a result, the approach provides continuous and totally balanced operation, and avoids overload-related problems.